HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors

Authors: panwang  pan
zhuo  su
chenguo  lin
zhen  fan
yongjie  zhang
zeming  li
tingting  shen
yadong  mu
yebin  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 11:48:29

Abstract: Despite recent advancements in high-fidelity human reconstruction techniques, the requirements for densely captured images or time-consuming per-instance optimization significantly hinder their applications in broader scenarios. To tackle these issues, we present **HumanSplat**, which predicts the 3D Gaussian Splatting properties of any human from a single input image in a generalizable manner.
Specifically, HumanSplat comprises a 2D multi-view diffusion model and a latent reconstruction Transformer with human structure priors that adeptly integrate geometric priors and semantic features within a unified framework. A hierarchical loss that incorporates human semantic information is devised to achieve high-fidelity texture modeling and impose stronger constraints on the estimated multiple views. Comprehensive experiments on standard benchmarks and in-the-wild images demonstrate that HumanSplat surpasses existing state-of-the-art methods in achieving photorealistic novel-view synthesis. Project page: https://humansplat.github.io.

Main paper: 
 introduction: 
 Realistic 3D human reconstruction is a fundamental task in computer vision with widespread applications in numerous fields, including social media, gaming, e-commerce, telepresence, etc. Previous works for single-image human reconstruction can be broadly categorized into explicit and implicit approaches. Explicit methods [1, 2, 3, 4] , such as those based on parametric body models like SMPL [5, 6] , estimate human meshes by directly optimizing parameters and clothing offset to fit the observed image. However, these methods often struggle with complex clothing styles and require lengthy optimization. Implicit methods [7, 8, 9, 10, 11] represent humans using continuous functions, such as occupancy [7] , SDF [12] , and NeRF [13] . While they excel at modeling flexible topology, these methods are limited in scalability and efficiency due to the high computational cost associated with training and inference. Recent advances in 3D Gaussian Splatting (3DGS) [14] have provided a balance between efficiency and rendering quality for reconstructing detailed 3D human models, which however rely on multi-view image [15, 16, 17] or monocular video [18, 19, 20] input. Recent popular human reconstruction studies [21, 22, 23, 24] focus on the score distillation sampling (SDS) technique [25] to lift 2D diffusion priors to 3D, but time-consuming optimization (e.g., 2 hours) is required for each instance. Some generalizable and large-reconstruction-model-based works [10, 26, 27, 28] can directly generalize the regression of 3D representations but either disregard human priors or require multi-view inputs, limiting the stability and feasibility in downstream applications.In this paper, we propose a novel generalizable approach HumanSplat for single-image human reconstruction by introducing a generalizable Gaussian Splatting framework with a fine-tuned 2D multi-view diffusion model and well-conceived 3D human structure priors. Different from existing human 3DGS methods, our approach directly infers Gaussian properties from a single input image, Figure 1 : Our method achieves state-of-the-art rendering quality while maintaining the fastest runtime. (a) Qualitative results: LGM [29] and GTA [30] are generalizable but in lower quality, TeCH [21] exhibits issues with multi-face rendering and is time-consuming. In contrast, our method achieves higher fidelity in a much shorter time. (b) Performance and runtime comparison: metrics are evaluated on the challenging Twindom dataset.eliminating the requirement for per-instance optimization or densely captured images. This empowers our method to generalize effectively in diverse scenarios while delivering high-quality reconstructions.The key insight behind our method is to reconstruct Gaussian properties from the diffusion latent space in a generalizable end-to-end architecture, integrating a 2D generative diffusion model as appearance prior and a human parametric model as structure prior. Specifically, facing this under-constraint reconstruction problem from single-view input with extensive invisible regions, we first leverage a 2D multi-view diffusion model (denoted as novel-view synthesizer) to hallucinate the unseen parts of clothed humans. Then a generalizable latent reconstruction Transformer is proposed to enable interaction among the generated diffusion latent embeddings and the geometric information of the structured human model, enhancing the quality of 3DGS reconstruction. During interactions, to mitigate the limitations of inaccurate human priors like the SMPL model, we devise a projection strategy to strike a balance between robustness and flexibility, projecting 3D tokens into the 2D space and conducting searches within adjacent windows using projection-aware attention. Another specific challenge of human reconstruction is to capture fine detail in visually sensitive areas, such as the face and hand. To address this problem, we exploit the semantic cues from the structure priors and propose semantics-guided objectives to further promote the fine details reconstruction quality. By synergistically amalgamating these crucial components into a unified framework, our method achieves state-of-the-art performance in striking the right balance between quality and efficiency.In summary, the main contributions of our work are as follows:• We propose a novel generalizable human Gaussian Splatting network for high-fidelity human reconstruction from a single image. To the best of our knowledge, it is the first to leverage latent Gaussian reconstruction with a 2D generative diffusion model in an end-to-end framework for efficient and accurate single-image human reconstruction.• We integrate structure and appearance cues within a universal Transformer framework by leveraging human geometry priors from the SMPL model and human appearance priors from the 2D generative diffusion model. Geometric priors stabilize the generation of high-quality human geometry, while appearance prior helps hallucinate unseen parts of clothed humans.• We enhance the fidelity of reconstructed human models by introducing semantic cues, hierarchical supervision, and tailored loss functions. Extensive experiments demonstrate that our method achieves state-of-the-art performance, surpassing existing methods. 
related work: 
 Single-Image Human Reconstruction. Single-image human reconstruction methods fall into explicit and implicit approaches. Explicit methods use parametric body models [5, 6] to estimate minimally clothed human meshes [1, 2, 31, 32, 33, 34, 35, 36, 37] , adding clothing via 3D offsets [3, 4, 38, 39, 40, 41] or garment templates [42, 43, 44] . However, these methods are constrained by topology, particularly with loose clothing. Implicit methods [7, 8, 9, 10, 11, 22, 27, 45, 46] , utilizing representations like occupancy, signed distance fields (SDF) and Neural Radiance Fields (NeRF), offer flexibility with topology, allowing for accurate depiction of 3D clothed humans. Recent methods [30, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61] combine implicit representation with explicit human priors for better robustness. Among these, GTA [30] uses Transformers with fixed embeddings for translating image features into 3D tri-plane features. TeCH [62] employs diffusion-based models for visualizing unseen areas but needs extensive optimization and accurate SMPL-X models. Another approach, HumanSGD [22] , uses diffusion models for texture inpainting but struggles with mesh inaccuracies from [8] . NeRF-based methods like SHERF [10] generate human NeRF from single images using hierarchical features, while ELICIT [11] leverages CLIP [63] for contextual understanding. These NeRF-based methods produce high-quality images but often struggle with detailed 3D mesh generation and require extensive optimization time.Human Gaussian Splatting. While implicit representations like SDF or NeRF struggle with balance optimization efficiency and rendering quality, the high efficiency of 3DGS [14] has advanced 3D human creation. Recent 3DGS methods have targeted multi-view videos, monocular video sequences, and sparse-view input. Multi-view video methods like D3GA [15] , HuGS [16] , and 3DGS-Avatar [17] exploit dense spatial and temporal information for detailed models. Additionally, HiFi4G [64] combine 3DGS with a dual-graph mechanism to preserve spatial-temporal consistency, ASH [65] utilizes mesh UV parameterization for real-time rendering, and Animatable Gaussians [66] improve garment dynamics via pose projection mechanism and 2D CNNs. The monocular video only provides sufficient observation as humans move, so the monocular human Gaussian Splatting methods [18, 19, 20, 67, 68, 69, 70, 71, 72, 73] often resort to using LBS for mapping to a canonical space and introducing additional regularization terms. For sparse-view images, GPS-Gaussian [28] achieves high rendering speeds without per-subject optimization by using a feed-forward way with efficient 2D CNNs encoding from diverse 3D human scan data. In comparison, our work stands out as the first one-shot generalizable human GS method, using only single image input and achieving high-quality reconstruction without any optimization and fine-tuning.Generalizable Large Reconstrucion Model. Large reconstruction model (LRM) [74, 75] presents that with sufficient parameters and training datasets, a deterministic feed-forward Transformer [76] is capable of learning a triplane feature for volumetric rendering from a single-view image. Targeting general 3D object, TriplaneGaussian [77] combines 3DGS with triplane as a hybrid 3D representation for fast rendering, where point clouds are first extracted from an image and then projected on triplane features to decode 3DGS attributes. Most recent methods further leverage 2D diffusion models that can generate multi-view images simultaneously [78, 79, 80, 81] to provide plausible and abundant inputs for the following reconstruction, so they focus on the sparse multi-view reconstruction task. Instant3D [81] follows the technique of LRM and produces triplane features from 4 posed images by a Transformer encoder. CRM [82] , MeshLRM [83] and InstantMesh [84] replaces triplane NeRF by FlexiCubes [85] to extract meshes directly.LGM [29] , GRM [86] and GS-LRM [87] predict 3DGS attributes from each input pixel [88, 89] and combine the Gaussians from multiple viewpoints as the final 3D output. For human task, CharacterGen [90] follows Instant3D [81] to adopt a twostage approach: first, generating a 2D multi-view canonical images, then using multi-view SDF reconstruction for cartoon digital characters, without direct interaction between stages. A more compact and elegant method HumanLRM [27] replaces training data of LRM [74] from general objects with human data to predict triplane NeRF, showcasing the capacities of the large model. While it relies heavily on the dataset, often leading to issues like missing limbs due to a lack of prior knowledge. In contrast, our 3DGS-based method leverages the latent diffusion and reconstruction model in an end-to-end framework and introduces human priors by addressing their inaccuracy with specific projection strategies, achieving better generalization and stability with even less data.3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See Sec. 5. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 SMPL [5] is a widely-used parametric human body model that is created by skinning and blend shapes, and learned from thousands of 3D body scans. The SMPL model [5] utilizes shape parameters β ∈ R 10 and pose parameters θ ∈ R 24×3 to parameterize the deformation of the human body meshM(β, θ) : β × θ → R 6890×3 .3D Gaussians Splatting [14] is a recently proposed 3D representation that formulates 3D content by a set of 3 attributes, where µ i ∈ R 3 is its location and Σ i ∈ R 3×3 implies its scaling s i ∈ R 3 and orientation quaternion q i ∈ R 4 in 3D space. These attributes constitute G ∈ R Np×14 can represent a 3D object, and by projecting to the image plane with opacity-based color composition in the depth order, it can be differentiably rendered into 2D images in real-time.N p colored Gaussians G = {g i } Np i=1 . Each Gaussian g i = exp -1 2 (x -µ i ) ⊤ Σ -1 i (x -µ i ) has opacity σ i ∈ R and color c i ∈ RGiven a single input image of a human body I 0 ∈ R H×W ×3 , our task is to reconstruct a 3D representation, i.e., 3DGS in this work, from which novel-view images can be rendered. As illustrated in Fig. 2 , our method leverages a 2D diffusion model and a novel latent reconstruction Transformer that adeptly integrates 2D appearance priors, human geometric priors, and semantic cues within a universal framework. First, we use SMPL estimator [91, 92] to predict human structure priors M, and CLIP [63] to generate the image embedding of the input image c. A latent temporal-spatial diffusion model [93] , referred to as the novel-view synthesizer (Sec. 3.3), is fine-tuned and produces multi-view latent features F i ∈ R h×w×c N i=1 , where N is the number of views, and h, w and c are the height, width and channel of the latent features. Then, we utilize a novel latent reconstruction Transformer (Sec. 3.4) that adeptly integrates human geometric prior and latent features within a universal framework, which predicts Gaussian attributes G := {(µ i , q i , s i , c i , σ i )|i = 1, ..., N p } and rendered into new images, where N p represents the number of Gaussian points. Furthermore, to achieve high-fidelity texture modeling and better constrain the estimated multiple views, we designed a hierarchical loss that incorporates human semantic priors (See 3.5). Note that during training, the network is trained using 3D scan data to ensure accurate supervision from various viewpoints, where the registered SMPL is fitted via the multi-view version of SMPLify [92] . During inference, the model employs PIXIE [91] to predict human structure priors M and synthesize novel views from only a single image input based on the trained model, without any fine-tuning and optimization.We take advantage of the pre-trained video diffusion model SV3D [93] as appearance prior. For the input image I 0 , we leverage a CLIP image encoder [63] to obtain the image embedding c and a pre-trained VAE [94] E to acquire latent feature F 0 as conditions. Subsequently, we progressively denoises gaussian noises into temporal-continuous N multi-view latent features {F i } N i=1 by a spatialtemporal UNet D θ [95] with the objective:EQUATIONwhere {F ϵ i } N i=1 is the noise-corrupted multi-view latent features, {e i , a i } N i=1 is the corresponding relative elevation and azimuth angles to I 0 , p(ϵ) is the noise probability distribution and λ(ϵ) ∈ R + is the loss weighting term related to the noise level ϵ. Since SV3D is merely pre-trained on the dataset of objects [96] , we further fine-tune it using human datasets to improve its effectiveness in the human reconstruction task. More technical details are available in Appendix A.1.Latent reconstruction Transformer contains two parts: latent embedding interaction (Sec. 3.4.1) and geometry-aware interaction (Sec. 3.4.2), which seamlessly incorporate latent features from novelview synthesizer and human structure priors into the reconstruction process. The two components together form a cohesive framework for accurately recovering intricate human details.We utilize a pretrained VAE [94] encoder E to encode the input image I 0 into a latent representation F 0 = E(I 0 ) ∈ R h×w×c and combine it with the generated {F i } N i=1 (Sec. 3.3). Following previous works [97, 98] , latent representations are concatenated with their corresponding Plücker embeddings [99] along the channel dimension, resulting in a dense pose-conditioned feature map. These latents are then divided into non-overlapping patches [100] and mapped to d-dimensional latent tokens by a linear layer. An intra-attention module is followed to thoroughly extract spatial correlations of latent tokens, as shown in Fig. 3 , by a standard Transformer [76] with N intra blocks of multi-head self-attention and feed-forward network (FFN):Fi = [FFN(SelfAttention(F i ))] ×Nintra .(2)Human Geometric Tokenizer. Given a human geometric model M ∈ R 6890×3 , to obtain the feature representation, we project M onto the input view and compute the feature vector through bilinear interpolation on the feature grids byEQUATIONwhere R 0 , t 0 are camera extrinsic parameters of the input view, K is the intrinsic parameters.After the projection operation, we obtain geometric human prior tokens by concatenating M and F 0 [Π 0 (M)], and mapping them to d dimension. F 0 [Π 0 (M)] stands for querying the input latent feature F 0 from projected locations. Similar to Latent Embedding Interaction (Sec. 3.4.1), an intraattention module is utilized to interact among these human prior tokens and get human geometric features H ∈ R 6890×d .Human Geometry-aware Attention. Empirically, we find that incorporating human priors [5, 6, 101] (e.g., SMPL model) into human reconstruction faces a significant dilemma of the trade-off between robustness and flexibility [30, 102] . On the one hand, the SMPL model provides geometric priors that alleviate common issues (e.g., broken body parts and various artifacts). On the other hand, it struggles to accurately capture and represent a broad spectrum of complex clothing styles, particularly more fluid garments like dresses and skirts. This limitation highlights a fundamental shortcoming in the model's capacity to accurately depict diverse apparel. As shown in Fig. 3 , HumanSplat addresses inaccuracies in human priors by projecting 3D tokens into 2D space and conducting local searches within adjacent windows, efficiently using priors while minimizing redundancy. Specifically, we introduce projection-aware attention within the inter-attention module, utilizing a window W(K win × K win ) and employing masked multi-head cross-attention. Here, the latent features Fi N i=1 serve as queries, and the human prior tokens H serve as keys and values. The process can be formulated as follows:EQUATIONwhere feature interactions occur when Π( H, K, R i , t i ) is within the window of Fi . It is noteworthy that the complexity of the cross-attention operation is reduced fromO(L F × L H ) to O(L F × K 2 win), compared to the vanilla multi-head cross-attention setting, where L F and L H represent the lengths of the latent and human geometric tokens respectively.Q Q Q Q Q Geometry-aware Interaction 𝐹 ! … K/V Inter-attention Module … Q Q Q Q Q Q Q Q Q Q … … {𝐹 " }Human Geometryaware Interaction Q Q Q Q Q … Novel-view SynthesizerFrom each output token Fi , we decode the attributes of pixel-aligned Gaussians G in the corresponding patch by a convolutional layer with a 1 × 1 kernel. These attributes are used to render novel-view images through 3D Gaussian Splatting rasterization. An ideal training objective should ensure that the rendered outputs closely match the supervised images, striving for overall consistency between the reconstructed renderings and the ground truth. However, human attention tends to be particularly focused on facial regions, where intricate details play a pivotal role in perception. Therefore, to enhance the fidelity of reconstructed human models, we optimize our objective functions to preferentially focus on the facial regions.Hierarchical Loss. Traditional approaches often overlook the semantic richness inherent in human anatomy, leading to reconstructions that lack crucial details and accuracy. To address this, we propose a novel framework that harnesses semantic cues, hierarchical supervision, and tailored loss functions to guide the training process. This ensures that not only are the overall structures of the human body accurately represented, but also that critical areas such as the face are reconstructed with exceptional detail and fidelity. Specifically, by incorporating a human prior model equipped with semantic information, which adheres to the widely-used definition of 24 human body parts, such as "right hand", "left hand", "head", etc. The ground-truth body part segmentation is obtained from SMPL meshes with predefined semantic vertices [103] . Therefore, we can utilize different attention weights for different body parts and render different levels of real images from different viewpoints to provide hierarchical supervision. This approach significantly facilitates the precise localization of essential body parts, including the head, hands and arms. Please refer to Appendix A.2 for more detailed procedural insights. The overall loss is defined as a weighted sum of part-specific losses:EQUATIONwhere i ∈ {1, ..., n} and j ∈ {1, ..., m} denote different resolution levels and human parts. Îi,j and I i,j are the predicted and ground-truth data for part j, λ i and λ j are weighting factors that reflect the relative importance of each resolution level and each body part, respectively. L MSE measures the MSE loss and L p measures the perceptual loss [104] .Reconstruction Loss. Reconstruction loss L Rec based on the N render rendered multi-view images is defined as where I i and Îi are the ground-truth images and render images via Gaussian Splatting, M i and Mi are original and rendered foreground mask. λ m and λ p are hyperparameters for balancing three kinds of losses. The final training objective is a combination of the two loss terms:EQUATIONL = L H + L Rec .4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
implementation details: 
 Training Details. HumanSplat is trained on 500 THuman2.0 [105] , 1500 2K2K [109] , and 1500 Twindom [106] high-fidelity human scans. We evenly position 36 cameras across each of three hierarchical cycles to capture the full body, half body, and face, with rendering resolution set to 512 × 512. We conduct 200 epochs of 256-res training with a learning rate of 1e-5 and a batch size of 32 over 2 days on 8 A100 (40G VRAM) GPUs, while 512-res finetuning costs 2 additional days. We train our model with AdamW [110] optimizer, whose β 1 , β 2 are set to 0.9 and 0.95 respectively. A weight decay of 0.05 is used on all parameters except those of the LayerNorm layers. We use a cosine learning rate decay scheduler with a 2000-step linear warm-up and the peak learning rate is set to 4e-4. For parts related to the head, hands, and arms, λ j are set to 2, while the rest human part are set to 1. The parameters λ i , λ p and λ m are set to 1. The model is trained for 80K iterations on 256-res and then fine-tuned on 512-res for another 20K iterations. To enable efficient training and inference, we adopt Flash-Attention-v2 [111] in the xFormers [112] library, gradient checkpointing [113] , and FP16 mixed-precision [114] . Fine-tuning the novel view synthesizer takes about 18 hours. Please refer to Appendix A.1 for more details about novel view synthesizer fine-tuning.Inference Time. The video diffusion model takes about 9s to generate multi-view latent features while the subsequent latent reconstruction for 3DGS takes only about 0.3s. Additionally, it can render novel views at a rate exceeding 150 FPS on a NVIDIA A100 GPU.Quantitative Comparison. We conduct a quantitative comparison on 21 THuman2.0 [105] and 51 Twindom [106] scans, rendering textured meshes from multiple angles and using PSNR, SSIM [115] , and VGG-LPIPS [104] as evaluation metrics. To ensure fairness, we fine-tune LGM [29] on our datasets and fully optimize each TeCH [21] Qualitative Comparison. As illustrated in Fig. 4 and Fig. 5 , by incorporating semantics-guided objectives, Humansplat achieves more detailed and higher fidelity results against GTA [30] and Figure 4 : Qualitative comparison of ours against TeCH [21] , GTA [30] and LGM [29] on Thu-man2.0 [105] , Twindom [106] and in-the-wild images. Our method achieves the highest quality. Note that TeCH achieves clearer results but fails to preserve the face identity.LGM [29] . Although TeCH [21] also generates high-quality images, its SDS optimization often results in overly saturated multi-face outcomes [116, 117] . Moreover, while TeCH delivers the highest PSNR on the Thuman2.0 dataset, the LPIPS metric in Tab. 1, confirms that our method provides more realistic results. As shown in Fig. 6 , Humansplat outperforms HumanSGD on inthe-wild images from Adobe Stock, effectively predicting 3D Gaussian Splatting properties without per-instance optimization. Notably, the results on in-the-wild images further demonstrate our strong generalization capability and our model provides superior textures on both the front and invisible regions, highlighting the effectiveness of incorporating appearance and geometric priors.Latent Reconstruction Transformer. HumanSplat leverages the Stable Diffusion latent space for feature extraction. Experimental results demonstrate that utilizing the compact latent space for reconstructions leads to better rendering quality compared to using the pixel space, as detailed in Tab. 2 (a). We also find that patching in the time dimension for latent decoding by temporal VAE [95] adversely affects the quality, so we employ the vanilla VAE [94] decoder in this work.Human Geometric Prior. Tab. 2 (b) illustrates the significance of human geometric prior in our approach. By utilizing SMPL as the geometric prior for human representation, we assess the robustness of HumanSplat with respect to different SMPL parameters. We report both the estimated [2] and Ground Truth SMPL parameters [92] and compare the model that does not incorporate a human structure prior. These experiments provide insight into the benefit of incorporating a human structure prior, as significant performance degradation is observed without it. Furthermore, thanks to treat human priors as keys and values, HumanSplat exhibits robustness in testing against challenging poses with imperfect estimation, with only minor declines in metrics. The effectiveness of the human geometric prior is also demonstrated qualitatively in Fig. 7 (a) .Window Size. As analyzed in Sec. 3.4.2, human priors are crucial for the robustness of the human reconstruction model. As evidenced in Tab. 2 (c), when the window size K win is set to 1 (dubbed "Projection Only"), employing human priors significantly enhances performance, improving the PSNR metric from 22.635 to 23.333. We further explore setting window size to K win = 2 in a latent space with a resolution of 64 × 64, which further improves the model's PSNR metric from 23.333 to 24.294. This improvement is attributed to Humansplat with K win = 2 is capable of handling loose clothing and tolerating imprecise SMPL parameters. In Tab. 3, we present results for the 2K2K dataset across various window sizes. Further refinement with a window size of K win = 3 continues to enhance the results, whereas using K win = 4 does not yield additional improvements.Semantics-guided Training Objectives. Thanks to the incorporation of semantic-guided hierarchical losses, HumanSplat can generate higher-quality textures for both the face and hand, while also promoting the convergence of the training process, as depicted in Fig. 7 (a) and (b). Furthermore, Fig. 7 (c) demonstrates that HumanSplat further produces 3D coherent segmentation results, which have potential applications in editing and specific generation of individual parts of 3D humans.  
conclusion: 
 We present HumanSplat, a pioneering generalizable human reconstruction network that derives 3D Gaussian Splatting properties from a single image. This model integrates generative diffusion and latent reconstruction Transformer models with human structure priors, enhanced by a tailored semantic-aware hierarchical loss. These innovations achieve high-fidelity reconstruction results in a feed-forward manner without any optimization or fine-tuning, especially in the important focal areas such as the face and hands. Extensive experiments demonstrate that HumanSplat surpasses existing state-of-the-art methods in both quality and efficiency, including robust performance in handling challenging poses and loose clothing. This capability opens up a broad spectrum of potential applications.Limitations and Future Works. Our method has some limitations that can be addressed in future works: (a). While effective with most attire, the model may struggle with intricate garments and accessories. Future enhancements should incorporate 2D data and innovative training strategies, which is a promising direction for improving the handling of diverse and unusual clothing styles. (b).Although already efficient, increasing the computational speed by constraining Gaussian's number or combining compact 3DGS with texture representation could further facilitate real-time applications, particularly on mobile devices. (c). Animating the reconstructed human models currently requires post-processing. Future work could introduce canonical space to reconstruct animatable avatars directly, streamlining the process and improving efficiency. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Appendix C.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Conditions of Diffusion Model. SVD and SV3D [93, 95] replace CLIP text embeddings [63] with the image embedding of the conditioning. Compared to text prompts, image prompts provide information more accurately and directly, and also save time that would otherwise be spent on text inversion or image caption [21, 102] . SV3D takes an image and the relative camera orbit elevation and azimuth angles as inputs to generate several corresponding novel-view images, which are continuous in 3D space and can be regarded as a video with the camera moving. In our work, we finetuned SV3D on human datasets with relative camera elevation and azimuth angles to generate 4 orthogonal views, which we found to be sufficient for reconstruction [102] .Transformer Block. The CLIP embeddings obtained from the conditioning image is fed into the cross-attention layers of each Transformer block, acting as keys and values, while the feature at that layer serves as the queries. Moreover, the camera trajectory and the diffusion noise timestep are incorporated into the residual blocks.Fine-tuning Details. We involves the widely used EDM [118, 119] framework, incorporating a simplified diffusion loss for finetuning. We fine-tune the novel-view synthesizer at 512-res uses 1000-step warmup with the peak learning rate 1e-4 in the cosine learning rate decay schedule. We use a per-GPU batch size of 8 objects during 256-res training, and a per-GPU batch size of 4 during 512-res finetuning stage. For each instance, we use 4 input views and 4 novel supervision views at each iteration of 256-res training and 512-res finetuning.Triangular CFG Scaling. Classifier-free Guidance (CFG) [120] is a widely used technique to trade off controllability with diversity. However, this scaling causes the last few frames in our generated orbits to be over-sharpened. To tackle this issue, we use a triangle wave CFG [93, 121] scaling during inference: linearly increase CFG from 1 at the front view to 2.5 at the back view, then linearly decrease it back to 1 .3D Consistency Evaluation. We also conduct an evaluation of the 3D consistency of our novel-view synthesizer using the advanced local correspondence matching algorithm, MASt3R [122] . This approach involves one-to-one image matching between input views and their generated novel views, with the average number of matching correspondences serving as our metric. Our synthesizer exhibits a notable improvement in 3D consistency for human novel-view generation after fine-tuning with human datasets. Specifically, the original SV3D achieves an average of 723.25 matching points, while our novel-view synthesizer increases this number to 930.13. In comparison, the ground truth demonstrates a matching point number of 1106.33. This evaluation highlights that fine-tuning on human datasets significantly enhances multi-view consistency metrics.Network Architecture. The network architecture of the proposed latent reconstruction Transformer is shown in Fig. 8 . The framework is composed of a latent encoder with intra-and inter-attention and a Gaussian parameter prediction module. It takes initial multi-view latent embeddings as input and performs projection-aware attention to aggregate human geometric information. From each output token, we decode the attributes of pixel-aligned Gaussians in the corresponding patch with a Conv 1 × 1 layer.Hierarchical Loss. Our approach is efficient and can be utilized online without pre-computation. We calculate the visible face triangles given the mesh and camera parameters. Then each visible triangular face is assigned the corresponding semantic label [103] . Additionally, it provides labels with superior 3D consistency and avoids issues associated with training instability. First, the hierarchical loss is specifically focused on the head and hands, excluding hair and clothing, which can be further supervised using reconstruction loss L Rec . Second, instead of using segmentation to reconstruct and then concatenate different components, we design our loss function based on part segmentation. Thank to this design, HumanSplat is tolerant of inaccuracies in segmentation. For practical applications demanding high input view reconstruction quality, which could potentially be directly derived from the input image. We improve the weights of the reconstruction loss for input views (referred to as reweighting loss), demonstrating that the current model with reweighting loss exhibits sufficient capacity to render input views with higher fidelity, as illustrated in Fig. 9 . We also conduct an ablation experiment for the reweighting loss function on the 2K2K Dataset [109] . There are slight declines in quantitative metrics, specifically, HumanSplat with reweighting loss achieves a numerical change of -0.141, -0.015, and +0.006 in PNSR, SSIM, and LPIPS. This suggests that reweighting loss is not "a bag of freebies" and HumanSplat without reweighting loss strikes a balance between the quality of novel views and the input views.As for novel view and novel pose rendering, more results are shown in Fig. 10 and Fig. 11 . For a single input image, we use Open-AnimateAnyone [123] for novel pose synthesis and HumanSplat for novel view synthesis. This functionality enables dynamic timestamps and real-time rendering of novel views, thereby enhancing immersive virtual exploration. Please also kindly refer to our supplementary video for more results.While our model demonstrates the capability to reconstruct photo-realistic 3D avatars, it also introduces potential risks such as privacy violations. To mitigate these concerns, it is imperative to establish robust ethical guidelines and legal frameworks. This necessitates a collaborative effort among researchers, developers, and policymakers to promote the responsible use of this technology and safeguard against its potential misuse. NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: - This paper presents a model for single-view 3D human reconstruction with Gaussian splats. It uses a fine-tuned diffusion-based novel view synthesizer to generate multiple views of the input and proposes a prior-aware latent reconstruction model for Gaussian splat prediction. The reviewers (**2EjT,**  **zEoa, eC4L**) ****acknowledge the technical contribution of using masked cross-attention to incorporate the SMPL mesh prior as tokens.
- Concerns are mainly raised around the influence of introduced SMPL prior (**zEoa, 6Egy,  eC4L**) and missed visual comparisons (**2EjT, 6Egy)**
- In the rebuttal, the authors did a good job of addressing the questions by providing ablation for SMPL pose priors and additional visual results for HumanSGD and in-the-wild results.
- The AC agrees with the reviewers and recommends its acceptance, and also requests the authors to incorporate the reasoning and results in the responses to the final version.

Reviews: review: paper_summary: paper_summary: - The paper, HumanSplat, focuses on photorealistic novel-view synthesis of humans from a single image.
- The key idea is to use a multi-view synthesizer based on SV3D to hallucinate the other views + latent reconstruction transformer to predict the 3DGS.
- Importantly, HumanSplat does not use optimization and is much faster than existing methods.
- Baselines:  Magic, HumanSGD, TeCH
- Evaluations are done on Thuman2.0 and Twindom datasets. Metrics: PSNR, SSIM, LPIPS
- The proposed methods consistently outperforms the existing methods.
strength_weakness: strength_weakness: - Measuring 3D consistency of SV3D: The novel views generated by fine-tuned SV3D is important for predicting the gaussian parameters. An ablative study on the effectiveness of the novel synthesizer would be really helpful to understand the in-the-wild generalization of the proposed method. 3D consistency could be measure using stereo-based matching or local correspondences.

- Qualitative results: Although, Table. 1 showcases better performance than HumanSGD on THuman2.0 dataset, notably in all metrics. The qualitative comparison between HumanSGD and HumanSplat is missing. HumanSGD results demonstrate much better modelling of human skin tone and overall 3D geometry. I understand the code is yet to be released, however, would it be possible to compare on similar stock images?

- Minor: consider renaming "project-aware" -> "projection-aware". The writing at times is a little too non-technical, eg. L206-207, consider rephrasing such instances.
questions: questions: - Understanding the generalizability of novel-view synthesizer. Is it limited to full-body images? Any quantitative insights into 3D consistency.
- Qualitative comparisons to HumanSGD on stock images.
limitations: limitations: Yes. The authors provide a discussion about utilizing 2D data and handling diverse clothing styles.
paper_summary: paper_summary: This paper proposes HumanSpat, a method that predicts 3D Gaussians from a single image of a human. The method comprises a 2D multi-view diffusion model and a latent reconstruction transformer that integrates human body prior.
strength_weakness: strength_weakness: - Missing ablation on human prior. The paper should include ablation experiments where there is no human structure prior, as well as experiments where the human structure prior is estimated instead of GT. These experiments can provide insight on the benefit of the specific human structure prior fusion design.
questions: questions: In Table 1, how was the reconstruction time computed? PIFu is very lightweight and should not take 30 seconds per human.
limitations: limitations: Limitations are addressed.
paper_summary: paper_summary: This paper proposes a generalizable human rendering framework from single images. The proposed method relies on different priors and achieves state-of-the-art results on multiple datasets.
strength_weakness: strength_weakness: - Qualitative visual results on the rendered images are not satisfactory. It is acceptable that renderings of unseen views of the human can be relatively poorer. It is surprising to see renderings of exactly the input views are also in low quality (Figure 4). This is usually a sign of poor model capacity, which may be attributed to the proposed objective functions or the model itself.
- It is not clear how the 'parts' of human are defined. Are they integer based index as in iuv? If so, how to obtain the ground truth body parts labels? Are they provided by the dataset or they have to be pre-computed by some off-the-shelf models. What if there are prediction errors in the 'ground truth' semantic labels? Is the proposed method robust to this kind of errors? I also have doubts on its effectiveness in terms of the overall rendering quality since there is no ablation study provided to show how it improves quantitative/qualitative results.
- As presented in Table 2 (b), it seems that the proposed method is very sensitive to geometric prior (i.e. SMPL estimations). Since the model relies on PIXIE for SMPL prediction, what if there are errors in the predictions (most likely for challenging poses or complex clothes/background), how will the proposed method perform under those cases? This on the contrary shows advantages of geometric prior free methods such as LRM and HumanLRM. If there is a benchmark on human rendering with challenging poses, I am afraid those methods will perform better than the proposed one.
- I wonder why there are not qualitative comparisons with SIFU but only quantitative metrics?
- I'd suggest to change Figure 1 (b) to a different style. The dashes in the graph are misleading and confuse readers as they are in different scales.
questions: questions: Please see the weakness section.
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: This paper presents a 3D human reconstruction system that takes a single RGB image and outputs 3D Gaussians, which can render the reconstructed 3D humans to any viewpoint. In contrast to existing 3DGS works, which requires per-instance optimization and cannot be generalized to unseen identities, the proposed system can be used for any unseen identities in zero-shot manner. To achieve this, the authors utilizes 2D mulit-view diffusion model and latent reconstruction transformer with human structure priors. A hierarchical loss, which incorporates human semantic information, is introduced. Experimental results demonstrate the powerful capability of the proposed work.
strength_weakness: strength_weakness: 1. Unclear writings
Overall, technical details are not enough to fully understand the manuscript. 
1-1. For example, Sec. 3.3 describes the video diffusion model. I can’t fully get how the ‘video’ diffusion model can be used for the ‘novel-view’ synthesizer. I understand that a video of an object can model different viewpoints when a camera is moving, but there should be a clear justification and reason why the authors chose this generative model for the novel-view synthesizer. In addition, it seems the video diffusion model does not take the target camera pose. Then which novel view is modeled with the video diffusion model given no camera pose? 
1-2. Also, for the hierarchical loss, how can the authors access the target human image of certain body parts? Do the authors already have GT body part segmentation? If the authors simply use rendered body part segmentation from GT SMPL meshes, I don’t think that is a good choice as SMPL meshes only model naked bodies, while there could be hair and loose clothes in the image.

2. Lack of comparison to SOTA methods
Please compare with more SOTA methods, such as ICON and ECON.

3. Lack of in-the-wild results
As the proposed work aims to generalizable 3D human reconstruction, there should be more diverse generalized results. Only three results in Fig. 4 show the results on in-the-wild inputs, which have quite simple poses. Please report more results from input images with diverse **human poses**, identities, and camera pose.
questions: questions: Please see the weaknesses section.
limitations: limitations: Please see the weaknesses section.

Comments: title: Reply to Rebuttal
comment: Dear Reviewer zEoa,

We appreciate your insightful comments and suggestions on our manuscript. We have thoroughly reviewed each point to address the highlighted concerns:

- Detailed the ablation experiments both without human structure priors and with **estimated human structure priors** (PIXIE).
- Explained the PIFu's time-consuming procedure.

If you find the rebuttal addresses your concerns or have any further questions, please feel free to contact us.

We truly value the chance to refine our work based on your invaluable feedback.

Best Regards,

Authors of #3255
Dear Reviewer eC4L,

Thank you for your insightful comments and suggestions regarding our manuscript. **We have thoroughly reviewed each point and have made substantial revisions to address the concerns highlighted.**

- Clarified the Novel-view Synthesizer (Sec 3.3) and Hierarchical Loss (Sec 3.5) 
- Compared with HumanSGD on Adobe Stock images.
- Added qualitative results on in-the-wild images, which reinforce our claims and address your concerns about diverse input images.

If you feel that the rebuttal addresses any of your concerns, **we kindly request that you consider updating your score** in light of these substantial improvements. 
If you have any further questions, **please do not hesitate to contact us.**

We genuinely value the opportunity to enhance our work based on your recommendations.

Best Regards,

Authors of #3255
**We are pleased to hear that our responses have addressed your concerns.** We truly appreciate your constructive comments throughout the review process, which have greatly helped in improving our work.
Thank you for the rebuttal. My concerns are addressed, I would like to keep the rating of weak accept (6).
Dear Reviewer 2EjT,

We appreciate your insightful comments and suggestions on our manuscript. We have thoroughly reviewed each point to address the highlighted concerns:

- Measured the 3D consistency metric of novel-views synthesizer based on your recommendation.
- Conducted comparisons with HumanSGD on Adobe Stock images.
- Addressed and clarified minor issues highlighted in your feedback.

If you find the rebuttal addresses your concerns or have any further questions, please feel free to contact us.

We truly value the chance to refine our work based on your invaluable feedback.

Best Regards,

Authors of #3255
**We are pleased to hear that our responses have addressed your major concerns.** We truly appreciate your constructive comments throughout the review process, which have greatly helped in improving our work.
Thank you for your thoughtful review and for taking the time to reconsider our submission after evaluating our rebuttal.

The concern you raised regarding "why HumanSplat cannot replicate the input view in high quality?" is indeed thought-provoking. Upon revisiting our methodology and related works, we recognize that HumanSplat represents 3D content using a set of colored Gaussians ${{\mathcal{G}_i}}$, inherently lacking a direct "shortcut" from the input view to the 3D representation.

Conversely, HumanSGD [22] is equipped with a "Multi-view Fusion" module (please refer to **Figure 5 of HumanSGD**). This module optimizes the UV texture map by minimizing the LPIPS and L1 loss between the input and rendered views. However, the downside is a **time-consuming optimization process** that may lead to random and unstable outcomes, such as the **multi-face Janus issue in novel views.**

**For the unsatisfactory input view's results, we will add a discussion to the limitation.** Notably, compared to generalizable methods such as GTA (NeurIPS 2023), SIFU (CVPR 2024), and LGM (ECCV 2024), HumanSplat exhibits fewer blurry and unnatural details in the input view. Compared to these optimization-based methods (e.g., HumanSGD and TeCH), 
$\textcolor{blue}{\textbf{we offer faster reconstruction times and enhanced robustness.}}$

We deeply value your expertise and time. If you have any further questions, please do not hesitate to contact us.
Thanks for the clear answers. Most of the concerns are well addressed.
I thank the authors for their rebuttal. The additional experiments as well as discussions are insightful. I'd like to raise my rating to borderline accept to acknowledge the efforts the authors put into the rebuttal. However, I still want to note on the poor quality of renderings. As a generalizable methods, it is understandable that unseen regions from the input image are poorly generated/rendered. But it is still not a good sign that the model replicates the input view poorly with blurry and unnatural details --- this may, again, indicate something is wrong with the methodology. The comparison results by HumanSGD, as provided in the rebuttal, however does a much better job replicating the input view. 
I'd like to acknowledge this paper's strength as well as note the weakness for AC's reference when making the final decision.