Title: HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors
Venue: NeurIPS 2024
Accepted: yes
Keywords: Single Image Human Reconstruction

Gaussian Splatting

Human Prior

Diffusion

Latent Reconstruction Transformer
Abstract: Despite recent advancements in high-fidelity human reconstruction techniques, the requirements for densely captured images or time-consuming per-instance optimization significantly hinder their applications in broader scenarios. To tackle these issues, we present **HumanSplat**, which predicts the 3D Gaussian Splatting properties of any human from a single input image in a generalizable manner.
Specifically, HumanSplat comprises a 2D multi-view diffusion model and a latent reconstruction Transformer with human structure priors that adeptly integrate geometric priors and semantic features within a unified framework. A hierarchical loss that incorporates human semantic information is devised to achieve high-fidelity texture modeling and impose stronger constraints on the estimated multiple views. Comprehensive experiments on standard benchmarks and in-the-wild images demonstrate that HumanSplat surpasses existing state-of-the-art methods in achieving photorealistic novel-view synthesis. Project page: https://humansplat.github.io.

HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors

Abstract

Despite recent advancements in high-fidelity human reconstruction techniques, the requirements for densely captured images or time-consuming per-instance optimization significantly hinder their applications in broader scenarios. To tackle these issues, we present HumanSplat , which predicts the 3D Gaussian Splatting properties of any human from a single input image in a generalizable manner. Specifically, HumanSplat comprises a 2D multi-view diffusion model and a latent reconstruction Transformer with human structure priors that adeptly integrate geometric priors and semantic features within a unified framework. A hierarchical loss that incorporates human semantic information is devised to achieve high-fidelity texture modeling and impose stronger constraints on the estimated multiple views. Comprehensive experiments on standard benchmarks and in-the-wild images demonstrate that HumanSplat surpasses existing state-of-the-art methods in achieving photorealistic novel-view synthesis. Project page: https://humansplat.github.io .

1 Introduction

Realistic 3D human reconstruction is a fundamental task in computer vision with widespread applications in numerous fields, including social media, gaming, e-commerce, telepresence, etc. Previous works for single-image human reconstruction can be broadly categorized into explicit and implicit approaches. Explicit methods [1, 2, 3, 4], such as those based on parametric body models like SMPL [5, 6], estimate human meshes by directly optimizing parameters and clothing offset to fit the observed image. However, these methods often struggle with complex clothing styles and require lengthy optimization. Implicit methods [7, 8, 9, 10, 11] represent humans using continuous functions, such as occupancy [7], SDF [12], and NeRF [13]. While they excel at modeling flexible topology, these methods are limited in scalability and efficiency due to the high computational cost associated with training and inference. Recent advances in 3D Gaussian Splatting (3DGS) [14] have provided a balance between efficiency and rendering quality for reconstructing detailed 3D human models, which however rely on multi-view image [15, 16, 17] or monocular video [18, 19, 20] input. Recent popular human reconstruction studies [21, 22, 23, 24] focus on the score distillation sampling (SDS) technique [25] to lift 2D diffusion priors to 3D, but time-consuming optimization (e.g., 2 hours) is required for each instance. Some generalizable and large-reconstruction-model-based works [10, 26, 27, 28] can directly generalize the regression of 3D representations but either disregard human priors or require multi-view inputs, limiting the stability and feasibility in downstream applications.

In this paper, we propose a novel generalizable approach HumanSplat for single-image human reconstruction by introducing a generalizable Gaussian Splatting framework with a fine-tuned 2D multi-view diffusion model and well-conceived 3D human structure priors. Different from existing human 3DGS methods, our approach directly infers Gaussian properties from a single input image,

multi-face

coarsetexture

janus issue

24

23

Ours

22

TeCH

PSNR

GTA(5.6Min)

TeCH(4.5Hours)

21

coarsetexture

LGM

SIFU

20

19

GTA

18

10

10

10²

10²

10

InputImage

LGM(9.5Sec)

Ours (9.3 Sec)

Reconstruction time(seconds) in logarithmic scale

(b)

a

eliminating the requirement for per-instance optimization or densely captured images. This empowers our method to generalize effectively in diverse scenarios while delivering high-quality reconstructions.

The key insight behind our method is to reconstruct Gaussian properties from the diffusion latent space in a generalizable end-to-end architecture, integrating a 2D generative diffusion model as appearance prior and a human parametric model as structure prior. Specifically, facing this under-constraint reconstruction problem from single-view input with extensive invisible regions, we first leverage a 2D multi-view diffusion model (denoted as novel-view synthesizer ) to hallucinate the unseen parts of clothed humans. Then a generalizable latent reconstruction Transformer is proposed to enable interaction among the generated diffusion latent embeddings and the geometric information of the structured human model, enhancing the quality of 3DGS reconstruction. During interactions, to mitigate the limitations of inaccurate human priors like the SMPL model, we devise a projection strategy to strike a balance between robustness and flexibility, projecting 3D tokens into the 2D space and conducting searches within adjacent windows using projection-aware attention. Another specific challenge of human reconstruction is to capture fine detail in visually sensitive areas, such as the face and hand. To address this problem, we exploit the semantic cues from the structure priors and propose semantics-guided objectives to further promote the fine details reconstruction quality. By synergistically amalgamating these crucial components into a unified framework, our method achieves state-of-the-art performance in striking the right balance between quality and efficiency.

In summary, the main contributions of our work are as follows:

We propose a novel generalizable human Gaussian Splatting network for high-fidelity human reconstruction from a single image. To the best of our knowledge, it is the first to leverage latent Gaussian reconstruction with a 2D generative diffusion model in an end-to-end framework for efficient and accurate single-image human reconstruction.

We integrate structure and appearance cues within a universal Transformer framework by leveraging human geometry priors from the SMPL model and human appearance priors from the 2D generative diffusion model. Geometric priors stabilize the generation of high-quality human geometry, while appearance prior helps hallucinate unseen parts of clothed humans.

We enhance the fidelity of reconstructed human models by introducing semantic cues, hierarchical supervision, and tailored loss functions. Extensive experiments demonstrate that our method achieves state-of-the-art performance, surpassing existing methods.

2 Related Work

Single-Image Human Reconstruction. Single-image human reconstruction methods fall into explicit and implicit approaches. Explicit methods use parametric body models [5, 6] to estimate

minimally clothed human meshes [1, 2, 31, 32, 33, 34, 35, 36, 37], adding clothing via 3D offsets [3, 4, 38, 39, 40, 41] or garment templates [42, 43, 44]. However, these methods are constrained by topology, particularly with loose clothing. Implicit methods [7, 8, 9, 10, 11, 22, 27, 45, 46], utilizing representations like occupancy, signed distance fields (SDF) and Neural Radiance Fields (NeRF), offer flexibility with topology, allowing for accurate depiction of 3D clothed humans. Recent methods [30, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61] combine implicit representation with explicit human priors for better robustness. Among these, GTA [30] uses Transformers with fixed embeddings for translating image features into 3D tri-plane features. TeCH [62] employs diffusion-based models for visualizing unseen areas but needs extensive optimization and accurate SMPL-X models. Another approach, HumanSGD [22], uses diffusion models for texture inpainting but struggles with mesh inaccuracies from [8]. NeRF-based methods like SHERF [10] generate human NeRF from single images using hierarchical features, while ELICIT [11] leverages CLIP [63] for contextual understanding. These NeRF-based methods produce high-quality images but often struggle with detailed 3D mesh generation and require extensive optimization time.

Human Gaussian Splatting. While implicit representations like SDF or NeRF struggle with balance optimization efficiency and rendering quality, the high efficiency of 3DGS [14] has advanced 3D human creation. Recent 3DGS methods have targeted multi-view videos, monocular video sequences, and sparse-view input. Multi-view video methods like D3GA [15], HuGS [16], and 3DGS-Avatar [17] exploit dense spatial and temporal information for detailed models. Additionally, HiFi4G [64] combine 3DGS with a dual-graph mechanism to preserve spatial-temporal consistency, ASH [65] utilizes mesh UV parameterization for real-time rendering, and Animatable Gaussians [66] improve garment dynamics via pose projection mechanism and 2D CNNs. The monocular video only provides sufficient observation as humans move, so the monocular human Gaussian Splatting methods [18, 19, 20, 67, 68, 69, 70, 71, 72, 73] often resort to using LBS for mapping to a canonical space and introducing additional regularization terms. For sparse-view images, GPS-Gaussian [28] achieves high rendering speeds without per-subject optimization by using a feed-forward way with efficient 2D CNNs encoding from diverse 3D human scan data. In comparison, our work stands out as the first one-shot generalizable human GS method, using only single image input and achieving high-quality reconstruction without any optimization and fine-tuning.

Generalizable Large Reconstrucion Model. Large reconstruction model (LRM) [74, 75] presents that with sufficient parameters and training datasets, a deterministic feed-forward Transformer [76] is capable of learning a triplane feature for volumetric rendering from a single-view image. Targeting general 3D object, TriplaneGaussian [77] combines 3DGS with triplane as a hybrid 3D representation for fast rendering, where point clouds are first extracted from an image and then projected on triplane features to decode 3DGS attributes. Most recent methods further leverage 2D diffusion models that can generate multi-view images simultaneously [78, 79, 80, 81] to provide plausible and abundant inputs for the following reconstruction, so they focus on the sparse multi-view reconstruction task. Instant3D [81] follows the technique of LRM and produces triplane features from 4 posed images by a Transformer encoder. CRM [82], MeshLRM [83] and InstantMesh [84] replaces triplane NeRF by FlexiCubes [85] to extract meshes directly. LGM [29], GRM [86] and GS-LRM [87] predict 3DGS attributes from each input pixel [88, 89] and combine the Gaussians from multiple viewpoints as the final 3D output. For human task, CharacterGen [90] follows Instant3D [81] to adopt a twostage approach: first, generating a 2D multi-view canonical images, then using multi-view SDF reconstruction for cartoon digital characters, without direct interaction between stages. A more compact and elegant method HumanLRM [27] replaces training data of LRM [74] from general objects with human data to predict triplane NeRF, showcasing the capacities of the large model. While it relies heavily on the dataset, often leading to issues like missing limbs due to a lack of prior knowledge. In contrast, our 3DGS-based method leverages the latent diffusion and reconstruction model in an end-to-end framework and introduces human priors by addressing their inaccuracy with specific projection strategies, achieving better generalization and stability with even less data.

3 Method

3.1 Preliminary

SMPL [5] is a widely-used parametric human body model that is created by skinning and blend shapes, and learned from thousands of 3D body scans. The SMPL model [5] utilizes shape parameters

Temporal Attention

CLIP

Input

Image

(Prompt)

SMPL

Cross-

Attention

Cross-

Attention

Cross-

Attention

Cross-

Attention

Self Attention

Cross Attention

(b) Latent Reconstruction Transformer

q

k/v

Human Geometric Tokenizer

…

Human Structure (Geometry & Semantic) Priors

ℳ

3DGS with

semantic label

Multi-view Rendering

C

Concatenation

Spatial Attention

Plücker Rays

×𝑁intra

FFN

Self

Attention

C

(a) Novel View Synthesizer

Condition

c

…

FFN

Ground Truth

×𝑁inter

(c) Semantics-Guided Objectives

C

Self Attention

FFN

×𝑁human

β ∈ R 10 and pose parameters θ ∈ R 24 × 3 to parameterize the deformation of the human body mesh M ( β , θ ) : β × θ ↦→ R 6890 × 3 .

3D Gaussians Splatting [14] is a recently proposed 3D representation that formulates 3D content by a set of N p colored Gaussians G = { g i } N p i =1 . Each Gaussian g i = exp ( -1 2 ( x -µ i ) ⊤ Σ -1 i ( x -µ i ) ) has opacity σ i ∈ R and color c i ∈ R 3 attributes, where µ i ∈ R 3 is its location and Σ i ∈ R 3 × 3 implies its scaling s i ∈ R 3 and orientation quaternion q i ∈ R 4 in 3D space. These attributes constitute G ∈ R N p × 14 can represent a 3D object, and by projecting to the image plane with opacity-based color composition in the depth order, it can be differentiably rendered into 2D images in real-time.

3.2 Overview

Given a single input image of a human body I 0 ∈ R H × W × 3 , our task is to reconstruct a 3D representation, i.e., 3DGS in this work, from which novel-view images can be rendered. As illustrated in Fig. 2, our method leverages a 2D diffusion model and a novel latent reconstruction Transformer that adeptly integrates 2D appearance priors, human geometric priors, and semantic cues within a universal framework. First, we use SMPL estimator [91, 92] to predict human structure priors M , and CLIP [63] to generate the image embedding of the input image c . A latent temporal-spatial diffusion model [93], referred to as the novel-view synthesizer (Sec. 3.3), is fine-tuned and produces multi-view latent features { F i ∈ R h × w × c } N i =1 , where N is the number of views, and h , w and c are the height, width and channel of the latent features. Then, we utilize a novel latent reconstruction Transformer (Sec. 3.4) that adeptly integrates human geometric prior and latent features within a universal framework, which predicts Gaussian attributes G := { ( µ i , q i , s i , c i , σ i ) | i = 1 , ..., N p } and rendered into new images, where N p represents the number of Gaussian points. Furthermore, to achieve high-fidelity texture modeling and better constrain the estimated multiple views, we designed a hierarchical loss that incorporates human semantic priors (See 3.5). Note that during training, the network is trained using 3D scan data to ensure accurate supervision from various viewpoints, where the registered SMPL is fitted via the multi-view version of SMPLify [92]. During inference, the model employs PIXIE [91] to predict human structure priors M and synthesize novel views from only a single image input based on the trained model, without any fine-tuning and optimization.

3.3 Video Diffusion Model as Novel-view Synthesizer

We take advantage of the pre-trained video diffusion model SV3D [93] as appearance prior. For the input image I 0 , we leverage a CLIP image encoder [63] to obtain the image embedding c and a pre-trained VAE [94] E to acquire latent feature F 0 as conditions. Subsequently, we progressively denoises gaussian noises into temporal-continuous N multi-view latent features { F i } N i =1 by a spatialtemporal UNet D θ [95] with the objective:

where { F ϵ i } N i =1 is the noise-corrupted multi-view latent features, { e i , a i } N i =1 is the corresponding relative elevation and azimuth angles to I 0 , p ( ϵ ) is the noise probability distribution and λ ( ϵ ) ∈ R + is the loss weighting term related to the noise level ϵ . Since SV3D is merely pre-trained on the dataset of objects [96], we further fine-tune it using human datasets to improve its effectiveness in the human reconstruction task. More technical details are available in Appendix A.1.

3.4 Latent Reconstruction Transformer

Latent reconstruction Transformer contains two parts: latent embedding interaction (Sec. 3.4.1) and geometry-aware interaction (Sec. 3.4.2), which seamlessly incorporate latent features from novelview synthesizer and human structure priors into the reconstruction process. The two components together form a cohesive framework for accurately recovering intricate human details.

3.4.1 Latent Embedding Interaction

We utilize a pretrained VAE [94] encoder E to encode the input image I 0 into a latent representation F 0 = E ( I 0 ) ∈ R h × w × c and combine it with the generated { F i } N i =1 (Sec. 3.3). Following previous works [97, 98], latent representations are concatenated with their corresponding Plücker embeddings [99] along the channel dimension, resulting in a dense pose-conditioned feature map. These latents are then divided into non-overlapping patches [100] and mapped to d -dimensional latent tokens by a linear layer. An intra-attention module is followed to thoroughly extract spatial correlations of latent tokens, as shown in Fig. 3, by a standard Transformer [76] with N intra blocks of multi-head self-attention and feed-forward network (FFN):

3.4.2 Geometry-aware Interaction

Human Geometric Tokenizer. Given a human geometric model M∈ R 6890 × 3 , to obtain the feature representation, we project M onto the input view and compute the feature vector through bilinear interpolation on the feature grids by

where R 0 , t 0 are camera extrinsic parameters of the input view, K is the intrinsic parameters. After the projection operation, we obtain geometric human prior tokens by concatenating M and F 0 [ Π 0 ( M )] , and mapping them to d dimension. F 0 [ Π 0 ( M )] stands for querying the input latent feature F 0 from projected locations. Similar to Latent Embedding Interaction (Sec. 3.4.1), an intraattention module is utilized to interact among these human prior tokens and get human geometric features ¯ H ∈ R 6890 × d .

HumanGeometry-aware Attention. Empirically, we find that incorporating human priors [5, 6, 101] (e.g., SMPL model) into human reconstruction faces a significant dilemma of the trade-off between robustness and flexibility [30, 102]. On the one hand, the SMPL model provides geometric priors that alleviate common issues (e.g., broken body parts and various artifacts). On the other hand, it struggles to accurately capture and represent a broad spectrum of complex clothing styles, particularly more fluid garments like dresses and skirts. This limitation highlights a fundamental shortcoming in the model's capacity to accurately depict diverse apparel. As shown in Fig. 3, HumanSplat addresses inaccuracies in human priors by projecting 3D tokens into 2D space and conducting local searches within adjacent windows, efficiently using priors while minimizing redundancy. Specifically, we introduce projection-aware attention within the inter-attention module, utilizing a window W ( K win × K win ) and employing masked multi-head cross-attention. Here, the latent features { ¯ F i } N i =1 serve as queries, and the human prior tokens ¯ H serve as keys and values. The process can be formulated as follows:

where feature interactions occur when Π ( ¯ H , K , R i , t i ) is within the window of ¯ F i . It is noteworthy that the complexity of the cross-attention operation is reduced from O ( L F × L H ) to O ( L F × K 2 win ) , compared to the vanilla multi-head cross-attention setting, where L F and L H represent the lengths of the latent and human geometric tokens respectively.

Input	Image

Input view

Reference view

Neighbor gaussians

Attention

Intra-attention	Module

Q

Q

Q

Q

Q

Geometry-aware	Interaction

𝐹

!

…

K/V

Inter-attention	Module

…

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

…

…

{𝐹

"

}

Latent Embedding

Interaction

Human Geometry-

aware Interaction

Q

Q

Q

Q

Q

…

Novel-view	Synthesizer

K

K

FO

F;

xN

3.5 Semantics-guided Objectives

From each output token ˜ F i , we decode the attributes of pixel-aligned Gaussians G in the corresponding patch by a convolutional layer with a 1 × 1 kernel. These attributes are used to render novel-view images through 3D Gaussian Splatting rasterization. An ideal training objective should ensure that the rendered outputs closely match the supervised images, striving for overall consistency between the reconstructed renderings and the ground truth. However, human attention tends to be particularly focused on facial regions, where intricate details play a pivotal role in perception. Therefore, to enhance the fidelity of reconstructed human models, we optimize our objective functions to preferentially focus on the facial regions.

Hierarchical Loss. Traditional approaches often overlook the semantic richness inherent in human anatomy, leading to reconstructions that lack crucial details and accuracy. To address this, we propose a novel framework that harnesses semantic cues, hierarchical supervision, and tailored loss functions to guide the training process. This ensures that not only are the overall structures of the human body accurately represented, but also that critical areas such as the face are reconstructed with exceptional detail and fidelity. Specifically, by incorporating a human prior model equipped with semantic information, which adheres to the widely-used definition of 24 human body parts, such as 'right hand', 'left hand', 'head', etc. The ground-truth body part segmentation is obtained from SMPL meshes with predefined semantic vertices [103]. Therefore, we can utilize different attention weights for different body parts and render different levels of real images from different viewpoints to provide hierarchical supervision. This approach significantly facilitates the precise localization of essential body parts, including the head, hands and arms. Please refer to Appendix A.2 for more detailed procedural insights. The overall loss is defined as a weighted sum of part-specific losses:

where i ∈ { 1 , ..., n } and j ∈ { 1 , ..., m } denote different resolution levels and human parts. ˆ I i ,j and I i,j are the predicted and ground-truth data for part j , λ i and λ j are weighting factors that reflect the relative importance of each resolution level and each body part, respectively. L MSE measures the MSE loss and L p measures the perceptual loss [104].

Reconstruction Loss. Reconstruction loss L Rec based on the N render rendered multi-view images is defined as

where I i and ˆ I i are the ground-truth images and render images via Gaussian Splatting, M i and ˆ M i are original and rendered foreground mask. λ m and λ p are hyperparameters for balancing three kinds of losses. The final training objective is a combination of the two loss terms: L = L H + L Rec.

4 Experiments

4.1 Implementation Details

Training Details. HumanSplat is trained on 500 THuman2.0 [105], 1500 2K2K [109], and 1500 Twindom [106] high-fidelity human scans. We evenly position 36 cameras across each of three hierarchical cycles to capture the full body, half body, and face, with rendering resolution set to 512 × 512 . We conduct 200 epochs of 256 -res training with a learning rate of 1e-5 and a batch size of 32 over 2 days on 8 A100 (40G VRAM) GPUs, while 512 -res finetuning costs 2 additional days. We train our model with AdamW [110] optimizer, whose β 1 , β 2 are set to 0.9 and 0.95 respectively. A weight decay of 0.05 is used on all parameters except those of the LayerNorm layers. We use a cosine learning rate decay scheduler with a 2000 -step linear warm-up and the peak learning rate is set to 4e-4 . For parts related to the head, hands, and arms, λ j are set to 2 , while the rest human part are set to 1 . The parameters λ i , λ p and λ m are set to 1 . The model is trained for 80 Kiterations on 256 -res and then fine-tuned on 512 -res for another 20 K iterations. To enable efficient training and inference, we adopt Flash-Attention-v2 [111] in the xFormers [112] library, gradient checkpointing [113], and FP16 mixed-precision [114]. Fine-tuning the novel view synthesizer takes about 18 hours. Please refer to Appendix A.1 for more details about novel view synthesizer fine-tuning.

Inference Time. The video diffusion model takes about 9 s to generate multi-view latent features while the subsequent latent reconstruction for 3DGS takes only about 0.3 s. Additionally, it can render novel views at a rate exceeding 150 FPS on a NVIDIA A100 GPU.

4.2 Comparison

Quantitative Comparison. We conduct a quantitative comparison on 21 THuman2.0 [105] and 51 Twindom [106] scans, rendering textured meshes from multiple angles and using PSNR, SSIM [115], and VGG-LPIPS [104] as evaluation metrics. To ensure fairness, we fine-tune LGM [29] on our datasets and fully optimize each TeCH [21] instance for comparison. As shown in Tab. 1, HumanSplat consistently outperforms previous generalizable methods across all datasets. Specifically, HumanSplat surpasses SIFU [102] by +1.92 (8.72 % ) in PSNR and reduces LPIPS from 0.079 to 0.055. Notably, HumanSplat also excels on the more challenging Twindom dataset, outperforming TeCH [21] by +2.15 (10.16 % ) in PSNR and reducing LPIPS from 0.188 to 0.125. Although the values can further decrease with model training, we control the convergence to the current extent in the dataset for better generalization under more general in-the-wild scenarios. Besides, we also report the reconstruction times for 512 × 512 input images on a NVIDIA A100 GPU. HumanSplat achieves a remarkably fast reconstruction time, approximately 9.3 seconds on a single NVIDIA A100 GPU, making it much more practical than TeCH's 4.5-hour reconstruction time.

Qualitative Comparison. As illustrated in Fig. 4 and Fig. 5, by incorporating semantics-guided objectives, Humansplat achieves more detailed and higher fidelity results against GTA [30] and

In-the-wild Images

Ground Truth

Ours

TeCH

GTA

LGM*

LGM [29]. Although TeCH [21] also generates high-quality images, its SDS optimization often results in overly saturated multi-face outcomes [116, 117]. Moreover, while TeCH delivers the highest PSNR on the Thuman2.0 dataset, the LPIPS metric in Tab. 1, confirms that our method provides more realistic results. As shown in Fig. 6, Humansplat outperforms HumanSGD on inthe-wild images from Adobe Stock, effectively predicting 3D Gaussian Splatting properties without per-instance optimization. Notably, the results on in-the-wild images further demonstrate our strong generalization capability and our model provides superior textures on both the front and invisible regions, highlighting the effectiveness of incorporating appearance and geometric priors.

4.3 Ablation Study

Latent Reconstruction Transformer. HumanSplat leverages the Stable Diffusion latent space for feature extraction. Experimental results demonstrate that utilizing the compact latent space for reconstructions leads to better rendering quality compared to using the pixel space, as detailed in

InputImage

LGM*

Ours

InputImage

Ours

InputImage

Ours

Ours

Ours

Tab. 2 (a). We also find that patching in the time dimension for latent decoding by temporal V AE [95] adversely affects the quality, so we employ the vanilla V AE [94] decoder in this work.

Human Geometric Prior. Tab. 2 (b) illustrates the significance of human geometric prior in our approach. By utilizing SMPL as the geometric prior for human representation, we assess the robustness of HumanSplat with respect to different SMPL parameters. We report both the estimated [2] and Ground Truth SMPL parameters [92] and compare the model that does not incorporate a human structure prior. These experiments provide insight into the benefit of incorporating a human structure prior, as significant performance degradation is observed without it. Furthermore, thanks to treat human priors as keys and values, HumanSplat exhibits robustness in testing against challenging poses with imperfect estimation, with only minor declines in metrics. The effectiveness of the human geometric prior is also demonstrated qualitatively in Fig. 7 (a).

Window Size. As analyzed in Sec. 3.4.2, human priors are crucial for the robustness of the human reconstruction model. As evidenced in Tab. 2 (c), when the window size K win is set to 1 (dubbed 'Projection Only'), employing human priors significantly enhances performance, improving the PSNR metric from 22.635 to 23.333. We further explore setting window size to K win = 2 in a latent space with a resolution of 64 × 64 , which further improves the model's PSNR metric from 23.333 to 24.294. This improvement is attributed to Humansplat with K win = 2 is capable of handling loose clothing and tolerating imprecise SMPL parameters. In Tab. 3, we present results for the 2K2K dataset across various window sizes. Further refinement with a window size of K win = 3 continues to enhance the results, whereas using K win = 4 does not yield additional improvements.

Semantics-guided Training Objectives. Thanks to the incorporation of semantic-guided hierarchical losses, HumanSplat can generate higher-quality textures for both the face and hand, while also promoting the convergence of the training process, as depicted in Fig. 7 (a) and (b). Furthermore, Fig. 7 (c) demonstrates that HumanSplat further produces 3D coherent segmentation results, which have potential applications in editing and specific generation of individual parts of 3D humans.

20.518/0.873

22.418/0.902

24.633/0.928

0.6

Strategies

W/O Priers

W/O Hierarchical Loss

Ours

0.5

LIPIS Score

0.4

0.3

t°0

0.1

5

10

Iteration Steps (k)

15

20

25

0C

InputImage

W/OPrior

W/OLgc

Our

LPIPSScoreson2K2KValidationSet

3DGaussian

SemanticLabeling

(a)

(q)

(c)

5 Conclusion

We present HumanSplat, a pioneering generalizable human reconstruction network that derives 3D Gaussian Splatting properties from a single image. This model integrates generative diffusion and latent reconstruction Transformer models with human structure priors, enhanced by a tailored semantic-aware hierarchical loss. These innovations achieve high-fidelity reconstruction results in a feed-forward manner without any optimization or fine-tuning, especially in the important focal areas such as the face and hands. Extensive experiments demonstrate that HumanSplat surpasses existing state-of-the-art methods in both quality and efficiency, including robust performance in handling challenging poses and loose clothing. This capability opens up a broad spectrum of potential applications.

Limitations and Future Works. Our method has some limitations that can be addressed in future works: (a). While effective with most attire, the model may struggle with intricate garments and accessories. Future enhancements should incorporate 2D data and innovative training strategies, which is a promising direction for improving the handling of diverse and unusual clothing styles. (b). Although already efficient, increasing the computational speed by constraining Gaussian's number or combining compact 3DGS with texture representation could further facilitate real-time applications, particularly on mobile devices. (c). Animating the reconstructed human models currently requires post-processing. Future work could introduce canonical space to reconstruct animatable avatars directly, streamlining the process and improving efficiency.

References

Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J Black. Collaborative regression of expressive bodies using moderation. In International Conference on 3D Vision (3DV) , 2021.

Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, and Marcus Magnor. Tex2shape: Detailed full human body geometry from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019.

Thiemo Alldieck, Marcus A. Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. Detailed human avatars from monocular video. In International Conference on 3D Vision (3DV) , 2018.

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG) , 2015.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Hao Li, and Angjoo Kanazawa. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019.

Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. PIFuHD: Multi-Level PixelAligned Implicit Function for High-Resolution 3D Human Digitization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu. Photorealistic monocular 3d reconstruction of humans wearing clothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Shoukang Hu, Fangzhou Hong, Liang Pan, Haiyi Mei, Lei Yang, and Ziwei Liu. Sherf: Generalizable human nerf from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Yangyi Huang, Hongwei Yi, Weiyang Liu, Haofan Wang, Boxi Wu, Wenxiao Wang, Binbin Lin, Debing Zhang, and Deng Cai. One-shot implicit animatable avatars with model-based priors. In IEEE Conference on Computer Vision (ICCV) , 2023.

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , 2019.

Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV) , 2020.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG) , 2023.

Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars. arXiv preprint arXiv:2311.08581 , 2023.

Arthur Moreau, Jifei Song, Helisa Dhamo, Richard Shaw, Yiren Zhou, and Eduardo PérezPellitero. Human gaussian splatting: Real-time rendering of animatable avatars. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, and Siyu Tang. 3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Mengtian Li, Shengxiang Yao, Zhifeng Xie, Keyu Chen, and Yu-Gang Jiang. Gaussianbody: Clothed human reconstruction via 3d gaussian splatting. arXiv preprint arXiv:2401.09720 , 2024.

Shoukang Hu and Ziwei Liu. Gauhuman: Articulated gaussian splatting from monocular human videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nie. Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Yangyi Huang, Hongwei Yi, Yuliang Xiu, Tingting Liao, Jiaxiang Tang, Deng Cai, and Justus Thies. TeCH: Text-guided Reconstruction of Lifelike Clothed Humans. In International Conference on 3D Vision (3DV) , 2024.

Badour AlBahar, Shunsuke Saito, Hung-Yu Tseng, Changil Kim, Johannes Kopf, and Jia-Bin Huang. Single-image 3d human digitization with shape-guided diffusion. In SIGGRAPH Asia 2023 Conference Papers , 2023.

Xin Huang, Ruizhi Shao, Qi Zhang, Hongwen Zhang, Ying Feng, Yebin Liu, and Qing Wang. Humannorm: Learning normal diffusion model for high-quality and realistic 3d human generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Xihe Yang, Xingyu Chen, Daiheng Gao, Shaohui Wang, Xiaoguang Han, and Baoyuan Wang. Have-fun: Human avatar reconstruction from few-shot unconstrained images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference on Learning Representations (ICLR) , 2023.

Jiteng Mu, Shen Sang, Nuno Vasconcelos, and Xiaolong Wang. Actorsnerf: Animatable fewshot human rendering with generalizable nerfs. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Zhenzhen Weng, Jingyuan Liu, Hao Tan, Zhan Xu, Yang Zhou, Serena Yeung-Levy, and Jimei Yang. Template-free single-view 3d human digitalization with diffusion-guided lrm. arXiv preprint arXiv:2401.12175 , 2024.

Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. Gps-gaussian: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. European Conference on Computer Vision (ECCV) , 2024.

Zechuan Zhang, Li Sun, Zongxin Yang, Ling Chen, and Yi Yang. Global-correlated 3ddecoupling transformer for clothed avatar reconstruction. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

Muhammed Kocabas, Chun-Hao P. Huang, Otmar Hilliges, and Michael J. Black. PARE: Part attention regressor for 3D human body estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu. PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) , 2023.

Jiefeng Li, Siyuan Bian, Qi Liu, Jiasheng Tang, Fan Wang, and Cewu Lu. NIKI: Neural inverse kinematics with invertible neural networks for 3d human pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Xin Chen, Zhuo Su, Lingbo Yang, Pei Cheng, Lan Xu, Bin Fu, and Gang Yu. Learning variational motion prior for video-based motion capture. arXiv preprint arXiv:2210.15134 , 2022.

Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. VIBE: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Nikos Kolotouros, Georgios Pavlakos, Michael J. Black, and Kostas Daniilidis. Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019.

Thiemo Alldieck, Marcus A. Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. Video based reconstruction of 3D people models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

Thiemo Alldieck, Marcus A. Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from a single RGB camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Hao Zhu, Xinxin Zuo, Sen Wang, Xun Cao, and Ruigang Yang. Detailed human shape estimation from a single image by hierarchical mesh deformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Verica Lazova, Eldar Insafutdinov, and Gerard Pons-Moll. 360-Degree textures of people in clothing from a single image. In International Conference on 3D Vision (3DV) , 2019.

Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. MultiGarment Net: Learning to dress 3D people from images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019.

Boyi Jiang, Juyong Zhang, Yang Hong, Jinhao Luo, Ligang Liu, and Hujun Bao. BCNet: Learning body and cloth shape from a single image. In European Conference on Computer Vision (ECCV) , 2020.

Yao Feng, Weiyang Liu, Timo Bolkart, Jinlong Yang, Marc Pollefeys, and Michael J Black. Learning disentangled avatars with hybrid 3d representations. arXiv preprint arXiv:2309.06441 , 2023.

Sang-Hun Han, Min-Gyu Park, Ju Hong Yoon, Ju-Mi Kang, Young-Jae Park, and Hae-Gon Jeon. High-fidelity 3d human digitization from single 2k resolution images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Muxin Zhang, Qiao Feng, Zhuo Su, Chao Wen, Zhou Xue, and Kun Li. Joint2human: Highquality 3d human generation via compact spherical embedding of 3d joints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Tong He, John P. Collomosse, Hailin Jin, and Stefano Soatto. Geo-PIFu: Geometry and pixel aligned implicit functions for single-view human reconstruction. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Enric Corona, Mihai Zanfir, Thiemo Alldieck, Eduard Gabriel Bazavan, Andrei Zanfir, and Cristian Sminchisescu. Structured 3d features for reconstructing relightable and animatable avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael J. Black. ICON: Implicit Clothed humans Obtained from Normals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction. In IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) , 2021.

Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. ARCH: Animatable Reconstruction of Clothed Humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Tong He, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, and Tony Tung. ARCH++: AnimationReady Clothed Human Reconstruction Revisited. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Yukang Cao, Guanying Chen, Kai Han, Wenqi Yang, and Kwan-Yee K. Wong. JIFF: Jointlyaligned Implicit Face Function for High Quality Single View Clothed Human Reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Tingting Liao, Xiaomei Zhang, Yuliang Xiu, Hongwei Yi, Xudong Liu, Guo-Jun Qi, Yong Zhang, Xuan Wang, Xiangyu Zhu, and Zhen Lei. High-Fidelity Clothed Avatar Reconstruction from a Single Image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Xueting Yang, Yihao Luo, Yuliang Xiu, Wei Wang, Hao Xu, and Zhaoxin Fan. D-if: Uncertainty-aware human digitization via implicit distribution field. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Yukang Cao, Kai Han, and Kwan-Yee K. Wong. Sesdf: Self-evolved signed distance field for implicit 3d clothed human reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Zhuo Su, Lan Xu, Zerong Zheng, Tao Yu, Yebin Liu, and Lu Fang. Robustfusion: Human volumetric capture with data-driven visual cues using a rgbd camera. In European Conference on Computer Vision (ECCV) , 2020.

Zhuo Su, Lan Xu, Dawei Zhong, Zhong Li, Fan Deng, Shuxue Quan, and Lu Fang. Robustfusion: Robust volumetric performance reconstruction under human-object interactions from monocular rgbd stream. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) , 2022.

Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J. Black. ECON: Explicit Clothed humans Optimized via Normal integration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Yuheng Jiang, Kaixin Yao, Zhuo Su, Zhehao Shen, Haimin Luo, and Lan Xu. Instant-nvr: Instant neural volumetric rendering for human-object interactions from monocular rgbd stream. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Xiaozheng Zheng, Chao Wen, Zhuo Su, Zeran Xu, Zhaohu Li, Yang Zhao, and Zhou Xue. Ohta: One-shot hand avatar via data-driven implicit priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Yangyi Huang, Hongwei Yi, Yuliang Xiu, Tingting Liao, Jiaxiang Tang, Deng Cai, and Justus Thies. TeCH: Text-guided Reconstruction of Lifelike Clothed Humans. In International Conference on 3D Vision (3DV) , 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on Machine Learning (ICML) , 2021.

Yuheng Jiang, Zhehao Shen, Penghao Wang, Zhuo Su, Yu Hong, Yingliang Zhang, Jingyi Yu, and Lan Xu. Hifi4g: High-fidelity human performance rendering via compact gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Haokai Pang, Heming Zhu, Adam Kortylewski, Christian Theobalt, and Marc Habermann. Ash: Animatable gaussian splats for efficient and photoreal human rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Zhe Li, Zerong Zheng, Lizhen Wang, and Yebin Liu. Animatable gaussians: Learning posedependent gaussian maps for high-fidelity human avatar modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Xiaozheng Zheng, Chao Wen, Zhaohu Li, Weiyi Zhang, Zhuo Su, Xu Chang, Yang Zhao, Zheng Lv, Xiaoyuan Zhang, Yongjie Zhang, Guidong Wang, and Lan Xu. Headgap: Few-shot 3d head avatar via generalizable gaussian priors, 2024.

Mingwei Li, Jiachen Tao, Zongxin Yang, and Yi Yang. Human101: Training 100+ fps human gaussians in 100s from 1 view. arXiv preprint arXiv:2312.15258 , 2023.

Xinqi Liu, Chenming Wu, Jialun Liu, Xing Liu, Chen Zhao, Haocheng Feng, Errui Ding, and Jingdong Wang. Gva: Reconstructing vivid 3d gaussian avatars from monocular videos. arXiv preprint arXiv:2402.16607 , 2024.

Zhijing Shao, Zhaolong Wang, Zhuang Li, Duotun Wang, Xiangru Lin, Yu Zhang, Mingming Fan, and Zeyu Wang. SplattingAvatar: Realistic Real-Time Human Avatars with MeshEmbedded Gaussian Splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Jing Wen, Xiaoming Zhao, Zhongzheng Ren, Alexander G Schwing, and Shenlong Wang. Gomavatar: Efficient animatable human modeling from monocular video using gaussianson-mesh. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Jiahui Lei, Yufu Wang, Georgios Pavlakos, Lingjie Liu, and Kostas Daniilidis. Gart: Gaussian articulated template models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

David Svitov, Pietro Morerio, Lourdes Agapito, and Alessio Del Bue. Haha: Highly articulated gaussian human avatars with textured mesh prior. arXiv preprint arXiv:2404.01053 , 2024.

Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In International Conference on Learning Representations (ICLR) , 2024.

Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint arXiv:2403.02151 , 2024.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS) , 30, 2017.

Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multiview diffusion for 3d generation. In International Conference on Learning Representations (ICLR) , 2024.

Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201 , 2023.

Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.15110 , 2023.

Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In International Conference on Learning Representations (ICLR) , 2024.

Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. European Conference on Computer Vision (ECCV) , 2024.

Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404.12385 , 2024.

Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191 , 2024.

Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient-based mesh optimization. ACM Transactions on Graphics (TOG) , 2023.

Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint arXiv:2403.14621 , 2024.

Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-lrm: Large reconstruction model for 3d gaussian splatting. In European Conference on Computer Vision (ECCV) , 2024.

Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Hao-Yang Peng, Jia-Peng Zhang, Meng-Hao Guo, Yan-Pei Cao, and Shi-Min Hu. Charactergen: Efficient 3d character generation from single images with multi-view pose canonicalization. ACM Transactions on Graphics (TOG) , 2024.

Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J Black. Collaborative regression of expressive bodies using moderation. In International Conference on 3D Vision (3DV) , 2021.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. European Conference on Computer Vision (ECCV) , 2024.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023.

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. International Conference on Learning Representations (ICLR) , 2024.

Eric Ming Chen, Sidhanth Holalkere, Ruyu Yan, Kai Zhang, and Abe Davis. Ray conditioning: Trading photo-consistency for photo-realism in multi-view image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Julius Plücker. Xvii. on a new geometry of space. Philosophical Transactions of the Royal Society of London , 1865.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR) , 2020.

Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard PonsMoll. Learning to reconstruct people in clothing from a single rgb camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Zechuan Zhang, Zongxin Yang, and Yi Yang. Sifu: Side-view conditioned implicit function for real-world usable clothed human reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Smpl segmentation. https://meshcapade.wiki/assets/SMPL_body_segmentation/ smpl/smpl_vert_segmentation.json .

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Twindom. https://web.twindom.com .

Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019.

Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Sang-Hun Han, Min-Gyu Park, Ju Hong Yoon, Ju-Mi Kang, Young-Jae Park, and Hae-Gon Jeon. High-fidelity 3d human digitization from single 2k resolution images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR) , 2019.

Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 , 2023.

Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling library. https://github.com/facebookresearch/xformers , 2022.

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 , 2016.

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations (ICLR) , 2018.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing (TIP) , 2004.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference on Learning Representations (ICLR) , 2023.

Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR) , 2021.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , 2021.

Qi Zuo, Xiaodong Gu, Lingteng Qiu, Yuan Dong, Zhengyi Zhao, Weihao Yuan, Rui Peng, Siyu Zhu, Zilong Dong, Liefeng Bo, et al. Videomv: Consistent multi-view generation based on large video generative model. arXiv preprint arXiv:2403.12010 , 2024.

Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. arXiv preprint arXiv:2406.09756 , 2024.

Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

A Implementation Details

A.1 Additional Novel View Synthesizer Details

Conditions of Diffusion Model. SVDand SV3D [93, 95] replace CLIP text embeddings [63] with the image embedding of the conditioning. Compared to text prompts, image prompts provide information more accurately and directly, and also save time that would otherwise be spent on text inversion or image caption [21, 102]. SV3D takes an image and the relative camera orbit elevation and azimuth angles as inputs to generate several corresponding novel-view images, which are continuous in 3D space and can be regarded as a video with the camera moving. In our work, we finetuned SV3D on human datasets with relative camera elevation and azimuth angles to generate 4 orthogonal views, which we found to be sufficient for reconstruction [102].

Transformer Block. The CLIP embeddings obtained from the conditioning image is fed into the cross-attention layers of each Transformer block, acting as keys and values, while the feature at that layer serves as the queries. Moreover, the camera trajectory and the diffusion noise timestep are incorporated into the residual blocks.

Fine-tuning Details. We involves the widely used EDM [118, 119] framework, incorporating a simplified diffusion loss for finetuning. We fine-tune the novel-view synthesizer at 512 -res uses 1000 -step warmup with the peak learning rate 1e-4 in the cosine learning rate decay schedule. We use a per-GPU batch size of 8 objects during 256-res training, and a per-GPU batch size of 4 during 512 -res finetuning stage. For each instance, we use 4 input views and 4 novel supervision views at each iteration of 256 -res training and 512 -res finetuning.

Triangular CFG Scaling. C lassifierf ree G uidance ( CFG ) [120] is a widely used technique to trade off controllability with diversity. However, this scaling causes the last few frames in our generated orbits to be over-sharpened. To tackle this issue, we use a triangle wave CFG [93, 121] scaling during inference: linearly increase CFG from 1 at the front view to 2.5 at the back view, then linearly decrease it back to 1 .

3D Consistency Evaluation. We also conduct an evaluation of the 3D consistency of our novel-view synthesizer using the advanced local correspondence matching algorithm, MASt3R [122]. This approach involves one-to-one image matching between input views and their generated novel views, with the average number of matching correspondences serving as our metric. Our synthesizer exhibits a notable improvement in 3D consistency for human novel-view generation after fine-tuning with human datasets. Specifically, the original SV3D achieves an average of 723.25 matching points, while our novel-view synthesizer increases this number to 930.13. In comparison, the ground truth demonstrates a matching point number of 1106.33. This evaluation highlights that fine-tuning on human datasets significantly enhances multi-view consistency metrics.

A.2 Additional Latent Reconstruction Transformer Details

Network Architecture. The network architecture of the proposed latent reconstruction Transformer is shown in Fig. 8. The framework is composed of a latent encoder with intra- and inter-attention and a Gaussian parameter prediction module. It takes initial multi-view latent embeddings as input and performs projection-aware attention to aggregate human geometric information. From each output token, we decode the attributes of pixel-aligned Gaussians in the corresponding patch with a Conv 1 × 1 layer.

Hierarchical Loss. Our approach is efficient and can be utilized online without pre-computation. We calculate the visible face triangles given the mesh and camera parameters. Then each visible triangular face is assigned the corresponding semantic label [103]. Additionally, it provides labels with superior 3D consistency and avoids issues associated with training instability. First, the hierarchical loss is specifically focused on the head and hands, excluding hair and clothing, which can be further supervised using reconstruction loss L Rec. Second, instead of using segmentation to reconstruct and then concatenate different components, we design our loss function based on part segmentation. Thank to this design, HumanSplat is tolerant of inaccuracies in segmentation.

LatentMulti-viewFeatures

3DGSAttributes

Patchify

1x1 Conv

u: position

q: rotation

s: base scale

Linear Embedding

Unpatchify

α: opaciry

↑

C: color

Linear Embedding

Intra-Attention

× Nintra

Deconv

Block

Inter-Attention

个

xNinter

个

B More Results

For practical applications demanding high input view reconstruction quality, which could potentially be directly derived from the input image. We improve the weights of the reconstruction loss for input views (referred to as reweighting loss ), demonstrating that the current model with reweighting loss exhibits sufficient capacity to render input views with higher fidelity, as illustrated in Fig.9. We also conduct an ablation experiment for the reweighting loss function on the 2K2K Dataset [109]. There are slight declines in quantitative metrics, specifically, HumanSplat with reweighting loss achieves a numerical change of -0.141, -0.015, and +0.006 in PNSR, SSIM, and LPIPS. This suggests that reweighting loss is not 'a bag of freebies' and HumanSplat without reweighting loss strikes a balance between the quality of novel views and the input views.

As for novel view and novel pose rendering, more results are shown in Fig. 10 and Fig. 11. For a single input image, we use Open-AnimateAnyone [123] for novel pose synthesis and HumanSplat for novel view synthesis. This functionality enables dynamic timestamps and real-time rendering of novel views, thereby enhancing immersive virtual exploration. Please also kindly refer to our supplementary video for more results.

C Broader Impacts

While our model demonstrates the capability to reconstruct photo-realistic 3D avatars, it also introduces potential risks such as privacy violations. To mitigate these concerns, it is imperative to establish robust ethical guidelines and legal frameworks. This necessitates a collaborative effort among researchers, developers, and policymakers to promote the responsible use of this technology and safeguard against its potential misuse.

Ours

(a) In-the-wild Image

Ours

(b)Image in TwindomDataset

n

n

n

n

n

n

V

★

B

3

n

n

n

(a)

(q)

n

V

n

n

n

SIFU

北

宝

1

北

北

宝

→Novel View Synthesis

北

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Sec. 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the information needed to reproduce the main experimental results of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code will be available at: https://humansplat.github.io .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Sec. 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Sec. 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Sec. 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a 3D human reconstruction system that takes a single RGB image and outputs 3D Gaussians, which can render the reconstructed 3D humans to any viewpoint. In contrast to existing 3DGS works, which requires per-instance optimization and cannot be generalized to unseen identities, the proposed system can be used for any unseen identities in zero-shot manner. To achieve this, the authors utilizes 2D mulit-view diffusion model and latent reconstruction transformer with human structure priors. A hierarchical loss, which incorporates human semantic information, is introduced. Experimental results demonstrate the powerful capability of the proposed work.\n\nSTRENGTHS:\nUtilizing diffusion-based generative models to render unseen novel viewpoints is a reasonable choice considering the aiming task is a significantly ill-posed one. The latent reconstruction Transformer effectively incorporates the human structure prior with the generative model’s output. The hierarchical loss could be useful to enforce a consistency between 3D human geometry and appearance.\n\nWEAKNESSES:\n1. Unclear writings\nOverall, technical details are not enough to fully understand the manuscript. \n1-1. For example, Sec. 3.3 describes the video diffusion model. I can’t fully get how the ‘video’ diffusion model can be used for the ‘novel-view’ synthesizer. I understand that a video of an object can model different viewpoints when a camera is moving, but there should be a clear justification and reason why the authors chose this generative model for the novel-view synthesizer. In addition, it seems the video diffusion model does not take the target camera pose. Then which novel view is modeled with the video diffusion model given no camera pose? \n1-2. Also, for the hierarchical loss, how can the authors access the target human image of certain body parts? Do the authors already have GT body part segmentation? If the authors simply use rendered body part segmentation from GT SMPL meshes, I don’t think that is a good choice as SMPL meshes only model naked bodies, while there could be hair and loose clothes in the image.\n\n2. Lack of comparison to SOTA methods\nPlease compare with more SOTA methods, such as ICON and ECON.\n\n3. Lack of in-the-wild results\nAs the proposed work aims to generalizable 3D human reconstruction, there should be more diverse generalized results. Only three results in Fig. 4 show the results on in-the-wild inputs, which have quite simple poses. Please report more results from input images with diverse **human poses**, identities, and camera pose.\n\nQUESTIONS:\nPlease see the weaknesses section.\n\nLIMITATIONS:\nPlease see the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generalizable human rendering framework from single images. The proposed method relies on different priors and achieves state-of-the-art results on multiple datasets.\n\nSTRENGTHS:\n- The topic of human modeling/rendering from partial observation is important and interesting.\n- Thanks to 3DGS, the proposed method achieves a good balance of rendering quality and speed.\n\nWEAKNESSES:\n- Qualitative visual results on the rendered images are not satisfactory. It is acceptable that renderings of unseen views of the human can be relatively poorer. It is surprising to see renderings of exactly the input views are also in low quality (Figure 4). This is usually a sign of poor model capacity, which may be attributed to the proposed objective functions or the model itself.\n- It is not clear how the 'parts' of human are defined. Are they integer based index as in iuv? If so, how to obtain the ground truth body parts labels? Are they provided by the dataset or they have to be pre-computed by some off-the-shelf models. What if there are prediction errors in the 'ground truth' semantic labels? Is the proposed method robust to this kind of errors? I also have doubts on its effectiveness in terms of the overall rendering quality since there is no ablation study provided to show how it improves quantitative/qualitative results.\n- As presented in Table 2 (b), it seems that the proposed method is very sensitive to geometric prior (i.e. SMPL estimations). Since the model relies on PIXIE for SMPL prediction, what if there are errors in the predictions (most likely for challenging poses or complex clothes/background), how will the proposed method perform under those cases? This on the contrary shows advantages of geometric prior free methods such as LRM and HumanLRM. If there is a benchmark on human rendering with challenging poses, I am afraid those methods will perform better than the proposed one.\n- I wonder why there are not qualitative comparisons with SIFU but only quantitative metrics?\n- I'd suggest to change Figure 1 (b) to a different style. The dashes in the graph are misleading and confuse readers as they are in different scales.\n\nQUESTIONS:\nPlease see the weakness section.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

comments: SUMMARY:
This paper proposes HumanSpat, a method that predicts 3D Gaussians from a single image of a human. The method comprises a 2D multi-view diffusion model and a latent reconstruction transformer that integrates human body prior.

STRENGTHS:
+ Well-designed generalizable model that incorporates human prior. The model design is intuitive and clear. The fusion of the human structure prior makes a lot of sense. The end-to-end design addresses major limitations in existing works, such as lacking of human prior makes the prediction missing limbs (as in Human lrm) and two stage approaches where the errors in the human pose prediction affects the clothed reconstruction.

+ Reconstruction time. Powered by 3DGS, the method has minimal reconstruction time and can render novel views at real-time speed.

WEAKNESSES:
- Missing ablation on human prior. The paper should include ablation experiments where there is no human structure prior, as well as experiments where the human structure prior is estimated instead of GT. These experiments can provide insight on the benefit of the specific human structure prior fusion design.

QUESTIONS:
In Table 1, how was the reconstruction time computed? PIFu is very lightweight and should not take 30 seconds per human.

LIMITATIONS:
Limitations are addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper, HumanSplat, focuses on photorealistic novel-view synthesis of humans from a single image.\n- The key idea is to use a multi-view synthesizer based on SV3D to hallucinate the other views + latent reconstruction transformer to predict the 3DGS.\n- Importantly, HumanSplat does not use optimization and is much faster than existing methods.\n- Baselines:  Magic, HumanSGD, TeCH\n- Evaluations are done on Thuman2.0 and Twindom datasets. Metrics: PSNR, SSIM, LPIPS\n- The proposed methods consistently outperforms the existing methods.\n\nSTRENGTHS:\n- The paper is well-written, organized and easy to follow.\n- The key idea of multi-view diffusion plus transformer to predict parameters of gaussians in the context of human avatar creation is novel.\n- An important technical contribution is the archictecture of the latent reconstruction transformer with geometry-aware interaction.\n- The experiments are done on multiple datasets along with informative ablative studies.\n- Table. 1 highlights the speed-up gains compared to existing methods. HumanSplat is almost as fast as LRM but utilizes an explict human prior and achieves much better performance.\n\nWEAKNESSES:\n- Measuring 3D consistency of SV3D: The novel views generated by fine-tuned SV3D is important for predicting the gaussian parameters. An ablative study on the effectiveness of the novel synthesizer would be really helpful to understand the in-the-wild generalization of the proposed method. 3D consistency could be measure using stereo-based matching or local correspondences.\n\n- Qualitative results: Although, Table. 1 showcases better performance than HumanSGD on THuman2.0 dataset, notably in all metrics. The qualitative comparison between HumanSGD and HumanSplat is missing. HumanSGD results demonstrate much better modelling of human skin tone and overall 3D geometry. I understand the code is yet to be released, however, would it be possible to compare on similar stock images?\n\n- Minor: consider renaming \"project-aware\" -> \"projection-aware\". The writing at times is a little too non-technical, eg. L206-207, consider rephrasing such instances.\n\nQUESTIONS:\n- Understanding the generalizability of novel-view synthesizer. Is it limited to full-body images? Any quantitative insights into 3D consistency.\n- Qualitative comparisons to HumanSGD on stock images.\n\nLIMITATIONS:\nYes. The authors provide a discussion about utilizing 2D data and handling diverse clothing styles.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}