Generalizable Human Gaussians from Single-View Image

## Abstract

Abstract In this work, we tackle the task of learning 3D human Gaussians from a single image, focusing on recovering detailed appearance and geometry including unobserved regions. We introduce a single-view generalizable Human Gaussian Model (HGM), which employs a novel generate-then-refine pipeline with the guidance from human body prior and diffusion prior. Our approach uses a ControlNet to refine rendered back-view images from coarse predicted human Gaussians, then uses the refined image along with the input image to reconstruct refined human Gaussians. To mitigate the potential generation of unrealistic human poses and shapes, we incorporate human priors from the SMPL-X model as a dual branch, propagating image features from the SMPL-X volume to the image Gaussians using sparse convolution and attention mechanisms. Given that the initial SMPL-X estimation might be inaccurate, we gradually refine it with our HGM model. We validate our approach on several publicly available datasets. Our method surpasses previous methods in both novel view synthesis and surface reconstruction. Our approach also exhibits strong generalization for cross-dataset evaluation and in-the-wild images.

## 1 Introduction

Automatic 3D human reconstruction from single image is crucial in augmented and virtual reality (AR/VR), game industry, filmmaking, etc. Previous works rely on strong 3D supervision such as the signed distance value or occupancy ; ; ; ; ; ; and focus on surface reconstruction, neglecting novel view synthesis quality, resulting in smoothed and blurred textures. With the development of neural radiance fields , novel view rendering quality has been greatly improved with for human appearance modelling ; ; . However, due to the ill-posed nature of single view reconstruction, the back and side views are always blurry and lack details without additional prior. Furthermore, these methods needs large amounts of query points sampled for volume rendering, which hinders practical real-time application in the industries. Some other methods optimize underlying appearance and geometry from scratch by introducing score distillation sampling during the optimization; . Although effective, these methods still suffer from long time optimization and over-saturation problems.
Recent 3D Gaussians generation methods ; combine multi-view diffusion models ; ; ; ; with generalizable multi-view Gaussians prediction models to generate 3D Gaussians with high quality and efficiency. We aim to extend this on human reconstruction. However, directly employ such methods to human reconstruction with complex texture and poses

Figure: Figure 1: Our method reconstructs detailed and geometrically consistent human Gaussian models from single view images, including loosing clothes, challenging pose and in-the-wild images.
Refer to caption: x1.png

gives unsatisfying results due to:
(1) Inconsistency across multiple views:
The multi-view images generated from diffusion model lacks consistency in appearance and pose across different viewpoints. This inconsistency stems from the inherent complexity of human body structure and movement, which leads to low-quality reconstruction results.
(2) Quality loss in front view reconstruction: multi-view diffusion process involves down-sampling and changing the original input image. This step results in significant quality degradation when reconstructing the front view image, compromising the fidelity to the original input.
3) Estimating SMPL-X parameters from single view input is ill-posed, directly applying initially estimated SMPL-X can lead to bending legs and wrong elevation issues in previous method; ; ; .

To address the above-mentioned problems, we introduce a novel Human Gaussians Model (HGM), which supports fast and high quality rendering from single view input, and generalize well to loosing clothes, challenging poses and in-the-wild images as shown in Fig. 1.
We do not use multi-view diffusion models due to the multiview inconsistency and resolution degradation problem. Instead, we propose a coarse-to-fine framework, where the diffusion model is adapted to refine back-view images rendered from our predicted coarse human Gaussians. In this way, we can keep the resolution and content of the original input image for high-fidelity reconstruction.
In order to model the complex structure of a human, we inject the human prior into the Gaussian prediction process. Specifically, our model consists of two branches: 1) The first branch is a UNet to directly predict Gaussians from the input image, as inspired by image splatter . 2) The second branch uses learnable tokens attached to SMPL-X vertices for structural feature extraction with attention layers, and then combined with UNet features with SparseConv and a transformer for Gaussian enhancement.
Recognizing the inaccurate estimate of SMPL-X from the pre-trained model, during inference, we iteratively refine the initial SMPL-X parameters with our HGM pre-trained with ground truth SMPL-X.
Given the loss of details of the back view by directly predicting the Gaussians from a single view, we further apply a ControlNet to refine the back view with the control signal from the back-view image rendered from the coarse stage.
We then input the original front view and refined back view images to our HGM model to get the final refined Gaussians. Meshes can be extracted from densely rendered depth map and TSDF fusion. Our model can be trained with only posed multiview images without 3D supervision and generalizes well to untrained datasets and in the wild images.

In summary, our contributions are:

- •
We introduce a generate-then-refine pipeline for single view human Gaussian reconstruction that leverages diffusion priors for back view refinement, avoiding the multi-view inconsistencies commonly observed in multiview diffusion models.
- •
Our proposed dual-branch reconstruction pipeline incorporates human priors by attaching learnable tokens to the SMPL-X vertices for structural feature extraction. We then fuse these features from the SMPL-X branch with the U-Net branch using Sparse Convolution and transformer.
- •
To address potential inaccuracies in initial SMPL-X estimations, we employ our Human Gaussian Model (HGM) to iteratively refine the estimated SMPL-X parameters, resulting in better alignment.
- •
Through extensive experimentation, we demonstrate the efficacy of our method in both novel view synthesis and 3D reconstruction tasks. Our approach consistently achieves state-of-the-art performance on various metrics and benchmarks.

## 2 Related works

Single-view Human Reconstruction.
PIFu , PIFuHD , PaMIR , and GTA are capable of inferring full textures from a single image. Techniques such as PHORHUM and S3F go further by separating albedo and global illumination. However, these methods lack information from other views or prior knowledge, such as diffusion models, often resulting in unsatisfactory textures. TeCH utilizes diffusion-based models to visualize unseen areas, producing realistic results. However, it requires time-intensive optimization per subject and is dependent on accurate SMPL-X. The emergence of Neural Radiance Fields (NeRF) has led to methods ; ; ; using videos or multi-view images to optimize NeRF for the capture of human forms. Recent advancements like SHERF and ELICIT aim to generate human NeRFs from single images. Although NeRF-based approaches are effective in creating high-quality images from various perspectives, they often struggle with detailed 3D mesh generation from single images and require extensive optimization time. More recently, SiTH proposes to combine a back-view hallucination model with an SDF-based mesh reconstruction model. Similarly, SIFU employs a text-to-image diffusion-based prior to generating consistent textures for invisible views. However, these methods require 3D annotations such as the SDF of the meshes and texture maps as strong supervision and still fail to generate renderings with high fidelity due to the limited 3D training data and representation. In addition, these methods suffer from SMPL estimation errors, leading to bending legs and wrong elevation of the reconstructed 3D humans. Compared to these methods, our approach can be trained solely on multi-view images and achieves much better novel view synthesis quality.

Human Gaussians.
3D Gaussians and differentiable splatting have gained broad popularity due to their efficiency in reconstructing high-fidelity 3D scenes from posed images using only a moderate number of 3D Gaussians. This representation has been quickly adopted for various applications, including imag or text-conditioned 3D generation and avatar reconstruction. Among these methods, Gauhuamn and HUGS ; are the first to propose optimizing human Guassians from monocular human videos. However, they are not applicable to single static human images. GPS-Gaussian propose a generalizable multi-view huaman Gaussian model with high quality rendering; however, it needs dense views 16 or 8, which cannot be directly applied to single-view human images. In contrast, our human model is able to generate human Gaussians from single view image with strong generalization.

Generalizable Gaussians with Multi-view Diffusion.
The Large Reconstruction Model (LRM) scales up both the model and the dataset to predict a neural radiance field (NeRF) from single-view images. Although LRM is primarily a reconstruction model, it can be combined with Diffusion Models (DMs) to achieve text-to-3D and image-to-3D generation as demonstrated by extensions such as Zero123, Image Dream Instant3D and DMV3D . Our method also builds on a strong reconstruction model and uses pre-trained 2D DMs to provide input images missing information in a feedforward manner.
Some concurrent works, such as LGM , AGG , and Splatter Image , also utilize 3D Gaussians in a feed-forward model. LGM combines novel view generation diffusion models with generalizable Gaussians in a feedforward manner, while GRM replaces the U-Net architecture with a pure-transformer one and scales up to large resolution. However, these methods face two main challenges when using pre-trained diffusion models. Firstly, the generated input view image becomes blurry compared to the original input, which affects the subsequent generalizable Gaussian model. Secondly, diffusion models can introduce multiview inconsistency, especially for human images with different poses, making direct adaptation unfeasible. We solve these problems by using ControlNet as the refinement tools without damaging the input image quality or introducing multi-view inconsistency.

## 3 Our Method

### 3.1 Preliminaries

3D Gaussian Splatting (3DGS).
Introduced by , 3D Gaussian splatting represents 3D assets or scenes using a collection of 3D Gaussians. Each Gaussian is characterized by its center $x\in\mathbb{R}^{3}$, scaling factor $s\in\mathbb{R}^{3}$, rotation $r\in\mathbb{R}^{3}$, opacity $\alpha\in\mathbb{R}$, and color features $c\in\mathbb{R}^{c}$. View-dependent effects can be modeled with spherical harmonics.
3D scenes can be explicitly represented by a set of Gaussians $G=\{G_{i}\}$, where $G_{i}=\{{x_{i},s_{i},r_{i},\alpha_{i},c_{i}}\}$ represents the attributes for the $i$-th Gaussian.
Compared with NeRF , 3DGS performs fast rendering by first projecting Gaussians onto the image plane as 2D Gaussians and performing alpha-blending for each pixel in front-to-back depth order. Building on this, Image Splatter proposes predicting Gaussians from a single image through image-to-image translation. Specifically, each pixel is converted to a Gaussian with corresponding attributes, supervised by multi-view images. Our model builds on this representation by directly predicting XYZ coordinates from the image instead of the depth.

### 3.2 Overview

Fig. 2 shows an overview of our framework. Given a single input human image $I$, our aim is to predict the corresponding human Gaussians, which can be further rendered for novel view synthesis and mesh extraction. As shown in the upper part of Fig. 2, our proposed method consists of three parts: 1) Coarse Gaussians prediction with SMPL-X refinement (cf. Sec. 3.3)
2) Back-view refinement with ControNet(cf. Sec. 3.4).
3) Two-view reconstruction (cf. Sec. 3.5).

Figure: Figure 2: Our framework and HGM model. (Top) Our framework consists of three steps: 1) Coarse Gaussians prediction with iterative SMPL-X refinement. 2) Back view refinement with ControlNet. 3) Two view reconstruction to get the refined $G_{refine}$. (Bottom) Our HGM model consists of two branches: Image Gaussians prediction by $\operatorname{UNet}$ and adding additional structural features extracted from SMPL-X branch. $f_{smpl}$ are sampled by the Gaussian centers from the SMPL-X volume $S^{vol}$ and fused with $f_{u}$ to the fusion transformer $\operatorname{Tr}_{mix}$ to obtain the Gaussian output.
Refer to caption: x2.png

### 3.3 Coarse Gaussians Prediction with SMPL-X refinement

#### 3.3.1 Our HGM model

The lower part of Fig. 2 shows our proposed Human Gaussian Model (HGM). The direct prediction of Gaussians from image pixels with UNet ; lacks human shape and pose prior, thus leading to unsatisfactory results. We therefore introduce a dual branch that utilizes SMPL-X to enforce human shape and pose prior into the Gaussian prediction process.
Specifically, for UNet branch, the collection of the RGB value and ray embedding for each pixel are concatenated into a 9-channel feature map as the input $F_{I}=\{c_{i},o_{i}\times d_{i},d_{i}\mid i=1,2,...,N\}$. Our HGM model predicts Gaussians from the U-Net as:

$$ $\displaystyle G_{u}=\operatorname{UNet}(F_{I}),$ (1) $$

We call it as Image Gaussians. For the SMPL-X branch, we attach learnable tokens to each of the SMPL-X vertices and extract the patch features of the image, denoted $F^{\prime}_{I}$. We use cross-attention between these learnable tokens and the patch features to obtain $F_{S}$. This approach takes advantage of the fact that SMPL-X vertices are defined in semantically similar areas across different identities. Consequently, the learned tokens can memorize the mapping from the training dataset to unseen identities during inference, effectively providing structural human priors. This mechanism enables our model to capture and utilize semantic consistent features across diverse identities, enhancing its ability to generalize to new subjects.
We apply SparseConvNet (SPConv $\Psi$) to propagate the SMPL-X features to the predefined whole bounding box, and we denote this feature as the SMPL-X volume feature:
$S^{vol}=\Psi(F_{S})$, where $F_{S}$ are the SMPL-X features.
The volume feature reconstructed from the SMPL-X vertex feature provides geometric cues of the target human body. The centers of the Image Gaussians from $G_{u}$ are then used to sample the propagated SMPL-X volume features, denoted as $f_{s}=S^{vol}(C_{u})$, where $C_{u}$
are the centers of $G_{u}$. SMPL-X $f_{s}$ features are then concatenated with the features of UNet $f_{u}$ for each Gaussian. This concatenated feature is fed into a transformer $\operatorname{Tr}_{mix}$ to obtain the coarse Gaussians shown in 2. Specifically, we predict the xyz coordinates residuals for the Image Gaussians and all the other updated Gaussian features.
$\operatorname{Tr}_{mix}$ is a transformer that contains multiple self-attention blocks among Gaussians to ensure that each Gaussian is aware of the other.

$$ $\displaystyle G_{coarse}=\operatorname{Tr}_{mix}(\left[f_{u},f_{s}\right]).$ (2) $$

Figure: Figure 3: Left: Our SMPL-X refinement pipeline. Right: Our back-view refinement ControlNet.
Refer to caption: x3.png

#### 3.3.2 SMPL-X Refinement

Given initial estimated SMPL-X is not accurate, we leverage our pre-trained HGM to iteratively refine the SMPL-X parameters based on the mask and normal matching as shown in Fig. 3 left part. Specifically, we use the initial estimated SMPL-X from pre-trained SMPL-X estimator to reconstruct the initial Gaussians and rendered the side-view masks and normals. We minimize the mask and normal difference between the Gaussian and SMPL-X renderings and back-propagate the loss to SMPL-X parameters. Then we interatively input the updated SMPL-X to our HGM model, so the Gaussian is also updated to give more accurate masks. For normal matching, we use pre-trained normal estimator from which also needs SMPL-X as input and iteratively update the SMPL-X parameters. Specifically, we compute the $\mathcal{L}_{SMPL-X}$ as shown in 3 and back propagate it to SMPL-X parameters. $\mathcal{L}_{normal}$, $\mathcal{L}_{front}$ and $\mathcal{L}_{side}$ are $\mathcal{L}_{1}$ losses between the rendered masks of SMPL-X model and coarse Gaussians by HGM. $\mathcal{L}_{normal}$ is the $\mathcal{L}_{1}$ loss between the rendered SMPLX normal and the pre-trained normal estimator . Note that both of the normal and side mask supervision are updated as the input of the HGM and normal estimator also contains the updated SMPL-X in each iteration. The whole process takes 100 iterations. We provide more analysis on our SMPL-X refinement in the appendix.

$$ $\displaystyle\mathcal{L}_{SMPL-X}=\lambda_{front}\mathcal{L}_{m_{front}}+ \lambda_{side}\mathcal{L}_{m_{side}}+\lambda_{n}\mathcal{L}_{normal}$ (3) $$

Figure: Figure 4: Our back-view refinement can generate more realistic back-view images, compared with back-view hallucination of SiTH .
Refer to caption: x4.png

### 3.4 Back-view Refinement

Back-view hallucination poses a significant challenge in single-view human reconstruction. As shown in Fig. 4, directly using diffusion models to generate a back-view image can result in incorrect perspective projection with the front-view image as well as unrealistic texture by .
The reason is that their diffusion is conditioned on the back-view mask, and thus can only be applied for orthogonal projection where back-view mask can be directly flipped with the front-view image.
However, the back-view mask is not available during our inference stage since our prediction is based on perspective projection. To address this issue, we adopt a generate-then-refine strategy that leverages the diffusion prior to produce a perspective-fitted and realistic back-view image that is suitable for the subsequent two-view fusion stage. We train a ControlNet to enhances realistic details based on our coarse results as shown in Fig. 3 right part. Specifically, we generate coarse back-view rendering by our HGM for the training dataset and only train the ControlNet and keep the base Stable Diffusion model as fixed.

$$ $\displaystyle\mathcal{L}_{CN}=\mathbb{E}_{\mathbf{z}_{0},t,\mathbf{c}_{t}, \boldsymbol{\epsilon}_{t}\sim\mathcal{N}(0,1)}\left[\left\|\boldsymbol{ \epsilon}-\epsilon_{\theta}(\mathbf{z}_{t},t,\mathbf{c}_{t},\mathbf{y})\right \|_{2}^{2}\right]$ (4) $$

where $y$ is the text prompt. We set it as ‘Best quality’ and the negtive prompt as ‘blur, bad anatomy, bad hands, cropped, worst quality’ during inference, $\mathbf{c}_{t}$ is the coarse back-view image from our HGM rendering $I_{B}^{c}$ , which is the ControlNet condition. We carefully design the reversing process by adding small amout of noise to the VAE encoded latent of $I_{B}^{c}$ to keep the original content as much as possible. The sampling process takes around 2 seconds. In Fig. 4, we showcase our generated and refined back-view results, comparing them with the back-view hallucination diffusion network in SiTH . Our results maintain high resolution and generate details such as the hair for the first woman and the wrinkles in the clothes, whereas SiTH produces artifacts and unrealistic hallucination results and are also not perspectively fitted to the input image.

### 3.5 Two-view Reconstruction

We combine the refined perspective-fitted back-view image $I_{B}^{r}$ with the front-view image $I$ and input them into the fusion HGM model to get:

$$ $\displaystyle G_{refine}=\operatorname{HGM(I,I_{B}^{r})}.$ (5) $$

Specifically, our fusion HGM model retains the design of the coarse HGM model with the additional refined back-view image as the input. The coarse HGM and fusion HGM models are trained separately with ground-truth one view and two views as input, as well as ground-truth SMPL-X.
The objective function for HGM training includes $\mathcal{L}_{2}$ color loss, $\mathcal{L}_{rgb}$, VGG-based LPIPS perceptual loss, $\mathcal{L}_{lpips}$ , and $\mathcal{L}_{2}$ background mask loss $\mathcal{L}_{bg}$ with ground truth masks. Each of these losses has corresponding weights that are treated as hyperparameters:

$$ $\displaystyle\mathcal{L}_{HGM}=\lambda_{rgb}\mathcal{L}_{rgb}+\lambda_{lpips} \mathcal{L}_{lpips}+\lambda_{bg}\mathcal{L}_{bg},$ (6) $$

where $\lambda_{rgb}=\lambda_{lpips}=\lambda_{bg}$=1.0. $\mathcal{L}_{HGM}$ is applied for coarse HGM and fusion HGM.

### 3.6 Implementation

Figure: Figure 5: Levraging our HGM model, SMPL-X parameters are iteratively refined to mitigate the issue of blended legs commonly seen in other approaches.
Refer to caption: x5.png

Our model is trained on 4 NVIDIA RTX A6000 with batch size of 4 for 20 hours. Our input image size is 512×512, and the number of Gaussians for each view is 256×256, totaling 65,536 Gaussians per view.
For SMPL-X estimation, we use PIXIE. Network structures and more implementation details are in the appendix.

## 4 Experiments

We conduct experiments on the publicly available 3D human datasets THuman2.0 , CustomHumans and HuMMan . Our method is compared with state-of-the-art (SOTA) methods in both novel view synthesis and 3D mesh reconstruction. We train our HGM on 500 human scans from THuman2.0 dataset following . We render the images with resolution of 512×512 and using weak perspective camera on 12 fixed cameras evenly distributed with the azimuths from 0 to 360 degree. During evaluation, all the methods are tested without the ground truth SMPL-X. We follow the train and test list from SIFU and SHERF to evaluate our method on THuman2.0 and HuMMan dataset. For CustomHumans dataset we use 45 scans for cross-dataset evaluation containing loosing clothes and challenging poses. For novel view synthesis, We use PSNR, SSIM, LPIPS as evaluation metrics. For 3D reconstruction, we use commonly used Chamfer Distance (CD), Point-to-Surface Distance (P2S), and Normal Consistency as the evaluation metrics.

### 4.1 Novel View Synthesis

**Table 1: Novel view synthesis comparison with SOTA methods.**
|  |  | THuman2.0 |  |  | CustomHumans |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| SiTH | 17.12 | 0.843 | 0.155 | 18.09 | 0.856 | 0.144 |
| LGM | 18.34 | 0.856 | 0.134 | 19.87 | 0.877 | 0.132 |
| SV3D | 19.11 | 0.892 | 0.117 | 20.86 | 0.902 | 0.112 |
| SIFU | 22.10 | 0.924 | 0.0794 | 20.83 | 0.898 | 0.117 |
| Ours | 23.54 | 0.938 | 0.0524 | 23.84 | 0.944 | 0.0514 |

For novel view synthesis, we compare our method with mesh SOTA human reconstruction methods ECON , SIFU and SiTH , as well as multiview diffusion reconstruction method LGM(finetuned with the same training data) and video diffusion method SV3D on THuman2.0 and CustomHuman .
We also compare our method with state-of-the-art HumanNeRF methods: SHERF , MPS-NeRF , and NHP on the HuMMan dataset .

As shown in the Tab. 1 and Tab. 2, our method significantly surpasses state-of-the-art single-view human reconstruction methods in all evaluation metrics for the three datasets.
As shown in Fig. 6, LGM generates incorrect blue color and inconsistent content. SiTH fails to model loose clothes due to the high dependency of the SMPL-X model. Side views are blurry and unrealistic in SIFU’s results. SV3D generates strange colors and wrong human pose. Compared with these methods, ours generates more realistic and consistent rendering especially for the unseen regions such as clothes wrinkles and hair that are well-fitted to the front views and more robust to initial SMPL-X estimation errors thanks to our iterative refinement. We provide rendering videos in 360 degree comparison with other methods in the Appendix. We also provide a visual comparison with the SOTA NeRF-based method SHERF on the HuMMan dataset in the appendix.

Figure: Figure 6: Novel view synthesis comparison with other approaches on THuman2.0 and CustomHumans dataset. The details are highlighted in the red boxes.
Refer to caption: x6.png

**Table 2: Novel view synthesis comparison with SOTA HumanNeRF methods on HuMMan.**
| Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| --- | --- | --- | --- |
| NHP | 18.99 | 0.845 | 0.182 |
| MPS-NERF | 17.44 | 0.824 | 0.193 |
| SHERF | 20.83 | 0.891 | 0.125 |
| Ours | 23.86 | 0.952 | 0.0591 |

### 4.2 3D reconstruction

For mesh reconstruction we extract the 3D mesh by densely rendering the depth map with Gaussian render and using TSDFusion to extract the surface followed by a fast optimizaiton based on the normal map obtained in section3.3.2. We compare our results with SOTA human surface reconstruction methods ECON , SIFU  and SiTH . Note that our method do not use the 3D ground truth for supervision, but can also achieve best performance compared with all those fully supervised methods. Thanks to our interative SMPL-X refinement. Our method alleviates commonly occurring problems of bent legs and incorrect postures found in previous methods, as shown in Fig.5. Our HGM model reconstructs more accurate geometry with the prior learned from our dual branch Gaussian reconstruction model as well as our innovative SMPL-X refinement.
We provide a qualitative 3D reconstruction comparison in Fig.7. As shown in the figure, ECON and SIFU suffer from bending legs and wrong arms problems. SiTH generates an over-smoothed surface and missing parts. Our method can reconstruct more accurate poses while preserving geometric details.

Table: Table 3: 3D reconstruction comparison with SOTA methods.

Figure: Figure 7: 3D reconstruction visualization compared with SOTA methods. Details are highlighted in the red boxes.
Refer to caption: x7.png

### 4.3 Ablation Studies

We conduct ablation studies to evaluate the effectiveness of our SMPL-X dual branch Gaussian prediction model,
coarse-to-fine refinement strategy, and
back-view refine ControlNet, as well as our SMPL-X refinement. We show the quantitative ablation results in Table 4. The performance decrease when any component is removed. SMPL-X dual branch plays an important role in adding human priors through structural features to Image Gaussians predicted by UNet. We visualize the rendered images using our model without SMPL-X dual branch as the guidance and those produced by our full model, as shown in Fig. 8. Without SMPL-X dual branch as guidance, the side view images exhibit significant artifacts, such as misaligned arms and unnatural shapes of clothes and heads, highlighted in the red boxes. This demonstrates the effectiveness of our SMPL-X dual branch Gaussian prediction design.The predicted Gaussians lack human shape and pose priors without SMPL-X guidance, resulting in unnatural shapes and poses.
The initial SMPL-X prediciton is not accurate, we ablates the effectiveness of our iterative SMPL-X refinement with the 3D prior learned in our HGM model. We also visualize the SMPL-X refinement in the appendix.
Two-view refinement double the number of the Gaussians to improve the reconstruction quality. Gaussians tend to concentrate more on the front view without the two-view refinement strategy, leading to poorer rendering of the back part.
Additionally, the diffusion-based refinement is crucial for improving the novel view synthesis quality, especially for the back-view images as shown in Fig. 4.
The best performance is achieved with all three components.

### 4.4 Discussions

Our method improves upon previous approaches like ECON and SIFU by leveraging our pre-trained HGM model to incorporate 3D priors, specifically side view masks, for enhanced SMPL-X refinement. Unlike earlier techniques that only align 2D front-view masks and normals, our approach achieves better reconstruction accuracy and alignment. The refined SMPL-X also benefits our human Gaussians reconstruction by providing structural information encoded in learnable tokens. Our 3D Gaussians-based method offers significant advantages in rendering speed, achieving 300 FPS compared to NeRF-based methods like SHERF , which manages only 2 FPS. Furthermore, the mesh extracted from our 3D Gaussians with normal refinement attains high 3D reconstruction quality. In summary, our method excels in both high-quality rendering and accurate 3D reconstruction, offering a comprehensive solution.

**Table 4: Ablation study for each component.**
|  |  | NVS |  |  | 3D reconstruction |  |
| --- | --- | --- | --- | --- | --- | --- |
| Components | PSNR($\uparrow$) | SSIM($\uparrow$) | LPIPS($\downarrow$) | Chamfer($\downarrow$) | P2S($\downarrow$) | NC($\uparrow$) |
| w/o SMPL-X dual branch | 21.85 | 0.908 | 0769 | 2.421 | 2.543 | 0.776 |
| w/o SMPL-X refine | 22.86 | 0.921 | 0.656 | 2.245 | 2.346 | 0.798 |
| w/o two-view refine | 22.95 | 0.924 | 0.641 | 2.301 | 2.343 | 0.781 |
| w/o Diffusion refine | 23.11 | 0.921 | 0.637 | 2.145 | 2.176 | 0.812 |
| Full model | 23.54 | 0.938 | 0.524 | 2.134 | 2.118 | 0.823 |

Figure: Figure 8: Ablation studies in terms of SMPL-X dual branch guidance. For each pair of images, left one is the results of our model w/o SMPL-X guidance. Details are highlighted in the red boxes.
Refer to caption: x8.png

## 5 Conclusion

In this paper, we introduce a novel generalizable single-view human Gaussian reconstruction framework. By incorporating human priors through a SMPL-X dual branch Gaussian prediction and diffusion priors using a refinement ControlNet, our method effectively handles invisible parts and varying poses. By incorporating our pre-trained HGM, inaccurate SMPL-X is iteratively refined, which benefits the Gaussian reconstruction quality. Combining all of these techniques, our method can generalize well to unseen subjects for high-quality and view-consistent reconstruction. We validate the proposed method on several benchmarks and demonstrate that it achieves state-of-the-art performance in both novel view synthesis and 3D reconstruction.

## Appendix A Appendix

We introduce the following content in the appendix: SMPL-X optimization and mesh optimization details, experimental environment, network structures, limitations, and more visualizations.

##### SMPL-X optimization and Mesh optimization

During the optimization, we render the side views and compute the side view mask loss every 10 iterations among the total 100 iterations. We show the visualization of our side view mask rendered from iteratively reconstructed Gaussisnas by HGM, initial SMPL-X, refined SMPL-X, and our finial-extracted meshes in Fig. 10, as shown in the figure, the side view masks effectively help refine the initial SMPL-X error for accurate reconstruction. For mesh refinement, we minimize the $\mathcal{L}_{1}$ loss between the predicted normal map and the rendered normal map. We also add Laplacian loss for the preservation of the local structure. We optimize 200 iterations in total. The whole Gaussian reconstruction takes around 40s and the mesh extraction takes another 30s.

##### Experimental environment

We conduct all the experiments on NVIDIA RTX A6000 GPU. The experimental environment is PyTorch 2.2.1 and CUDA 12.2.

##### Network structures

Our UNet model consists of 6 down blocks, 1 middle block and 5 up blocks, with an input image size of 512×512 and an output Gaussian feature map size of 256×256. We use 2 input views, resulting in a total of 256×256×4 = 131,072 output Gaussians. Each block contains several residual layers and an optional down-sample or up-sample layer. For the last 3 down blocks, the middle block, and the first 3 up blocks, we insert cross-view self-attention layers after the residual layers. The final feature maps are processed by a 1×1 convolution layer to produce 14-channel pixel-wise Gaussian features. We adopt SiLU activation and group normalization for the UNet.
Our $\operatorname{Tr}_{mix}$ share the same structure, consisting of multiple self-attention blocks. Specifically, they each have 5 up blocks and 5 down blocks. The down-sample channels are [64, 128, 256, 512, 1024], and the up-sample channels are [1024, 512, 256, 128, 64]. The input dimensions for $\operatorname{Tr}_{mix}$ is 256, respectively. The output dimension for both is 14, which matches the dimension of the Gaussian features. For the attention blocks, we use a memory-efficient attention implementation.

Figure: Figure 9: Novel view synthesis comparison SHERF on HuMMan dataset.
Refer to caption: x9.png

##### Limitations

Currently, our method is hard to generate high-quality hands and faces, which could potentially be solved by utilizing the SMPL-X model and regional diffusion guidance such as SDS loss for further refinement.

Figure: Figure 10: SMPL-X refinment visualizaiton.
Refer to caption: x10.png

##### Additional results

We provide visual comparison with the SOTA NeRF-based method: SHERF [] shown in 9. While SHERF predicts blurry results and loses fidelity, our method preserves high-frequency details and generates realistic back views such as wrinkles and hair that fit well to the front views.
We also provide more mesh reconstruction results in Fig. 11.

Figure: Figure 11: Mesh reconstruction visualization for THuman 2.1 data and in the wild images.
Refer to caption: x11.png

We also provide more visual comparison with other methods for loosing clothes in Fig. 12 and Fig. 13 and extreme poses in Fig. 14. As shown in the figures, our method robustly recovers the realistic details of the loosing clothes, while SiTH shrinks to the SMPL-X body. LGM generates unnatural colors. SIFU and PIFu generate blurry rendering with artifacts.

Figure: Figure 12: Novel view synthesis comparison for loosing clothes. The details are highlighted in the red boxes.
Refer to caption: x12.png

Figure: Figure 13: Novel view synthesis comparison for loosing clothes. The details are highlighted in the red boxes.
Refer to caption: x13.png

Figure: Figure 14: Novel view synthesis comparison for extreme poses from HuMMan dataset. The details are highlighted in the red boxes.
Refer to caption: x14.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: The inference time is not reported; could you provide the specific timings for each step?
soundness: 3
strengths: - The problem addressed is quite challenging, and the designed network framework is reasonable, although the process is somewhat lengthy.
- The overall writing of the paper is clear and coherent.
summary: This work learns 3D human Gaussians from a single image, aiming to recover detailed appearance and geometry, including unobserved regions. 
The paper proposes a single-view generalizable framework that utilizes a generate-then-refine strategy guided by SMPL and diffusion priors. 
To enhance realism and accuracy, the method refines the SMPL model and employs SMPL priors to improve initial coarse human Gaussian estimations using sparse convolution and attention mechanisms.
weaknesses: - The results are quite unsatisfactory, as there is a noticeable discrepancy between the synthesized face and the original image. While this issue is mentioned in the limitations section, Figure 9 suggests that the images represent two different individuals.
- The comparison with other methods is insufficient, including approaches like TeCH[1] and GTA[2].
- The qualitative results appear somewhat blurry, with some input images also seeming unclear, particularly in the video results.
- Minor issues:  change "in 2" to "in Eq. 2" in line 248; clearly indicate the correspondence between F_I and F_S in the figures; missing references such as [3].

[1] TeCH: Text-guided Reconstruction of Lifelike Clothed Humans

[2] Global-correlated 3ddecoupling transformer for clothed avatar reconstruction

[3] HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors

----
I may change my opinion depending on the authors' rebuttal and whether they can address my concerns.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: The patch feature $F'_I$ on line 236 is ambiguous. it is unclear if this is derived from DINO or a trained model.
soundness: 3
strengths: 1. Feature Combination: The integration of pose-aware and pixel-space features improves the quality of the coarse GS prediction.
2. Enhanced Quality: This approach surpasses existing single-view human reconstruction methods in output quality.
summary: This paper proposes a novel approach to reconstruct human models from a single view. It first uses a coarse HGM model to predict GS representation. Besides feature from image space like the splatter image paper, this HGM model additionally aligns features based on the fitted SMPL-X template via a dual path. Then, the SMPL-X parameters and GS prediction are iteratively refined to improve mutual alignment. Finally, a ControlNet model generates a back view, which, combined with the front view, is processed by a two-view HGM for the final reconstruction.
weaknesses: 1. Incremental Novelty: There are several papers on 3D reconstruction topic that have adopted diffusion model to synthesize more views as the first step. This paper improves similar idea by using stronger controlNet signal, i.e. the rendering of course GS. This idea, while effective, is not very inspiring to me.

2. SMPL-X Refinement Concerns: Although justified in ablation studies, the proposed iterative SMPL-X refinement is a EM-like approach, while the previous works always refine SMPL parameters regarding the original input image. The proposed iterative refinement may depend heavily on initial GS and SMPL-X accuracy, making it sensitive and prone to degenerate solutions. Also, the wording in Section 3.3.2 is a bit redundant and would benefit from refinement for clarity and conciseness.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1) According to the paper, it looks like back-view refinement is quite effective. If back-view refinement improves the final Gaussians, can the improved Gaussian be a good source for back-view refinement? In other words, I wonder if performance can continue to improve if processes 2) and 3) in Figure 2 are performed cyclically. This experiment would be a good way to demonstrate how much the paper's approach improves reconstruction performance.
soundness: 4
strengths: In recent years, 3D Gaussian splatting (3DGS) has shown impressive results in 3D vision fields. Despite the promising results of 3DGS, the exploration of adapting 3D Gaussians for single-view 3D human reconstruction is still lacking. This paper points out the challenges of adapting 3D Gaussians for 3D human reconstruction. I think solving the proposed generate-then-refine pipeline could be one of the good research directions in 3D human reconstruction. The proposed approach sounds plausible, and can be plugged into most reconstruction algorithms. The paper is written in an easy-to-understand manner.
summary: This paper presents a generalizable single-view 3D human reconstruction system, which uses 3D Gaussians as 3D representations. Although the 3D Gaussian representation is promising due to its rendering speed, naively adapting 3D Gaussians in 3D human reconstruction often produces errored reconstructions. To resolve this issue, this paper introduces generate-then-refine pipeline that 1) reconstructs 3D Gaussians, 2) refines SMPL-X & back-view, and 3) reruns the 3D Gaussian reconstruction. The experiments demonstrate that it achieves state-of-the-art in novel-view synthesis and 3D reconstruction compared to existing methods.
weaknesses: I have several concern points below.

1) About SMPL-X refinement strategy

It is unclear why the proposed SMPL-X refinement pipeline actually helps. As described in Section 3.3.2, the SMPL-X is optimized by two-type losses: normal loss and rendering loss. The optimization targets of two losses can be inaccurate: normal GT is from normal estimator, rendering T is from coarse Gaussians. Furthermore, as the coarse Gaussians are processed based on the initial SMPL-X (Figure 2), it seems that coarse Gaussians will not deviate much from the initial SMPL-X. It is questionable whether the proposed SMPL-X improvements always guarantee improved reconstruction quality. For example, if using a distinguished SMPL-X estimator, the refinement pipeline can lead to performance degradation.

2) Validation of SMPL-X refinement

Although SMPL-X refinement is claimed to be an important contribution, there is no detailed experiment on it. Although Table 4 indirectly demonstrates this, more explicit experiments on SMPL-X improvements (measuring MPJPE for before refine. / after refine.) are required.  Additionally, to strengthen the effect of SMPL-X refinement, experiments on diverse SMPL-X initial estimator [1,2,3] are also needed.


3) Description about ControlNet in back-view refinement

The paper does not specified how ControlNet in back-view refinement is trained. Back-view refinement is also a major contribution of this paper, so it is important to clarify the training procedure for the reproducibility. I would recommend describing ControlNet's architecture, learning strategy, and training dataset. Specifically, I curious about how to augment coarse image (ControlNet condition) for the training. Furthermore, I also recommend that conducting a quantitative comparison with SiTH's back-view estimation (Figure 4) to justify the proposed refinement module.

4) Writing

a. In Section 3.3.2, it is unclear whether SMPL-X only optimizes parameters or whether SMPL-X and HGM are jointly optimized.

b. In L123, it is an overclaim that using multi-view images is main strength over existing methods. This algorithm setup using 12 rendered multi-view images (L350) is not significantly different from that using a 3D scan. Fundamentally, this algorithm also used 3D scans (L349), and did not show results when trained with 12 multi-view images for the existing methods.



[1] Moon et al., Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation, CVPRW 2022.

[2] Lin et al., One-stage 3d whole-body mesh recovery with component aware transformer, CVPR 2023.

[3] Zhang et al., PyMAF-X: Towards well-aligned full-body model regression from monocular images., IEEE 2023.


In conclusion, the proposed approach looks plausible and well-designed, but I am concerned that the proposed generate-then-refine strategy is actually effective. I think it could be a good academic paper if it includes a more clear explanation about SMPL-X refinement (1, 2) and back-view refinement (3), which are the core of the generate-then-refine strategy.

Meta-review: This paper tackles the challenging problem of single-view 3D human reconstruction using 3D Gaussian representations. The method effectively addresses common limitations in detail recovery and novel-view synthesis by a generate-then-refine framework leveraging SMPL-X and diffusion priors. The integration of SMPL refinement, ControlNet-based back-view generation, and sparse convolution enhances reconstruction quality. Though concerns about SMPL-X refinement validation and back-view training details were noted, the reviewers generally agree that the contributions are sound via several rounds of discussions. The revised appendix is clear, and the method achieves state-of-the-art results. With revisions to address specific weaknesses, this paper is a valuable contribution to the research community.

Decision: Accept (Poster)