Generalizable Monocular 3D Human Rendering via Direct Gaussian Attribute Diffusion

Authors: Yingzhi Tang, Qijian Zhang, Junhui Hou
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: novel view synthesis, 3D human body reconstruction, 3D Gaussian splatting

Abstract:
This paper leverages 3D Gaussian Splatting to tackle the challenging task of generating novel views of humans from given single-view images. Existing methods typically adopt an indirect supervision manner, i.e., splat-based rasterization for differentiable rendering. However, the intricate coupling of various 3D Gaussian attributes complicates precise error backpropagation during optimization, often
resulting in convergence to local optima. In contrast, we propose a novel direct paradigm to train a conditional diffusion model directly supervised by proxy-ground-truth 3D Gaussian attributes. Specifically, we propose a two-stage construction process to derive consistent and smoothly distributed proxy-ground-truth 3D Gaussian attributes. Subsequently, we train a point-based conditional diffusion model customized to learn the data distribution of these proxy attributes. The resulting diffusion model can generate the 3D Gaussian attributes for the input
single-view image, which are further rendered into novel views. Extensive experimental results showcase the significant performance advancement of our method over state-of-the-art approaches. Source code will be made publicly available.

Decision: reject
Reviewer scores: 3, 10, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730100383956

thread_type:
official_review

<thread_object>
creator: reviewer_73dc

<content>
summary:
This paper proposes Human-DAD, which adapts a new paradigm for training a generalizable 3DGS, to reconstruct the 3D human body from a single-view image. Unlike a common strategy in optimizing 3DGS from multi-view images, the authors proposed to train the model with direct supervision (applying loss function directly into Gaussian attributes). To apply good supervision, this paper uses two-stage workflow to construct 3D pseudo-GTs of 3D Gaussians: optimizing 3D Gaussian per scene and then unifying the distribution of Gaussian attributes along the camera views. Furthermore, to reconstruct 3D human Gaussians from the single-view image, the authors device a diffusion model conditioned from SMPL-semantic features and pixel-aligned features. The proposed Human-DAD outperforms existing state-of-the-art 3D human reconstruction methods.

strengths:
In recent years, 3D Gaussian splatting (3DGS) has shown impressive results in 3D vision fields. Despite the promising results of 3DGS, the exploration of adapting 3DGS for single-view 3D human reconstruction is still lacking. This paper can provide several good intuitions about the 3DGS for 3D human reconstruction, especially in the proposed 'distribution unification' of 3D Gaussian pseudo-GTs.

weaknesses:
I have several concern points as below.


1) Discussion about direct vs indirect supervision

a. This paper strongly suggests the inefficacy of the traditional 'indirect paradigm' in reconstructing 3D humans with 3DGS.
In 3DGS for 3D objects, I agree that recent works apply image loss to train their model instead of direct supervision [4]. However, to the best of my knowledge, I haven't heard about employing 3DGS for single-view 3D human reconstruction (also in the submitted paper). If there is a reference algorithm, it would be good to refer to it and conduct the experiment based on it. If not, the proposals in this paper may be a hasty conclusion.

b. This paper validates the effectiveness of the direct paradigm in their constructed algorithm (Tab. 1). However, their experiment setting in Tab. 1 cannot be representative of covering recent 3D human reconstruction approaches. So, it is difficult to suggest that a direct paradigm is better by only referring Tab. 1. I recommend that experiment Tab. 1 should be conducted in more general settings, such as [1].

c. Direct supervision is not always better than indirect supervision. In the two-stage workflow (Fig. 2), the algorithm reduces the randomness of fitted 3DGS, but it does not completely solve it. In generating 3D pseudo-GTs, enormous 3DGS can be mapped into multi-view images, similar to many-to-one mapping (L44) in indirect supervision. This discussion needs to be taken into consideration


2) Distribution unification

As shown in Fig. 4 and Tab. 1, the distribution unification (L250-257) is effective. However, it is difficult to understand why the training approach of Fig. 2 (right) helps the Gaussian distribution. The approach is very simple, batch-wise training. For example, if the network learns identity (A -> A), it would be a good local minima to minimize the photometric loss. What is the main component that to leads unifying the Gaussian distribution? A detailed discussion of this should be included in the manuscript.


3) Evaluation of 3D geometry

This paper only produces evaluation in rendered images (PSNR, SSIM, ...) and does not report 3D metrics (P2S, Chamfer Distance, ...), although the used datasets have 3D GT scans.


4) Paper writing

a. In L32-38, it does not clarify the categorization: Reconstruction-based vs NerF-based. The reconstruction-based approaches also use neural fields. I think that more suitable words are image-based vs video/multi-view-based. I recommend that this paper needs more clarification about it.

b. In L128, the paper mentioned Zou et al. follow PIFu. It's not clear which part of PIFu they followed (PIFu does not use triplane representation). 

c. It is unclear what the authors used datasets for Tab. 1 and Tab. 3.


[1] Zou et al., Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers, CVPR, 2024.


In conclusion, I'm concerned about accepting this paper, for two main reasons.
a. It is hasty to suggest direct supervision is beneficial, given the content of this paper.
b. Distribution unification of 3DGS is difficult to agree on, without extensive explanation of its design reason.

questions:
1) Diffusion model
It is unclear why the diffusion model was chosen. I conjecture the main strength of the diffusion model is generative power.
Does the algorithm produce diverse reconstruction candidates corresponding to the single-view image?

2) Reconstruction quality
In Fig. 8, why the reconstructed human face is blurred, unlike Fig. 6?
In Fig. 7, why the SHERF's results are more darker than others, unlike Fig. 6?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730134186509

thread_type:
official_review

<thread_object>
creator: reviewer_8d88

<content>
summary:
This paper proposes to train a conditional diffusion model for generating novel views of humans from given single-view images, directly supervised by proxyground truth 3D Gaussian attributes. The authors first create a dataset of proxy ground-truth Gaussian attributes with addtional neural network to smooth the dataset distrbution. A point-based conditional diffusion model is then employed to learn the data distribution of these attributes. Experimental results showcase the performance advancement of their method over current approaches.

strengths:
The method is well-written, with clear motivation behind each module design. The final results demonstrate improvement compared to previous methods like LGM. The idea of using a neural network to constrain the distribution of target 3D Gaussian attributes makes sense and is effective.

weaknesses:
- The main claim of the paper is that the direct supervision approach (using 3D regression loss) is superior to the indirect supervision method (using rendering loss). However, direct supervision approach also has limitations with memory consumption and the time-intensive nature of direct supervision. Moreover, with more iterations or larger batch size, will it be possible that the indirect supervision method could achieve similar performance? The **supervision gap** suggested by the paper isn’t clearly demonstrated to be an intrinsic limitation.
- Although the method outperforms other generalizable novel view synthesis techniques like LGM, the image quality remains limited since only one single input image are used. Another potential direction in this field is incorporating a 2D diffusion prior to enhance information, as demonstrated in Human 3Diffusion [1]. A comparison to these baselines is needed.
- The figures are a bit too small to clearly distinguish all the differences in results.
[1]: Xue Y, Xie X, Marin R, et al. Human 3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models[J]. arXiv preprint arXiv:2406.08475, 2024.

questions:
- How does the model perform on in-the-wild and out-of-distribution evaluations?
- Is it feasible to extend this approach to support multi-view input and obtain much better novel view synthesis results? Several works focusing on this can be considered like [2], [3].
  [2]: Kwon Y, Kim D, Ceylan D, et al. Neural human performer: Learning generalizable radiance fields for human performance rendering[J]. Advances in Neural Information Processing Systems, 2021, 34: 24741-24752.
  [3]: Gao Q, Wang Y, Liu L, et al. Neural novel actor: Learning a generalized animatable neural representation for human actors[J]. IEEE Transactions on Visualization and Computer Graphics, 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674525000

thread_type:
official_review

<thread_object>
creator: reviewer_jqqm

<content>
summary:
This paper uses the 3D GS representation to generate novel views of humans from single-view images. Different from existing methods that train with supervision from 2D image space, it proposes to learn a diffusion model that directly models the GS attributes in the 3D GS space. To prepare a proxy-ground-truth GS dataset for training, the position of GS is obtained by downsampling from GT point cloud, and the other attributes are optimized using multi-view supervision and a network-based regularization. A GS attribute diffusion model is then trained with the condition of point cloud position, pixel-align feature and semantic feature. Experiments demonstrated the superior performance of the proposed approach.

strengths:
1. This paper designs a pipeline to prepare the ground truth GS dataset regularized by network overfitting is new for me.
2. The proposed GS attribute diffusion model condition on image feature maps and SMPL semantic maps makes sense.
3. The proposed method shows better performance compared with the state of the arts.

weaknesses:
1. For GS dataset preparation, stage 1: per-scene overfitting seems to be ineffective in regulating the distribution only relying on the inherent smoothness of a neural network. So, it further needs a network for additional distribution unification. However, if I'm not wrong, it still mainly depends on the smoothness of the neural network without other regularization. I wonder if the use of a neural network in the first stage is necessary, which makes the pipeline over complicated.
2. I'm suspicious of the core argument that GS space supervision leads to better performance than image space supervision. Empirically, we are finally evaluating the NVS results in image space. It's not straightforward that a small error in GS space could indicate a small error or better quality in the image space. For the ablation study in Table 3, the performance difference using direct/indirect supervision is not that significant, the image-supervision method even gets better scores in SSIM and LPIPS, which reflect the perceptual quality.
3. Similarly, for the ambiguity problem the author mentioned when using 2D space supervision, I don't think it's related to supervision space but to deterministic/stochastic paradigm or loss design. 
4. The diffusion model design is very confusing, especially for the extra steps for alpha, scale, and quaternion attributes.
5. It seems that using stable diffusion model to generate the back view and using position generator (HaP) [1] to generate point cloud are of great importance to better performance. But these two components are existing works, not the technical contribution of this paper.

[1] Yingzhi Tang, Qijian Zhang, Junhui Hou, and Yebin Liu. Human as points: Explicit point-based 3d human reconstruction from single-view rgb images. arXiv preprint arXiv:2311.02892, 2023.

questions:
1. It seems that this work mainly focuses on the texture of humans while using an existing work for shape generation. I wonder about the importance of shape during evaluation. Is it possible to extract the texture of other comparable method and re-map it to your point could or ground truth point cloud to exclude the influence of different shapes?
2. I also wonder if the diffusion network just maps the front and back view to the point cloud. Could you show some results that use front view and back view to perform 3DGS optimization to map the texture?
4. I wonder how the extra step of the diffusion model is performed. Is it not the diffuse-denoise process modeling anymore but a straightforward one-step regression? If so, I think the description is confusing and misleading.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691049826

thread_type:
official_review

<thread_object>
creator: reviewer_kxve

<content>
summary:
This paper proposes a new method, Human-DAD, and shows it outperforms previous SOTAs on the task of novel view synthesis of humans from a single-view image. The authors showed that *directly* predicting 3D Gaussian attributes enhances the effectiveness of the training process, compared to the traditional *indirect* supervision through photometric losses. To enable the *direct* paradigm training, a two-stage data construction method is proposed to obtain proxy ground truth 3D Gaussian attributes. Using the proxy ground truth as training data, a conditional Diffusion Model, GSDIFF, is trained to output 3D Gaussian attributes of a scene given a single-view image of human.

The paper contains extensive experiments, ablation studies and user studies, proving its superiority over other baselines and previous SOTAs.

strengths:
1. The writing of the paper is in general very clear and detailed. The presentation of the flowchart is very useful for better understanding the multi-stage method. 
2. The method is innovative and non-trivial. For example, in stage1 of the Gaussian attribute dataset construction, instead of doing a vanilla 3DGS per scene optimization, the authors trained a point cloud transformer to directly output the other attributes given Gaussian positions. As shown later in the Experiment section, the inherent smoothness tendency of the learned network is essential for achieving high quality NVS later on.
3. The design of the diffusion model is very careful and smart. Instead of naively taking the single-view image as input, the diffusion model is conditioned on the Gaussian positions, pixel-aligned features and SMPL semantic features. Off-the-shelf mono-depth prediciton model and instruction based image2image model are used in the process to obtain high quality condition signals. 
4. As shown in Table2, Human-DAD outperforms all the previous SOTAs that it has been compared to on all the metrics reported by the authors. The qualitative results also show the impressive visual quality of the output.

weaknesses:
1. Wrong index to the referenced figure in the paper. I believe it should be Fig8, not Fig3 at line 503.
2. Same issue as above at line 507-508
3. The paper will be stronger if an ablation study of naively conditioning the diffusion model on input RGB image is provided in the paper.
4. Reference should be given for the claim at line 41-43.
5. The 2-stage Gaussian attribute data construction process, though effective, might have limited generality over heterogeneous data. The paper uses a single dataset, Thuman2, for proxy GT construction. If given additional human data with diverse backgrounds, different scales or shifted domains etc., whether this carefully designed and calibrated pipeline can be as effective remains unclear. 
6. What auxiliary constraints are imposed should be elaborated at line 248 in the main paper.

questions:
1. At line 291, the authors mention the semantic lables are embedded in the latent space through MLPs. Could you elaborate on this? Are the MLPs pre-trained or trained separately by the authors?
2. Zooming in on the results in the last column of Figure6, I saw artifacts of thin dark lines, which almost look like countour lines. Can you explain what might be the cause of such artifacts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730756447859

thread_type:
official_review

<thread_object>
creator: reviewer_grmh

<content>
summary:
This paper introduces a method that renders novel views of human avatars, given only a single image. The authors propose a pipeline of multiple stages, where they first learn a prior and then they can render a human under novel views. They highlight their paradigm as "direct", using extracted attributes for supervision, in contrast with other approaches (indirect) that use differentiable rasterization and pixel-level supervision during training.

strengths:
+ Learning a generalizable approach for human avatars is an interesting topic. Most approaches in this area are identity-specific. Rendering a novel human, given only a single image, is useful. This method seems to work in this case.

+ The authors show better performance than other methods.

weaknesses:
- The claim of "direct" paradigm is not clear. The method essentially consists of multiple stages. It is then supervised by extracted features, that can also propagate the error to later stages.

- The writing of the paper is not clear. 

- It was not directly obvious what the ablation studies show with respect to the contributions of various components as they present single cases of failure as opposed to complete numerical results, so it is hard to understand where the significant improvements in the overall results come from.

- There are no animation results. Most of similar methods show both novel views (reconstruction), but also animation (e.g. SHERF). It will be good that the authors include animation results as well, to be able to validate the consistency of their solutions.

- Even though the authors show good results, the overall contribution and novelty of the paper is not well supported. Components like transformers, stable diffusion, etc have been used by other papers as well. Learning pixel-aligned features from different views have been proposed by previous works, like the generalizable NeRF ActorsNeRF (ICCV 2023) and MonoHuman (CVPR 2023). Stable diffusion has been used by works for humans, like DreamHuman (NeurIPS 2024), DreamAvatar (CVPR 2024), AvatarPopUp (ECCV 2024), and other works for 3D objects (e.g. DreamFusion). It is not clear to this reviewer what the additional challenges of monocular reconstruction are in the particular set up of this paper and how the particular method addresses them.

Missing citations of generalizable models for humans, including GM-NeRF (CVPR 2023), Neural Human Performer (NeurIPS 2021), ActorsNeRF (ICCV 2023).

questions:
Please see the above weaknesses.

What is causing the large numerical improvement?

Can the authors provide animation results? Would their method work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732377983016

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reponse to all reviewers

comment:
## Thanks very much for all reviewers’ time and efforts in reviewing our paper and providing valuable comments.


### We are very grateful to the positive acknowledgments of this work:

1. Reviewer GRmh thinks that our approach is **interesting**, **useful**, and with **better performance**.
2. Reviewer kXVE points out that our method is **innovative** and **non-trivial**, our design is **careful** and **smart**, our writing is **clear** and **detailed**, and the **comprehensive experiments** demonstrate our **superior performance**.
3. Reviewer jqqm acknowledges the **novelty** of the preparation process of the ground-truth Gaussian Splatting dataset and the **technical rationality** of Gaussian attribute diffusion, as well as our **performance advantage**.
4. Reviewer 8d88 acknowledges our **effective idea** and **improved performance**, and thinks that our paper is **well-written** and our module design is **clearly motivated**.
5. Reviewer 73DC thinks that our work provides **good intuitions** about human 3DGS, especially for the preparation of proxy ground-truth Gaussian attributes.


### During the rebuttal period, we made the following efforts to address the raised concerns for improving the quality of this work:

a. We clarified the definition of the direct paradigm and the indirect paradigm;

b. We explained why the direct paradigm is needed and why it is more efficient than the indirect paradigm.

c. We explained why we selected the diffusion model and gave the details of training the diffusion model.

d. We provided an example of animation on Human-DAD.

e. We conducted an ablation study by training an image-conditioned diffusion model to predict the 3D Gaussian attributes.

f. We evaluated GTA, SiTH and Human-DAD with ground truth 3D shapes.

g. We ran more iterations to train the indirect model and trained an indirect model with a larger batch size.

h. We compared Human-DAD with Human3Diffusion.

j. We evaluated the 3D geometric performance of different methods.

### Below we will briefly summarize each raised Weakness (W) and Question (Q), and provide the corresponding response item by item.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733130697385

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you for your feedback on our paper. We would like to re-emphasize that the focus of our direct paradigm is particularly on the task of human novel view synthesis within the scope of our designed pipeline. As demonstrated in the experiments, our approach offers a more efficient and effective solution for our targeted specific task compared with other methods. We believe that our findings indeed bring new and valuable insights to the community.

Accordingly, we will revise some of our statement and conclusion to avoid potential issues of over-claiming and readers’ misunderstanding that the direct paradigm is universally superior to the indirect paradigm across all 3DGS task scenarios.

We kindly hope that you could reconsider your rating in light of the direct nature of our paradigm, which is designed and evaluated for our specific task. We believe that the results we presented have already well supported the efficiency and effectiveness of our approach.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Human-DAD. The claimed main contribution is to adopt a "direct" supervision paradigm for learning 3D Gaussian attributes in the 3D Gaussian Splatting (3DGS) space, distinguishing it from existing methods that rely on pixel-level supervision in 2D image space.

Commonly agreed strength includes its ability to generalize 3D human avatar reconstruction from single-view images, addressing a gap in the field where most existing methods rely on identity-specific optimization. While the performance is superior, reviewers also appreciated the novelty of the approach, including the innovative pipeline design, the use of direct supervision for 3D Gaussian attributes, and the introduction of 'distribution unification' for pseudo-ground truth construction.

Common key issues raised include skepticism about the core claim that direct supervision in 3DGS space is superior to indirect supervision in 2D space, as the provided evidence is unconvincing. Reviewers also noted that the experiments in Table 1 lack generality and should be conducted in broader settings for more robust validation. The absence of evaluations on 3D geometry metrics was brought up. Additionally, the paper's writing was criticized for being unclear, with ambiguous explanations of key concepts, methods, and experimental setups.

While the authors tried to address the issues, the major issue regarding the core claim of the paper—whether direct supervision is truly superior—remained unresolved. Ultimately, the paper received four ratings below 5 out of 5, leading the ACs to reach a consensus on rejection.