ZERO-1-to-G: Taming Pretrained 2D Diffusion Models for Direct 3D Generation

Authors: Xuyi Meng, Chen Wang, Jiahui Lei, Kostas Daniilidis, Jiatao Gu, Lingjie Liu
Primary area: generative models
Keywords: native 3D diffusion, 3D generative model, gaussian splats

Abstract:
Recent advances in 2D image generation have achieved remarkable quality, largely driven by the capacity of diffusion models and the availability of large-scale datasets. However, direct 3D generation is still constrained by the scarcity and lower fidelity of 3D datasets. In this paper, we introduce Zero-1-to-G, a novel approach that addresses this problem by enabling direct 3D generation on Gaussian splats through 2D diffusion models. Our key insight is that Gaussian splats, a 3D representation, can be decomposed into multi-view images encoding different attributes. This reframes the challenging task of direct 3D generation within a 2D diffusion framework, allowing us to leverage the rich priors of pretrained 2D diffusion models. To incorporate 3D awareness, we introduce cross-view and cross-attribute attention layers, which capture complex correlations and enforce 3D consistency across generated splats. This makes Zero-1-to-G the first direct 3D generative model to effectively utilize 2D pretrained diffusion priors, enabling efficient training and improved generalization to unseen objects. Extensive experiments on both synthetic and in-the-wild datasets demonstrate superior performance in 3D object generation, offering a new approach to high-quality 3D generation.

Decision: reject
Reviewer scores: 5, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730359070374

thread_type:
official_review

<thread_object>
creator: reviewer_vjy6

<content>
summary:
This paper introduces Zero-1-to-G, a novel 3D generative framework that generates 3D Gaussian splats from a single input image. The framework decomposes the generation process into five distinct attribute generations for fixed camera viewpoints, allowing a fine-tuned 2D diffusion model and a VAE decoder to generate multi-view, multi-attribute images.

strengths:
* Decomposing the 3D Gaussian splat generation problem into multi-view and multi-attribute image generation is interesting.
* The paper is structured well, making it easy to understand and follow.

weaknesses:
**Weakness 1**

The primary concern is the novelty of the paper. LGM [1] is designed to quickly generate Gaussian splats from various multi-view images, with the advantage of compatibility with different multi-view diffusion models. So, I think Zero-1-to-G can be regarded as a combination of an existing multi-view diffusion model and LGM, extending the cross-domain attention mechanism of Wonder3D [2] to five attributes.

**Weakness 2**

Experimental results are not sufficient to prove the effectiveness of Zero-1-to-G. Many One-Image-to-3D [2, 3, 4, 5] models report 3D reconstruction results (e.g. Chamfer Distance) on the GSO dataset. Since Zero-1-to-G takes 8.7 seconds to generate a 3D Gaussian splat, it would be better to show the performance with fewer sampling steps and a similar inference time to LGM.

[1] Tang et al., LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation, ECCV 2024.

[2] Long et al., Wonder3D: Single Image to 3D using Cross-Domain Diffusion, CVPR 2024.

[3] Woo et al., HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3D, CVPR 2024.

[4] Liu et al., SyncDreamer: Generating Multiview-consistent Images from a Single-view Image, ICLR 2024.

[5] Xu et al., DMV3D: DENOISING MULTI-VIEW DIFFUSION USING 3D LARGE RECONSTRUCTION MODEL, ICLR 2024.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730544979422

thread_type:
official_review

<thread_object>
creator: reviewer_cxys

<content>
summary:
This paper presents a novel method for 3d generation by finetuning a pretrained 2D diffusion model to generate 3D gaussian splatter images directly according to a single-view input image. The method integrates the generative priors from the pretrained diffusion model into the task of direct 3D generation and achieve good results.

strengths:
* The paper proposes a novel framework for 3D generation by finetuning a pretrained 2D diffusion model for directly 3D generation, which is novel to me.
* The experiments are comprehensive and well-executed.
* The paper is well-written and easy to understand for people familiar with 3D generation tasks.

weaknesses:
* The paper adopts LGM to build to overall dataset. This raises the concern of the overall quality of the training dataset. In line 216-217, the author declares that fitting-based reconstruction methods lead to excessively high-frequency signals which is hard for pretrained VAE to decode. However, in the paper, the author actually finetuned the decoder part of VAE with LGM outputs, which is confused. Why we could not finetune a VAE with the  fitting-based reconstruction data? The author should clarify this.
* The discuss of Limitation is not included in the paper.
* Overall, representing and generating Gaussian as images have been proposed by Splatter Image(CVPR2024), so the technological contribution of this paper is relatively limited. But I like the idea of finetuning a diffusion model to enjoy the generative priors, so I vote to a borderline accept.

[1] Splatter Image: Ultra-Fast Single-View 3D Reconstruction (https://arxiv.org/abs/2312.13150

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730561213466

thread_type:
official_review

<thread_object>
creator: reviewer_wyeo

<content>
summary:
This paper introduces a new approach called ZERO-1-TO-G for 3D generation using Gaussian splats through 2D diffusion models. The key contribution of this paper is to represent 3D objects as multi-view, pixel-aligned Gaussian images and to fine-tune the VAE decoder so that the 3D objects can be generated directly. Experiments on GSO demonstrate promising results, with the proposed method outperforming several baselines in image-conditioned 3D generation tasks.

strengths:
1. Due to the scarcity of 3D datasets, this paper presents a method to unleash the power of pretrained 2D diffusion models for direct 3D generation, with the potential to further enhance the performance of 3D generation.
2. The paper is generally well-written and easy to follow. The authors clearly explain the motivation behind their work, and the key components of their proposed method. The figures are helpful in illustrating the overall fine-tune VAE+LDM pipeline.
3. The experimental results show better performance than baselines.
4. The paper includes relevant ablations validating the individual design choices.

weaknesses:
1. My main concern is that, despite the use of cross-view and cross-attribute attention layers to enhance 3D consistency, the generated results still exhibit inconsistencies across multiple views. How can we address these inconsistencies in the multi-view Gaussian splat results?
2. Methods such as 3DShape2Vec and 3DILG conduct experiments on shape autoencoding, which serves as the upper bound for the generated 3D objects. I believe that this experiment is also critical for ZERO-1-TO-G.
3. Missing comparisons experiment of generated 3d models for text-to-3d.
4. Missing reference: 
[1] Cat3d: Create anything in 3d with multi-view diffusion models. NIPS 2024
[2] GPLD3D: Latent Diffusion of 3D Shape Generative Models by Enforcing Geometric and Physical Priors. CVPR 2024

questions:
Please refer to the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730623228656

thread_type:
official_review

<thread_object>
creator: reviewer_k7wt

<content>
summary:
This paper aims to leverage 2D priors in pre-trained image diffusion models for 3D content generation. The key idea is to adapt the previous work of Splatter Image as a proxy 2D representation of 3D information, such that multi-view image diffusion architectures can be applied.

strengths:
1. The idea of leveraging 2D diffusion priors for 3D generation is promising, and the adaptation of Splatter Image structure is technically sound.
2. A series of correspondingly needed modifications are applied. For example, the different Gaussian attribute maps are normalized/reorganized to RGB space, and some self-attention operations are added for the modeling of both intra-view and inter-view distributions.

weaknesses:
1. Generally, I tend to think that the proposed processing pipeline is a relatively simple and straightforward combination of existing techniques. First, the concept of Splatter Image is an existing work, and in this paper the adaptations are limited to some normalization and channel duplication operations. Second, the attention-related operations are also well-explored in previous methods, such as the "reshaping for attention" in MVDream and the "switcher mechanism" in Wonder3D.

2. I disagree with the authors' claim that "Zero-1-to-G is the first direct 3D generative model based on 2D diffusion frameworks". In terms of conceptual novelty and the target of leveraging 2D diffusion architectures, Omage [R1] and GIMDiffusion [R2] are even more representative works. 
--[R1] X. Yan, et al., “An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion,” 2024.
--[R2] S. Elizarov, et al., “Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation,” 2024.

3. I cannot see essential advantages of the proposed generation paradigm, because it still falls into the scope of multi-view image diffusion. Note that in the original design of Splatter Image, it can encode the whole 3D scene within a single image by modeling Gaussian position as a combination of depth and offset. Yet in this paper the Splatter Image still uses (x, y, z) coordinates and requires multi-view representations. Thus, the joint modeling of multi-view consistency is still a major challenge. Besides, the pre-trained diffusion priors are somewhat damaged, because the VAE component (the decoder) need to be retrained.

4. The ground-truth Splatter Image representations are generated by LGM, which fundamentally limiting the potential of the proposed method.

questions:
Is it possible to use the single-image format of Splatter Image? In this way, the whole 3D scene can be encoded in one image-like tensor, thus avoiding the difficulty of constraining multi-view consistency. Of course, the VAE component also need to be retrained to adapt to the image-like tensor.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732393432179

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all the reviewers for their valuable feedback on our work. In response to common concerns, we emphasize that our main contribution lies in the ***novel insight*** that splatter images—a 3D representation—can be decomposed into forms resembling real images, with each attribute image aligning closely with the distribution of pretrained 2D diffusion models. This allows us to leverage Stable Diffusion with minimal yet impactful modifications, enabling a direct 3D diffusion model that achieves high-quality 3D generation.

Our contribution can be summarized in three key aspects:

1. **Leveraging Pretrained Priors**:  
   We find that each attribute image of splatter image is modeled within the distribution of Stable Diffusion, therefore we propose to use the pretrained prior of Stable Diffusion as a good starting point to train our 3D generation model.

2. **Addressing Limited 3D Data**:  
   Our method overcomes the challenge of limited 3D training data by utilizing pretrained 2D priors learned from large-scale internet data. This enables efficient training with strong generalization capabilities.

3. **Ensuring 3D Consistency**:  
   Unlike multi-view 2D methods that rely on the joint distribution of multi-view images, our approach directly learns in a 3D latent space. This ensures superior 3D consistency compared to existing multi-view pipelines.

We hope this perspective inspires future research in this area. Furthermore, our modular and compatible network design provides flexibility for researchers looking to extend or build upon our approach.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Zero-1-to-G, a framework leveraging 2D diffusion priors for 3D Gaussian splat generation. The proposed method shows promise in adapting 2D pre-trained diffusion models for 3D tasks; however, there are still concerns about its novelty and validation. The mentioned key contributions, such as fine-tuning a VAE decoder and multi-view attention mechanisms, are likely incremental extensions of prior works. Moreover, reliance on LGM-generated training data, and limited experimental comparisons with strong baselines further affect its impact. I’d recommend the authors to consider all these and further improve the work.