DMV3D: Denoising Multi-view Diffusion Using 3D Large Reconstruction Model
Authors: Yinghao Xu
Hao Tan
Fujun Luan
Sai Bi
Peng Wang
Jiahao Li
Zifan Shi
Kalyan Sunkavalli
Gordon Wetzstein
Zexiang Xu
Kai Zhang
Venue: ICLR 2024 spotlight (2024)
Keywords: 3D Generation; Single-view 3D Reconstruction; text-to-3D
Primary area: generative models

Abstract: We propose DMV3D, a novel 3D generation approach that uses a transformer-based 3D large reconstruction model to denoise multi-view diffusion. Our reconstruction model incorporates a triplane NeRF representation and, functioning as a denoiser, can denoise noisy multi-view images via 3D NeRF reconstruction and rendering, achieving single-stage 3D generation in the 2D diffusion denoising process. We train DMV3D on large-scale multi-view image datasets of extremely diverse objects using only image reconstruction losses, without accessing 3D assets. We demonstrate state-of-the-art results for the single-image reconstruction problem where probabilistic modeling of unseen object parts is required for generating diverse reconstructions with sharp textures. We also show high-quality text-to-3D generation results outperforming previous 3D diffusion models. Our project website is at: https://dmv3d.github.io/.

DMV3D: DENOISING MULTI- DIFFUSION USING 3D LARGE RECONSTRUCTION MODEL
Yinghao Xul Hao Tan'
Fujun Luan't
Sai Bi
Peng Wang!. Jiahao Li
Zifan Shi Kalyan Sunkavalli
Gordon Wetzstein Zexiang Xu
Kai Zhang
Adobe Research Stanford
HHKU
4HKUST STTIC
ABSTRACT
We propose DMV3D, a novel 3D generation approach that uses a transformer- based 3D large reconstruction model to denoise multi-view diffusion. Our reconstruction model incorporates a triplane NeRF representation and can denoise noisy multi-vier images via NeRF reconstruction and rendering. achieving single- stage 3D generation in ~000 on single A100 GPU. We train DMV3D on large- scale multi-view image datasets of highly diverse objects using only image reconstruction losses, without accessing 3D assets. We demonstrate state-of- the-art results for the single-image reconstruction problem where probabilistic modeling of unseen object parts is required for generating diverse reconstructions with sharp textures. We also show high-quality text-to-3D generation results outperforming previous 3D diffusion models. Our project website is at: https: // just imyhxu gi thub. io/pro jects /dv33d/
INTRODUCTION
The advancements in 2D diffusion models (Ho et al., 2020; Song et al., 2020a; Rombach et al., 2022a) have greatly simplified the image content creation process and revolutionized 2D design workflows. Recently, diffusion models have also been extended to 3D asset creation in order to reduce the manual workload involved for applications like VR, AR, robotics, and gaming. In particular, many works have explored using pre-trained 2D diffusion models for generating NeRFs (Mildenhall et al., 2020) with score distillation sampling (SDS) loss (Poole et al., 2022; Lin al., 2023a). However, SDS-based methods require long (often hours of) per-asset optimization and can frequently lead to geometry artifacts, such as the multi-face Janus problem.
On the other hand, attempts to train 3D diffusion models have also been made enable diverse 3D asset generation without time-consuming per-asset optimization (Nichol et al., 2022; Jun & Nichol, 2023). These methods typically require access ground-truth 3D models/point clouds for training, which are hard to obtain for real images. Besides, the latent 3D diffusion approach (Jun & Nichol, 2023) often leads to an unclean and hard-to-denoise latent space (Chen et al., 2023b) on highly diverse category-free 3D datasets due to two-stage training, making high-quality rendering challenge. To circumvent this, single-stage models have been proposed (Anciukevičius et al., 2023; Karnwaar et al., 2023), but are mostly dategory-speciffe and focus simple classes.
Our goal is to achieve fast, realistic, and generic 3D generation. To this end, we propose DMV3D, diffusion model that can generate 3D (triplane) NeRFs from text or lingle-image input conditions via direct model inference. Our model allows for the generation of diverse high-fidelity 3D objects within 30 seconds per asset (see Fig. 1). In particular, DMV3D) is 2D multi-view image diffusion model that integrates 3D NeRF reconstruction and rendering into its denoiser, trained without direct 3D supervision, in an end-to-end manner. This avoids both separately training 3D NeRF encoders for latent-space diffusion (as in two-stage models) and tedious per-asset optimization (as in SDS methods).
novel single-stage
In essence, our approach uses 3D reconstruction model as the 2D multi-view denoiser in a multi- view diffusion framework. This inspired by RenderDiffusion (Anciukevičius et al., 2023)
"This work is done while the author is an intern at Adobe Research.
1 denotes equal advisory.

---

birthday cupcake,
Text-to-3
Image-to-31
Segient Anything Model
Input
Instance
Instance
Inpui
Instance
Instance 2
Figure 1: Top left: our approach achieves fast 3D generation (~30s on A100 GPU) from text single-image input; the latter one, combined with 2D segmentation methods (like SAM (Kirillov al., 2023)), reconstruct objects segmented from natural images. Bottom: single-image-to-3D model, we can produce multiple reasonable 3D assets from the same image. Top right: we demonstrate a scene comprising diverse 3D objects generated by our models.
probabilistic
achieving 3D generation through single-view diffusion. However, their single-view framework relies on category-specific priors and canonical poses and thus cannot easily scaled up to generate arbitrary objects. In contrast, consider sparse set of four multi-view images that surround an object, adequately describing 3D object without strong self-occlusions. This design choice is inspired by the observation that humans can easily imagine complete 3D object from few surrounding views with little uncertainty. However, utilizing such inputs essentially requires addressing the task of sparse-view 3D reconstruction highly challenging even without noise in the inputs.
Long-standing problem and known to be
We address this by leveraging large transformer models that have been shown to be effective and scalable in solving various challenging problems (Jun & Nichol, 2023; Nichol et al., 2022; Hong et al., 2023; Brown al., 2020; Shen et al., 2023). In particular, built upon the recent 3D Large Reconstruction Model (LRM) (Hong et al., 2023), we introduce novel model for joint reconstruction and denoising. More specifically, our transformer model can, from a sparse set of noisy multi-view images, reconstruct clean (noise-free) NeRF model that allows for rendering (denoised) images at arbitrary viewpoints. Our model conditioned on the diffusion time step, designed to handle any noise levels in the diffusion process. It can thus be directly plugged as the multi-view image denoiser in an multi-view image diffusion framework.
We enable 3D generation conditioned single images/texts. For image conditioning, fix of the sparse views as the noise- free input and denoise other views, similar to 2D image inpainting (Xie al., 2023). We apply attention-based text conditioning and classifier-free guidance, commonly used in 2D diffusion models, to enable text-to-3D generation. We train our model on large-scale datasets consisting of both synthetic renderings from Objaverse (Deitke al., 2023) and real captures from MVImgNet (Yu et al., 2023) with only image-space supervision. Our model achieves state-of-the-art results on single-image 3D reconstruction, outperforming prior SDS-based methods and 3D diffusion models. We also demonstrate high-qualit, text-to-3D results outperforming previous 3D diffusion models. In sum, our main contributions are:
A novel single-stage diffusion framework that leverages multi-view 2D image diffusion model to achieve 3D generation;
An LRM-based multi-view denoiser that can reconstruct noise-free triplane NeRFs from noisy multi-view images:
A general probabilistic approach for high-quality text-to-3D generation and single-image reconstruction that uses fast direct model inference (~30s single A100 GPU).
Our work offers a novel perspective address 3D generation tasks, which bridges 2D and 3D generative models and unifies 3D reconstruction and generation. This opens up opportunities to build foundation model for tackling variety of 3D vision and graphics problems.
2

---

Figure 2: SAM DMV3D. We use SAM (Kirillov et al., 2023) segment any objects from real scene photo and reconstruct their 3D shape and appearance with our method, showcasing our model's potential in enabling 3D-aware image editing experiences.
2 RELATED WORK
Sparse-view Reconstruction. Neural representations (Mescheder et al., 2019; Park et al., 2019; Mildenhall et al., 2020; Sitzmann et al., 2019; 2020; Chen et al., 2022; Müller et al., 2022) offer promising platform for scene representation and neural rendering (Tewari et al., 2022). Applied to novel-view synthesis, these approaches have been successful in single-scene overfitting scenarios where lots of multi-view training images are available. Recent efforts (Yu al., 2021; Chen al., 2021: Long et al., 2022: Wang et al., 2021; Lin et al., 2023b; Jain et al., 2021) have extended these ideas to operate with a sparse set of views, showcasing improved generalization capabilities to unseen scenes. As non-generative methods, these approaches struggle on covering the multiple modes in the large-scale datasets and thus can not generate diverse realistic results. In particular, LRM (Hong et al., 2023) tackles the inherent ambiguous single-image-to-3D
the recently-proposse problem in a deterministic way, resulting in blurry and washed-on oott textures for unseen part of the objects due to mode averaging. We resolve this issue by building probabilistic conge-conditioned 3D generation model through denosing multi view diffusion.
3D Generative Adversarial Networks (GANs). GANs have made remarkable advancements in 2D image synthesis (Brock et al., 2018; Karras et al., 2018; 2019; 2020; 2021). 3D GANs (Nguyen- Phuoc et al., 2019; Schwarz et al., 2020; Chan et al., 2021; 2022; Niemeyer & Geiger, 2021; Gu et al., 2021; Skorokhodov et al., 2022; Xu et al., 2022; 2023; Shi et al., 2022; Gao et al., 2022: Skorokhodov et al., 2023) extend these capabilities
generating 3D-aware assets from unstructured collections of single-view 2D images in an unsupervised manner. GAN architectures, however, are difficult train and generally best suited for modeling datasets of limited scale and diversity (Dhariwal & Nichol, 2021).
3D-aware Diffusion Models (DMs). DMs have emerged as foundation models for visual computing, offering unprecedented quality, fine-grained control, and versatility for 2D image generation (Ho al., 2020; Song et al., 2020b; Rombach et al., 2022a: Po al., 2023). Several strategies have been proposed to extend DMs to the 3D domain. Some of these approaches (Nichol et al., 2022; Jun & Nichol, 2023; Shue et al., 2023; Gupta et al., 2023; Ntavelis et al., 2023) use direct 3D supervision. The quality and diversity of their results, however, is far from that achieved by 2D DMs. This is partly due to the computational challenge of scaling diffusion network models up from 2D to 3D, but perhaps more so by the limited amount of available 3D training data. Other approaches in this category build on optimization using a differentiable 3D scene representation along with the priors encoded in 2D DMs (Poole et al., 2022; Lin et al., 2023a; Wang et al., 2022; 2023). While showing some success, the quality and diversity of their results is limited by the SDS-based loss function (Poole et al., 2022). Another class of methods uses 2D DM-based image- to-image translation using view conditioning (Liu et al., 2023b; Chan et al., 2023; Gu et al., 2023). While these approaches promote multi-view consistency, they do not enforce it, leading to flicker and other view-inconsistent effects. Finally, several recent works have shown success in training 3D diffusion models directly on single-view or multi-view image datasets (Karnewar et al., 2023; Chen et al., 2023b; Shen et al., 2023) for relatively simple scenes with limited diversity.
Prior RenderDiffusion (Anciukevičius et al., 2023) and concurrent Viewset Diffusion (Szymanowicz et al., 2023) work are closest to our method. Both solve the 3D generation problem using 2D DMs with 3D-aware denoisers. Neither of these methods, however, has been demonstrated to work on highly diverse datasets containing multi-view data of 1M objects. Our novel LRMM-bsed (Hong et al., 2023) 3D denoiser architecture overcomes this challenge and enables ofate-of of-te----he-ar the-art results for scalable, diverse, and high-quality 3D generation.
3

---

Triplane position erabeddings
Triplane
Transformer
tolemn
Image tokeniter
Cross-Att
(DLN))
Reshsap Upocappl
Imugee tokess
Rendering Joss
Figure 3: Overview of our method. We denoise multiple views (three shown in the figure to reduce clutterness; four used experiments) for 3D generation. Our multi-viww denoiser is large transformer model that reconstructs a noise-free triplane NeRF from input noisy images with camera poses (parameterized by Plucker rays). During training, supervise the triplane NeRF with rendering loss at input and novel viewpoints. During inference, we render denoised images at input viewpoints and combine them with inputs to obtain less noisy inputs for the next denoising step. We output the clean triplane NeRF at final denoising step, enabling 3D generation. Refer to Sec. 3.3 for how to extend this model to condition on single image or text.
METHOD
We now present our single-stage 3D diffusion model. particular, we introduce novel diffusion framework that uses a reconstruction based denoiser to denoise noisy multi-view images for 3D generation (Sec. 3.1). Based on this, we propose novel LRM-based (Hong al., 2023) multi- view denoiser conditioning
diffusion time step to progressively denoise multi-view images via 3D NeRF reconstruction and rendering (Sec. 3.2). We further extend our model support text and image conditioning, enabling controllable generation 3.3).
3.1 MULTI-VIEW DIFFUSION AND DENOISING
Denoising Diffusion Probabilistic Models (DDPM) transforms the data distribution 2%g ~ (ax) using Gaussian noise schedule in the forward diffusion process. The generation process is the reverse process where images are gradually denoised. The diffused data sample æ, at timestep t can be written as x, / Va + /1 ãge, where (O, I) represents Gaussian noise and the monotonically decreasing ā, controls the Signal-Noise-Ratte (SNR) of noisy sample 24.
Diffusion.
Multi-view diffusion. The original Zy distribution addressed 2D DMs is the (single) image distribution in a dataset. We instead consider the (joint) distribution of multi-view images (I,,..; Iv), where each set of I are image observations of the same 3D scene (asset) from viewpoints C = (c1..., The diffusion process is equivalent to diffusing each image independently with the same noise schedule:
I +
I}
Note that this diffusion process identical to the original one in DDPM, despite that we consider a specific type of data distribution I denoting per-object 2D multi-view images.
Reconstruction-bent based denoising. The reverse of the 2D diffusion process is essentially denoising. In this work, we propose leverage 3D reconstruction and rendering to achieve 2D multi-view image denoising, while outputting clean 3D model for 3D generation. In particular, we leverage 3D reconstruction module E(-) to reconstruct 3D representation S from the noisy multi-view images I, and render denoised images with differentiable rendering module R(-):
I, c), S, = E(I,,t,C)
where I,,, represents rendered image from S, at specific viewpoint
Denoising the multi-view input I, is done by rendering S, at the viewpoints C, leading to the prediction of noise-free In. This is equivalent to 2o prediction in 2D DMs (Song al., 2020a);
4

---

one can solve for æ;-1 from the input æt and prediction zo enable progressive denoising during inference. However, unlike pure 2D generation, we find only supervising Io prediction at input viewpoints cannot guarantee high-quality 3D generation (see Tab. 3). often leading to degenerate 3D solutions where input images are pasted on view-aligned planes. Therefore, we propose to supervise novel-view renderings from the 3D model S, well, which leads the following training objective:
Lrecon (t) = -Imee-Ina ( t,.
(3)
where Juu and Crui represent the full set of images and poses (from both randomly selected input and novel views), and E(..-) is an image reconstruction loss penalizing the difference between groundtruth and rendering ([E(T,, 1, C),c). c). Note that our framework is general - potentially any 3D representations (S) can be applied. In this work, we consider a (triplane) NeRF (Chan et al., 2022) representation (where R(-) becomes neural volumetric rendering (Mildenhall et al., 2020)) and propose LRM-hhaed reconstructor FI(-) (Hong al., 2023).
3.2 RECONSTRUCTOR MULTI-VIEW DENOISER
We build our multi-view denoiser upon LRM (Hong et al., 2023) and uses large transformer model to reconstruct clean triplane NeRF (Chan et al., 2022) from noisy sparse-view posed images. Renderings from the reconstructed triplane NeRF are then used as denoising outputs.
Reconstruction and Rendering, As shown in Fig. 3, we use a Vision Transformer (DINO (Caron et al., 2021)) to convert input images = [II (I...., IN] to 2D tokens, and then use a transformer to map learnable triplane positional embedding to the final triplane representing the 3D shape and appearance of an asset; the predicted triplane is then used to decode volume density and color with an MLP (not shown in Fig. 3 to avoid clutternes)) for differentiable volume rendering. The transformer model consists of a series of hiplane-to-image- cross-attention and uriplane-to-tripee self-attention layers in the LRM work (Hong al., 2023). We further enable time conditioning for diffusion baffusion-based progressive denoising and introduce a new technique for camera conditioning.
Time Conditioning. Our transformer- based model requires different designs for tine-conditioning, compared to CNN-based DDPM (Ho et al., 2020). Inspired by DiT (Peebles & Xie, 2022), we condition time by injecting the adal_N-Zero block (Ho et al., 2020) into the self- and cross- attention layers of our model to effectively handle inputs with different noise levels.
Camera Conditioning, Training our model on datasets with highly diverse camera intrinsics and extrinsics, e.g., MVImgNet (Yu et al., 2023), requires an effective design of input camera conditioning to facilitate the model's understanding of cameras for 3D reasoning. basic strategy is, as in the case of time conditioning, to use adaLN-Zero block (Peebles & Xie, 2022) on the camera parameters (as done in Hong al. (2023); Li et al. (2023)). However, we find that conditioning on camera and time simultaneously with the same strategy tends to weaken the effects of these two conditions and often leads to an unstable training process and slow convergence. Instead, we propose a novel approach parameterizing cameras with sets of pixel-aligned rays. In particular, following Sitzmann et al. (2021); Chen et al. (2023a), we parameterize rays using Plucker coordinates computed from the camera parameters, and x denotes cross-procts product We concatenate the Plucker coordinates with image pixels, and send them the ViT transformer for 2D image tokenization, achieving effective camera conditioning.
(o x d, d), where and d are the origin and direction of pixel ray
3.3 CONDITIONING ON SINGLE IMAGE OR TEXT
The methods described thus far enable our model to function as unconditional generative model. We now introduce how to model the conditional probabilistic distribution with a conditional denoiser E(I, t, C, y), where y is text or image, enabling controllable 3D generation.
Image Conditioning. We propose a simple but effective image-conditioning strategy that requires no changes to our model architecture. We keep the first view I, (in the denoiser input) noise-free to serve as the conditioning image, while applying diffusion and denoising on other views. In this case, the denoiser essentially learns to fill in the missing pixels within the noisy unseen views using cues extracted from the first input view, similar to the task of image inpainting which has been shown to be addressable by 2D DMs (Rombach al., 2022a). In addition, to improve the generalizability of our
5

---

image-conditioned model, generate triplanes in coordinate frame aligned with the conditioning view and render other images using poses relative to the conditioning one. We normalize the input view's pose in the same way LRM (Hong et al., 2023) during training, and specify the input view's pose in the same way too during inference.
Text Conditioning. To add text conditioning into our model, we adopt a strategy similar to that presented in Stable Diffusion (Rombach et al., 2022a). We use the CLIP text encoder (Radford al., 2021) generate embeddings and inject them into our denoiser using cross-attention. Specifically, we include an additional cross-attention layer after each self-attention block in the ViT and each cross-attention block the triplane decoder.
3.4 TRAINING AND INFERENCE
Training. During the training phase, uniformly sample time steps t within the range [1, [1, and add noise according to a cosine schedule. We sample input images with random camera poses. We also randomly sample additional novel viewpoints to supervise the renderings (as discussed in Sec. 3.1) for better quality. We minimize the following training objective with conditional signal y:
L = Be..cin.T,,ddeiei
f(1, ((E(,,, t, v).))))
(4)
For the image reconstruction loss f(-,)) -). we use a combination of L2 loss and LPIPS loss (Zhang al., 2018), with loss weights being and 2, respectively.
Inference. For inference, we select four viewpoints that uniformly surround the object in circle to ensure good coverage of the generated 3D assets. We fix the camera Field-of- Views to 50 degrees for the four views. Since we predict triplane NeRF aligned with the conditioning image's camera frame, we also fix the conditioning image's camera extrinsics to have identity orientation and (0, 22 0) position, following the practice of LRM (Hong et al., 2023). We output the triplane NeRF from the final denoising step as the generated 3D model. We utilize DDIM (Song et al., 2020a) algorithm to improve the inference speed.
EXPERIMENTS
In this section, present extensive evaluation of our method. In partieular, we briefly describe our experiment settings (Sec. 4.1), compare our results with previous works (Sec. 4.2), and show additional analysis and ablation studies (Sec. 4.3).
4.1 SETTINGS
Implementation details. We use AdamW optimizer train our model with initial learning rate of -4, We also apply a warm-up of 3K steps and cosine decay on the learning rate, We train our denoiser with 256 x 256 imput images and render 128 x 128 image crops for supervision. To save GPU memory for NeRF rendering, we use the deferred back back-propagation technique (Zhang et al., 2022). Our final model is large transformer with 44 attention layers (counting all the self- and cross-attention layers in the encoder and decoder) outputting 64 x 64 x triplanes with 32 channels. We use 128 NVIDIA A100 GPUs train this model with batch size of per GPU for 100K steps, laking about 7 days. Since the final model takes a lot of resources, it is impractical for us to evaluate the design choices with this large model for our ablation study. Therefore, we also train small model that consists of 36 attention layers to conduct our ablation study. The small model is trained with 32 NVIDIA 1100 GPUs for 200K steps days). Please refer Tab. in the appendix for an overview of the hyper-parameter settings.
Datasets. Our model requires only multi-view posed images to train. We use rendered multi-view images of 7300 objects from the Objaverse (Deitke et al., 2023) dataset. For each object, we render 32 images under uniform lighting at random viewpoints with fixed 50 FOV, following the settings of LRM (Hong et al., 2023). To train text-to-3D model, we use the object captions provided by Cap3D (Luo et al., 2023), which covers a subset of ~660k objects. For image-condition (single-view reconstruction) model, we combine the Objaverse data with additional real captures of ~ 220k objects from the MVImgNet (Yu et al., 2023) dataset, enhancing our model's generalization to out-of-domain inputs (see Fig. 7). We preprocess the MVImgNet dataset in the same way as

---

Table 1: Evaluation Metrics of single-image 3D reconstruction ABO and GSO datasets.
ABO dataset
GSO dataset
FID
CL.IP
PSNR 1
IPIPS
CD
FID
CLIPP
PSNR
LPIPS 4
CD
Point-B
112.29
0.806
17.03
0,363
0.127
123.70
0.741
15.60
0.308
0.099
Shap-E
79.80
0.864
15.29
0,331
0.097
97.05
0.805
14.36
0.289
Zero-1-to-3
31.59
0.194
32.44
17.36
0,182
One-2-3-45
190.81
12.00
0,514
0,163
139.24
12.42
0.448
Magic123
34,93
0.928
18.47
0.180
0.136
34,06
0.901
18.68
0.159
0.113
Ours (S)
36.77
0,915
22.62
0.194
0.059
35.16
0.888
21.80
0.150
0.046
Ours
27.88
0.949
24.15
0.127
0.046
30.01
0.928
22.57
0.126
0.040
Input
Ours
Shap-E
Point-E
Magic123 One-2-3-45
Figure 4: Qualitative comparisons on single-image reconstruction.
LRM (Hong et al., 2023): for each capture, we crop out the object of interest for all views, remove the background, and normalize the cameras tightly fit the captured object into the box [-1, 1J3. In general, these datasets contain large variety of synthetic and real objects, allowing us to train generic category-free 3D generative model.
We evaluate our image-conditioned model on novel synthetic datasets, including 100 objects from the Google Scanned Object (GSO) (Downs et al., 2022) and 100 objects from the Amazon Berkeley Object (ABO) (Collins et al., 2022) datasets. This allows for direct comparison of single-view reconstruction with the groundtruth. For each object, select 20 views that uniformly cover an object from the upper hemisphere to compute metrics; we pick slightly skewed side view as input.
4.2 RESULTS AND COMPARISONS
Single-image reconstruction. We compare our image-conditioned model with previous methods, including Point-E (Nichol et al., 2022). Shap-E (Jun & Nichol, 2023), Zero-I-to- (Liu et al., 2023b), One-2-3-44 (Liu et al., 2023a), and Magic 123 (Qian et al., 2023), on single-image reconstruction. We evaluate the novel-view rendering quality from all methods using PSNR, LPIPS (Zhang et al., 2018), CLIP similarity score (Radford et al., 2021) and FID (Heusel et al., 2017), computed between

---

Ours
Shap-E
Point-E
'a bowl of vegetables'
'a rusty old car'
"a voselized doy"
Figure 5: Qualitative comparisons on Text-to-3D.
the rendered and GT images, In addition, we also compute the Chamfer distance (CD) for geometry evaluation, for which we use marching cubes extract meshes from NeRFs. Note that accurate quantitative evaluation of 3D generation remains a challenge in the field due to the generative nature of this problem; we use the most applicable metrics from earlier works to assess our model and baselines.
Tab. reports the quantitative results on the GSO and ABO testing sets respectively. Note that our models (even ours (S)) can outperforms all baseline methods, achieving the best scores across all metries for both datasets. Our high generation quality is reflected by the qualitative results shown in Fig. 4; our model generates realistic results with higher-quality geometry and sharper appearance details than all baselines.
In particular, the two-stage 3D DMs, Shap-E (3D encoder + latent diffusion) and Point-E (point diffusion points-to-SD regression), lead to lower-qualit, 3D assets, often with incomplete shapes and blurry textures; this suggests the inherent difficulties in denoising 3D points or pretrained 3D latent spaces, a problem our model avoids. On the other hand, 7ero-1-to-3 leads to better quantitative results than Shap-E and Point-E appearnce, because it's 2D diffusion model finetuned from the pretrained Stable Diffusion (Rombach et al., 2022b) to generate novel-view images. However, Zero-I-to-3 alone cannot output 3D model needed by many 3D applications and their rendered images suffer from severe inconsistency across viewpoints. This inconsistency also leads to the low reconstruction and rendering quality from One-2-3-45, which attempts to reconstruct meshes from Zero-1-to-33 image outputs. On the other hand, the per-asset optimization -aased method Magic123 can achieve rendering quality comparable to Zero-1-to-3 while offering a 3D mdoel. However, these methods require long (hours of) optimization time and also often suffer from unrealistic Janus artifacts (see the high heels object in Fig. 4). In contrast, our approach is a single-stage model with 2D image training objectives and directly generates a 3D NeRF model (without per-asset optimization) while denoising multi-view diffusion. Our scalable model learns strong data priors from massive training data and produces realistic 3D assets without Janus artifacts. In general, approach leads to fast 3D generation and state-of-the single-image 3D reconstruction results.
Text-to-DD. We also evaluate our text-to-3D genera- tion results and compare with 3D diffusion models Shap-E (Jun & Nichol, 2023) and Point-E (Nichol et al., 2022), that are also category-agnoste and sup- port fast direct inference. For this experiment, we use Shap-E's 50 text prompts for the generation, and evaluate the results with CLIP precisions (Jain et al., 2022) and averaged precision using two different VIT models, shown in Tab. 2. From the table, we can see that our model achieves the best precision. We also show qualitative results in Fig. 5, in which our results clearly contain more geometry and appearance details and look more realistic than the compared ones.
Table 2: Evaluation Metries on Text-to-3D.
VIT-B/32
ViT-L/14
Method
R-Prec
AP
R-Prec
AP
Point-E Shap-E Ours
33.33
40.06
46.4
54.13
38.39
46.02
51.40
58.03
39.72
47.96
55.14
61.32
4.3 ANALYSIS, ABLATION, AND APPLICATION
We analyze conditoond model and verify design choices using small model architecture for better energy efficiency. Refer to Tab. 6 in the appendix for an overview of the hyper-parameter settings for this small model.
8

---

Table 3: Ablation on GSO dataset (DMV3D-S).
#Views
FID 4
CLIP
PSNR
S
LPIPS 4
CD J
4 (Ours)
35.16
0.888
21.798
0.852
0.150
0.0459
70.59
0.788
17,560
0.832
0.304
0.0775
47.69
0.896
20.965
0,851
0.167
0.0544
0.899
21.545
0.0454
W50 Novel
102.00
0.801
17.772
0.838
0.289
0.185
Wio Plucker
43.31
0. 0.883
20.930
0.842
0.185
0.0505
Input
Novel-view
Input
Novel-view
Figure 6: Robustness to out-of-domain inputs: synthetic (top left), real (bottom left, top right), and generated images (bottom right).
#Views. We show quantitative and qualitative comparisons of our models trained with different numbers (1, 2, 4, 6) of input views in Tab. 3 and Fig. 8. We can see that our model consistently achieves better quality when using more images. benefiting from capturing more shape and appearance information. However, the performance improvement of 6 views over four views is marginal, where some metrics (like PSNR, FID) from the 4-view model is even better. We therefore use four views as the default setting to generate all of our main results.
Multiple instance generation. Similar other DMs, our model can generate various instances from the same input image with different random seeds as shown in Fig. 1, demonstrating the diversity of our generation results. In general, we find the multiple instance results can all reproduce the frontal input view while containing varying shape and appearance in the unseen back side.
Input sources. Our model category-agnostic and generally works on various input sources as shown in many previous figures (Fig. 1,2,4). We show additional results in Fig, 6 with various inputs out of our training domains, including synthetic renderings, real captures, and generated images. Our method can robustly reconstruct the geometry and appearance of all cases.
Ablation of MVImgNet. We compare our models trained with and without the real MVImgNet dataset on two challenging examples. As shown in Fig. 7, we can see that the model without MVImgNet can lead to unrealistic flat shapes, showcasing the importance of diverse training data.
Ablations of novel-view supervision and Plucker rays. We compare with our ablated models including one trained without the novel-view supervision, and one without the Plucker ray conditioning (using the adaLN-Zero block conditioning instead). We can also see that the novel view rendering supervision is critical for our model. Without it, all quantitative scores drop by large margin due to that the model cheats by pasting the input images instead of reconstructing plausible 3D shapes. In addition, our design of Plucker coordinate-bel camera conditioning is also effective, leading to better quantitative results than the ablated model.
view-aligned planes
Application. The flexibility and generality of our method can potentially enable broad 3D) applications. One useful image editing application is to lift any objects in a 2D photo to 3D by segment them (using methods like SAM (Kirillov et al., 2023)) and reconstruct the 3D model with our method, as shown in Fig. and 2.
5 CONCLUSION
We present a novel single-stage diffusion model for 3D generation which generates 3D assets by denoising multi-view image diffusion. Our multi-view denoiser is based on a large transformer model (Hong et al., 2023), which takes noisy multi-view images to reconstruct clean triplane NeRF, outputting denoised images through volume rendering. Our framework supports text- and image-conditioning inputs, achieving fast 3D generation via direct diffusion inference without per- asset optimization. Our method outperforms previous 3D diffusion models for text-to-3D generation and achieves state-of-the-art quality on single-view reconstruction on various testing datasets.

---

Ethics Statement. Our generative model is trained the Objaverse data and MVImgNet data. The dataset (about 1M) is smaller than the dataset in training 2D diffusion models (about 100M to 1000M). The lack of data raise two considerations. First, it can possibly bias towards the training data distribution. Secondly, might not be powerful enough to cover all the vast diversity in testing images and testing texts. Our model has certain generalization ability but might not cover as much modes as the 2D diffusion model can. Given that our model does not have the ability to identify the content that out of its knowledge, it might introduce unsatisfying user experience. Also, our model can possibly leak the training data if the text prompt or image input highly align with some data samples. This potential leakage raises legal and security considerations, and is shared among all generative models (such as LLM and 2D diffusion models).
Reproducibility Statement. We provide detailed implementation of our training method in the main text and also provide the model configurations in Table 6 of the appendix. We will help resolve uncertainty of our implementation open discussions.
Acknowledgement. We would like to thank Nathan Carr, Duygu Ceylan, Paul Guerrero, Chun- Hao Huang, and Niloy Mitra for discussions about this project. We also thank Yuan Liu for providing testing images from Syncdreamer.
REFERENCES
Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: /096, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aven 3d generative adversarial networks. In JEEE Conf. Comput. Vis. Pattern Recog., 2022.
Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. Ini. Conf. Comput. Vis., 2023.
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Int. Conf. Comput. Vis., 2021.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference Computer Vision (ECCV), 2022.
Eric Ming Chen, Sidhanth Holalkere, Ruyu Yan, Kai Zhang, and Abe Davis. Ray conditioning: Trading photo-realism for photo-consistency in multi-view image generation. In Int. Conf. Comput. Vis., 2023a.
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single- stage diffusion nerf: A unified approach to 3d generation and reconstruction. arXiv preprint 2rXiv:2304.06714, 2023b.
10

---

Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang. Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. Abo: Dataset and benchmarks for real-world 3d object understanding. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 21 211222211111 136, 2022.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. IEEE Conf. Compus. Vis. Pattern Recog., pp. 13142-13153, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 14:8780-87944 2021.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pp. 2553 2560. IEEE, 2022.
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcie, and Sanja Fidler. Get3d: generative model of high quality textured shapes learned from images. Adv. Neural Inform. Process. Syst., 2022.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2110.00 0tti::2110009883 2021.
Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion. In Int. Conf. Mach. Learn., 2023.
Anchit Gupta, Wenhan Xiong, Yixin Nie, lan Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint (3t::230300537/1 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Adv. Neural Inform. Process. Syst., 2017.
Jonathan Ho and Tim Salimans. arXiv:2207.12598 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Adv Neural Inform. Process. Syst., 2020.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.00 1.04400, 2023.
Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Int. Conf. Comput. Vis., 2021.
Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. 2022.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023.
Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy Mitra. Holodiffusion: Training a 3d diffusion model using 2d images. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In Int. Conf. Learn. Represent., 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.

---

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In IEEE Conf. Comput. Vis. Pattern Recog.. 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Adv. Neural Inform. Process. Syst. 2021.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.0266 2023.
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model.
2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yi Yu Liu, and Tsung- Ying-Yiiinn Lin. Magic3d: High-resolution text-to-3d content creation. In JEEE Conf. Comput. Vis. Pattern Recog., pp. 300-309, 2023a.
Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung-Yr Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi. Vision transformer for nerf-based view synthesis from single input image. In IEEE Winter Conf. Appl. Comput. Vis., 2023b.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization, 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; shro-shot one image to 3d object. arXiv preprint arXiv: //328, 2023b.
Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. 2022.
Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. arXiv preprint arXiv:2306.07279, 2023.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Eur. Conf. Comput. Vis., 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding, ACM Trans. Graph., 41(4):102:1-002:15 July 2022. doi: 10.114557522233355017,, URL httph:ttt////...tt http://bbb..rro/0.114////3/332// 355222222.33 3530127.
Thu Nguyen-Phuce Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. In Int. Conf. Comput. Vis., 2019.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. IEEE Conf. Comput. Vis. Pattern Recog., 2021.
Evangelos Ntavelis, Aliaksandr Siarohin, Kyle Olszewski, Chaoyang Wang, Lue Van Gool, and Sergey Tulyakov. Autodecoding latent 3d diffusion models. arXiv preprint wxXiv:230.005445, 2023.
12

---

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T Barron, Amit H Bermano, Eric Ryan Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al. State of the art on diffusion models for visual computing. arXiv preprini arXiv:2310.07204, 2023.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One image to high-quality object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. 17843, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern 2ecog..202aa. 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the 1EEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022b.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In Adv. Neural Inform. Process. Syst., 2020.
Bokui Shen, Xinchen Yan, Charles Qi, Mahyar Najibi, Boyang Deng, Leonidas Guibas, Yin Zhou, and Dragomir Anguelov. Gina-3d: Learning to generate implicit neural assets in the wild. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 4913-4926, 2023.
Zifan Shi, Sida Peng, Yinghao Xu, Geiger Andreas, Yiyi Liao, and Yujun Shen. Deep generative models on 3d representations: A survey. arXiv preprint arXiv:2210. 15663, 2022.
Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34:19313-19322 2021.
Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka. Epigraf: Rethinking training of 3d gans. In Adv. Neural Inform. Process. Syst., 2022.
Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu. Jian Ren, Hsin-Ying Ying Lee, Peter Wonka, and Sergey Tulyakov. 3d generation on imagenet. In International Conference on Learning Representations, 2023. URL https: ptattps://ppereviiii http///ppp///eeeeeeeeee/eeeee net
13

---

Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020b.
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Viewset diffusion:(O-) image- conditioned 3d generative models from 2d data. arXiv preprint (xXiv:2306.0788/ 2023.
Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, al. Ad- vances in neural rendering. Computer Graphics Forum, volume 41, pp. 703-735. Wiley Online Library, 2022.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Ych, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774, 2022.
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin-Brualle Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi- view image-based rendering. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.162 2023.
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz. and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12422-222435 2023.
Yinghao Xu, Sida Peng, Ceyuan Yang, Yujum Shen, and Bolei Zhou. 3d-aware image synthesis via learning structural and textural representations. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Yinghao Xu, Menglei Chai, Zifan Shi, Sida Peng. Ivan Skorokhodov, Aliaksandr Siarohin, Ceyuan Yang. Yujun Shen, MiinnYYing Lee, Bolei Zhou, et al. Discoscene: Spatially disentangled generative radiance fields for controllable 3d-aware scene synthesis. Vis. Pattern Recog., 2023.
In JEEE Conf. Comput.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one few images. In IEEE Conf. Comput. Vis. Pattern Recog, 2021.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 9150-9161, 2023.
Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely. Arf: Artistic radiance fields, 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In CVPR, 2018
14

---

A APPENDIX
A.1 ROBUSTNESS EVALUATION.
We evaluate our model GSO (Downs et al., 2022) renderings that different camera Field-Of- Views (FOVs) and lighting conditions to justify its robustness. Specifically, while MVImgNet dataset include diverse camera FOVs and lighting conditions, the Objaverse renderings we are also trained on share a constant 50% FOV and uniform lighting. We evaluate the robustness of our image- conditioned model by testing images with other FOV angles and complex environmental lightings. As shown in Tab. 4, our model is relatively robust to the FOV of the captured images, though quality indeed drops when the actual FOV deviates more from the 50 FOV assume during inference (see Sec. 3.4). However, it exhibits lower sensitivity to lighting variations, leading to similar quality across different lighting conditions. When the lighting is non-uniform, our model bakes the shading effects into the NeRF appearance, yielding plausible renderings.
Table Robustness on GSO dataset.
Appearance
Geometry
Lighting/Fov
FID 4
CLIP f
PSNR
SSIM f
LPIPS
CD
Ours
30.01
0.928
22.57
0.845
0.126
0.0395
Fov10
35.69
0.912
19.136
0.820
0.207
0.0665
Fov30
32.309
0,921
20.428
0.839
0.166
0.0527
Fov70
32.095
0.921
20.961
0.860
0.154
0.0616
Fov90
34.438
0.912
19.952
0.855
0.190
0.0754
city
33.31
0.916
21.19
0,831
0.142
0.0437
night
36.32
0.907
20.383
0.829
0.161
0.0413
sunrise
33.264
0.917
21.080
0,843
0,140
0.0423
studio
36.32
0.927
21.383
0.839
0.141
0.0428
A.2 QUANTATIVE EVALUATION ON MVIMGNET.
MVImgNet (Yu et al., 2023) contains a diverse set of real data, which helps improve our generalization capabilities for real data out-of-domain data, demonstrated in Fig 7. We also perform quantative evaluation on the model with and without MVImgNet on the GSO dataset (Downs et al., 2022) in Tab. 5. The reconstructed results in terms of appearance and geometry are similar to the previous results only trained with Objaverse, indicating that MV WImgNet improves generalization without compromising the quality of reconstruction. We train both settings for an equal number of 100K iterations with exactly the same learning rate schedules and computes.
Input
w. MelmageNet
w.o. MvImageNet
Figure 7: Qualitative comparison of our model trained with and without MVImgNett.
Table 5: Ablation of MVImgNet.
Appearance
Geometry
#Views
FID 4.
CLIP
PSNR t
SSIM
LPIPS 4
CD 4
W: MvImageNet
30.01
0.928
22.57 21.85
0.845
0.126
0.0395
W.O MvImageNet
27.76
0.924
0.850
0.128
0.0378
15

---

A.3 IMPLEMENTATION DETAILS.
Our experiments are implemented in the PyTorch and the codebase is built upon guided diffu- sion (Dhariwal & Nichol, 2021). For the A optimizer, use a weight-decay 0.05 and beta (0.9,0.95) 0.95). Table 6 presents the detailed configuration of our various image-conditioned models. The architecture of the text-conditioned model closely mirrors that of the image-conditioned models, with the primary distinction being the approach to injecting the condition signal. For text- conditioned models, we employ the CLIP text encoder to derive text embeddings, integrating them into our denoiser through cross-attention layers. Specifically, in each transformer block within the encoder and decoder, a new cross-attention layer is introduced between the original attention and FFN. In such case, text-conditioned models consistently exhibit larger sizes than their image- conditioned counterparts, resulting in a slightly slower inference speed. During inference, we adopt classifier-fis -ree guidance approach Ho & Salimans (2022) with scale of 5 to generate 3D assets conditioned on text.
Small
Large
Image resolution
256x256
256x256
Encoder
Patch size
16
8
Att. Layers
12
12
Att. channels
768
768
Triplane tokens
x 32 3
32 x 3
Att. channels
768
1024
Decoder
Att. layers
24 (12a+12c)
32 (16a+16c) 2
Triplane upsample
1
Triplane shape
32 x 32 x 3 x 32
64 x 64 x 3 x 32
Rendering patch size
128
Ray-marching steps
48
128
Renderer
ML.P layers
10
10
MLP width
64
64
Activation
ReLU
ReLU
Times steps
1000
1000
Diffusion
Prediction target
Fp
ip
Schedule
cosine
cosine
Learning rate
4e-4
4e-4
Traininig
Optimizer
AdamW
AdamW
Warm-up steps
3000
3000
Batch size per GPU
8
#GPUS
32
128
Iterations
200
100K
Training time
4 days
7 days
Source
MV MVImgNet & Objaverse
MVIngNet & Objaverse
Dataset
Mixing ratio
1:3
1:3
Resolution
256
256
Table 6: Implementation details for our models. Att. denotes the attention. and e represents the self-attention and cross attention.
A.4 VIEW NUMBERS
We have compared the effects of using different numbers of views quantitatively in Tab. 3. Here, we also present qualitative results in Fig. When there is only one view, the predicted novel view very blurry. However, when the view number increases to four, the results become much clearer. When using six views, the improvement compared to four views not significant, consistent to the metrics reported in Tab. 3, indicating performance saturation. Therefore, model uses four views as the default configuration.
A.5 MORE COMPARISON.
We also include more qualitative comparison single-view image reconstruction in Fig. 9.
16

---

Input
#view 1
#view 2
#view
#view 6
Figure 8: Qualitative comparison on different view numbers.
Shap-E
Point-E
One-2-3-45
Magic 123
Ours
Figure 9: Qualitative comparison on single-image reconstruction.
17


Reviews:
[{"review_id": "oL0ZLtFk2x", "reviewer": "Reviewer_3SCV", "summary": "The paper proposes a 3D generation method that uses a transformer-based 3D large reconstruction model to denoise multi-view diffusion. The proposed method supports both text- and image-conditioned 3D generation. Experimental results seem promising.", "questions": "Please see the weakness above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of directly denoising a triplane-based NeRF is interesting. The result of multi-view diffusion is promising.", "weaknesses": "1. Does the method use a pre-trained stable diffusion model or train the DDPM from scratch? If from scratch, how is the generalization ability guaranteed?\n\n2. For multi-view diffusion, what is the number of views for training and inference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BJfGVhUBQy", "reviewer": "Reviewer_nAvZ", "summary": "This paper presents an approach to 3D generation via a single-stage diffusion model. By denoising multi-view image diffusion, the authors aim to generate realistic 3D assets. Central to this methodology is a large transformer model that processes multi-view noisy images to reconstruct a clean triplane NeRF, subsequently yielding denoised images through neural rendering. The proposed method showcases flexibility, supporting both text- and image-conditioning inputs, and claims rapid 3D generation without requiring per-asset optimization. The approach is evaluated and shown to be superior to previous 3D diffusion models in certain domains.", "questions": "- Given the inherent training characteristics of diffusion models, how does DMV3D achieve a training timeframe analogous to LRM? It would be helpful if the authors could provide an explanation.\n- It is important to clarify the extent of overlap between this work and the LRM submission in order to understand the distinctiveness of the contributions in this manuscript.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(+) The paper showcases impressive results in 3D generation compared to prior methods. \n\n(+) The method's ability to accommodate text- and image-conditioning inputs augments its versatility, making it potentially suitable for diverse applications.\n\n(+) The paper is well-structured and clearly explains both the methodology and foundational design choices.", "weaknesses": "Although the paper showcases promising results and a solid methodology; however, its level of novelty is unclear:\n\n- It appears that the proposition combines techniques that have been used before. The 3D diffusion part of the proposal seems to have been influenced by \"Viewset Diffusion (ICCV 2023)\", while the design and training approach of the large-scale transformer model is similar to \"LRM: LARGE RECONSTRUCTION MODEL FOR SINGLE IMAGE TO 3D\", which was also submitted to ICLR 2024.\n- Concerns have been raised about potential overlap with the LRM manuscript, questioning submission singularity.\n\nBesides, a balanced perspective is lacking due to the absence of discussion on the paper's limitations, which could provide valuable insights for potential areas of improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fmP5tKmVP1", "reviewer": "Reviewer_PYdg", "summary": "The paper presents a method to generate novel 3D objects based on a diffusion model that encloses a large reconstruction model. To leverage the strong generative power of 2D models while improving the 3D consistency of the generated objects, the diffusion model operates on the domain of multi-view images and internally learns a transformer-based reconstruction model to build a 3D representation, which is later rendered into denoised output images. In order for the model to generalize across different categories, the reconstruction model uses the DINO features to bootstrap the features used for deriving tokens. Results show that by training jointly on the Objaverse dataset and MVImageNet dataset, the model is able to generate diverse shapes, conditioned either on images or texts. The usage of the transformer-based large reconstruction model improves the 3D consistency while maintaining a good generation quality.", "questions": "- How the camera viewpoints are sampled during the training process? Would the reconstruction model easily fall into a local minima where the 3D results become trivial by generating planes that are parallel to the image planes?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written and well presented. The notations are clear and the illustrations are informative. It's easy to read and understand most of the technical details and design choices.\n- Though conceptually similar to, e.g., MVDiffusion and RenderDiffusion (or diffusion with forward models), the method elegantly combines the advantages of both works with the help of a generalizable large reconstruction model using transformers, hence lifting the previous constraints within only one single category.\n- The method naturally enables conditioning over images by fixing the diffusion variables, leading to a new scheme for bridging 2D and 3D domains.\n- The generated shapes are of high quality and surpass the baselines by a considerable margin.", "weaknesses": "- In contrast to Image-based diffusion models, the runtime efficiency might be compromised since the reconstruction model operates during every iteration of sampling. Investigating whether the reconstruction can be repurposed or distributed over the intermediate denoising phases could be insightful, especially since the current intermediate reconstructed model is discarded (would be great if they could be visualized), leading to potential wastage.\n\n- The textures produced lack sharpness. Exploring the proposed framework's performance on higher-resolution images and 3D triplanes would be intriguing. Additionally, employing a hybrid representation that decouples geometry and textures could yield enhanced results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0uAwO5JwiM", "reviewer": "Reviewer_X5Dv", "summary": "The paper proposes DMV3D, a 3D generation approach that uses a transformer-based 3D large reconstruction model to denoise multi-view diffusion. The reconstruction model incorporates a triplane NeRF representation and, functioning as a denoiser, can denoise noisy multi-view images via 3D NeRF reconstruction and rendering, achieving single-stage 3D generation in the 2D diffusion denoising process. The model is trained on large-scale multi-view image datasets of extremely diverse objects using only image reconstruction losses, without accessing 3D assets.", "questions": "1. As mentioned in weakness 1,  I would like to see some results of \"bunny seating on pancake\", this kind of generation. Even it is hard to do text to 3d, since the training set doesn't have compound objects, is it possible to do 2d conditioned 3d generation with this kind of prompt?\n\n2. The author mentioned in the 2d conditioned 3d generation task, they do not add noise to the reference view, however, some of other diffusion models usually also add noise to the reference view and each step, use the gt x0 of the that view and add new noise in ancestral sampling. The logic behind is the model is trained with noise images paired with the corresponding time step embedding, the clean image strategy will shock the model in inference. I wonder, in inference, if this clean ref image strategy can bring benefit over adding noise from x0.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The first contribution of this paper is to scale 3D diffusion generative models to very diverse categories and objects. Previous models such as DiffRF, etc. can only generalize within some shapenet like datasets with no more than 13 categories.\n\n2. The model demonstrates a novel method to conduct multiview diffusion. Instead of build attentions across views like mvdreamer or syncdreamer, they use attention to attend with learnable triplane tokens and with each other, therefore incorporating the 3D spatial prior in the process.\n\n3. The model shows good results of 3d generation, especially high quality geometry, which alwyas fail in SDS or nerf2nerf lines of works.", "weaknesses": "1.It seems the model learns from the objaverse and mvimagnet, which contain mostly single objects or separated objects. even the examples in out of domain results, in figure 6, the objects are not complicated as people use in SD-based models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)