MVDream: Multi-view Diffusion for 3D Generation
Authors: Yichun Shi
Peng Wang
Jianglong Ye
Long Mai
Kejie Li
Xiao Yang
Venue: ICLR 2024 poster (2024)
Keywords: Image Generation
3D Generation
Diffusion Model
Multi-view consistency
Primary area: generative models

Abstract: We introduce MVDream, a diffusion model that is able to generate consistent multi-view images from a given text prompt. Learning from both 2D and 3D data, a multi-view diffusion model can achieve the generalizability of 2D diffusion models and the consistency of 3D renderings. We demonstrate that such a multi-view diffusion model is implicitly a generalizable 3D prior agnostic to 3D representations. It can be applied to 3D generation via Score Distillation Sampling, significantly enhancing the consistency and stability of existing 2D-lifting methods. It can also learn new concepts from a few 2D examples, akin to DreamBooth, but for 3D generation.

MVDREAM:
MULTI-VIEW DIFFUSION FOR 3D GENERATION
Yichun Shi', Peng Wang', Jianglong ; Long Mai', Kejie Li', Xiao Yang' 1 ByteDance, USA, 2 University of California, San Diego {yichun. peng. wang, ke jiee 1i, xiax yangg @bytedance, com {j [jianglong yeh, mai. t. long88 88| Agmail.com
ABSTRACT
We introduce MVDream, a diffusion model that is able to generate consistent multi- view images from a given text prompt. Learning from both 2D and 3D data, a multi- view diffusion model can achieve the generalizability of 2D diffusion models and the consistency of 3D renderings. We demonstrate that such a multi-view diffusion model is implicitly generalizable 3D prior agnostic to 3D representations. It can be applied to 3D generation via Score Distillation Sampling, significantly enhancing the consistency and stability of existing 2D-lifting methods. It can also learn new concepts from a few 2D examples, akin to DreamBooth, but for 3D generation. Our project page is https: /MV-Dream. github.
INTRODUCTION
3D content creation is an important step in the pipeline modern game and media industry, yet is labor-intensive task that requires well-trained designers to work for hours days to create single 3D asset. A system that can generate 3D content in an easy way for pro-prffessionll users is thus of great value. Existing 3D object generation methods be categorized into three types: (1) template-basee generation pipeline, (2) 3D generative models, and (3) 2ID-lifting methods. Due to limited accessible 3D models and large data complexity, both template-based generators and 3D) generative models struggle to generalize effectively to arbitrary object generation. Their generated content is often confined common real-world objects with relatively simple topology and texture. Yet in industry, popular 3D assets usually come as a mixture of complicated, artistic, and sometimes non-reale non-realistic structures and styles (Ske).
Recently, 2D-lifting methods have shown that pre-trained 2D generation models can be potentially applied 3D generation. The typical representations are Dreamfusion (Poole al., 2023) and Magic3D (Lin et al., 2023a) systems, which utilize 2D diffusion models as supervision for the optimization of 3D representation via score distillation sampling (SDS). Trained on large-scale 2D) image datasets, these 2D models are able to generate unseen and counterfactual scenes whose details can be specified through a text input, making them great tools for creating artistic assets.
Nevertheless, in 2D-lifting techniques, challenges arise due to the lack of comprehensive multi- view knowledge or 3D-awareness during score distillation. These challenges encompass: (1) The multi-face Janus issue: The system frequently regenerates content described by the text prompt. (2) Content drift across different views. Examples can be seen in Fig. (1). The multi-face issuc can stem from various factors. For instance, certain objects, like blades, may be nearly invisible from some angles. Meanwhile, vital parts of a character or animal might be hidden or self-occluded from specific viewpoints. While humans assess these objects from multiple angles, 2D diffusion model cannot, leading it to produce redundant and inconsistent content.
In spite of all the weaknesses of 2D-lifting methods, we believe that large-scale 2D data is crucial to generalizable 3D generation. Therefore, we propose multi-view diffusion models, which can be used multi-view 3D prior agnostic to 3D representations. The proposed model simultaneously generates a set of multi-view view images that are consistent with each other. It can leverage pre-trained 2D diffusion models for transfer learning to inherit the generalizability. Then, by jointly training the model on multi-view images (from 3D assets) and 2D image-text pairs, we find that it can achieve both good consistency and generalizability. When applied to 3D generation through score distillation,
Work done during internship ByteDance

---

Multi-face Janus Problem
Content Drift Problem
Figure I: Typical multi-view consistency problems of 2D-lifting methods for 3D generation. Left: "A photo of a horse walking" where the horse has two faces. Right: "a DSLR photo of plate of fried chicken and waffles with maple syrup on them" where the chicken gradually becomes a waffle.
our multi-view supervision proves significantly more stable than that of single-view 2D diffusion models. And we can still create unseen, counterfactual 3D contents as from pure 2D diffusion models. Inspired by DreamBooth (Ruiz et al., 2023), we can also employ our multi-view diffusion model to assimilate identity information from a collection of 2D images and it demonstrates robust multi-view consistency after fine-tuning. Overall, model, namely MVDream, successfully generates 3D Nerf models without the multi-view consistency issue. either surpasses or matches the diversity seen in other state-of ootte-of-hhe-rrttt methods.
2 RELATED WORK AND BACKGROUND
2.1 3D GENERATIVE MODELS
The significance of 3D generation has driven the application of nearly all deep generative models to this task. Handerson er al. explored Variational Auto Encoders (Kingma & Welling, 2014) for textured 3D generation (Henderson & Ferrari, 2020; Henderson et al., 2020). However, their studies mainly addressed simpler models leveraging multi-view data. With Generative Adversarial Networks (GANs) yielding improved results in image synthesis, numerous studies have investigated 3D-aware GANs (Nguyen-Phuu Phaycnn-Phooccc et al., 2019; 2020; Niemeyer & Geiger, 2021; Deng et al., 2022; Chan et al., 2022; Gao al., 2022). These methods, attributed to the absence of reconstruction loss involving ground-truth 3D or multi-view data, can train solely on monocular images. Yet, akin to 2D GANs, they face challenges with generalizability and training stability for general objects and scenes. Consequently, diffusion models, which have shown marked advancements in general image synthesis, have become recent focal points in 3D generation studies. Various 3D diffusion models have been introduced for tri tri-planes (Shue et al., 2023; Wang et al., 2023b) or feature grids (Karnewar et al., 2023). Nevertheless, these models primarily cater specific objects like faces and ShapeNet objects. Their generalizability to the scope of their 2D counterparts remains unverified, possibly due to 3D representation constraints or architectural design. It is pertinent to note ongoing research endeavors to reconstruct object shapes directly from monocular image inputs (Wu et al., 2023; Nichol et al., 2022; Jun & Nichol, 2023), aligning with the increasing stability of image generation techniques.
2.2 DIFFUSION MODELS FOR OBJECT NOVEL VIEW SYNTHESIS
Recent research has also pursued the direct synthesis of novel 3D views without undergoing recon- struction. For instance, Watson et al. (2023) first applied diffusion models to view synthesis using the ShapeNet dataset (Sitzmann et al., 2019). Zhou & Tulsiani (2023) extended the idea to the latent space of stable diffusion models (Rombach et al., 2022) with epipolar feature transformer. Chan et al. (2023) enhanced the view consistency by re-projecting the latent features during diffusion denoising. Szymanowiez al. (2023) proposed multi-view reconstructer using unprojected feature grids. limitation shared across these approaches is the boundedness to their respective training data, with no established evidence of generalizing to diverse image inputs. Liu al. (2023) propose to fine-tune pre-traing trained image variation diffusion model (sdv) on an extensive 3D render dataset for novel view synthesis. Notwithstanding, the synthesized images from such studies still grapple with geometric consistency, leading to discernible blurriness in the output 3D models. Recently, Tang et al. (2023b) proposed a multi-view diffusion model for panorama with nomography-guiche attention, which different from ours where explicit 3D correspondence is not available.
2.3 LIFTING 2D DIFFUSION FOR 3D GENERATION
Given the bounded generalizability of 3D generative models, another thread of studies have attempted to apply 2D diffusion priors to 3D generation by coupling it with 3D representation, such
2

---

Resi oock
30 Se fAlle rdinn
CressAttonion
Training Loss
Multi view Generation
Generation
Distillation
Timestep
Camera
3D model
Rendered images
Multi-view Diffusion UNet
Figure 2: Illustration of the multi-view diffusion model. We keep the structure of text-to-image UNets by making two slight changes: (1) changing the self-attention from 2D to 3D for cross-view view connection (2) adding camera embeddings for each view. Multi-view renderings used to train the diffusion model. During testing, the pipeline used in a reverse way: the multi-view diffusion model serves 3D prior to optimize the 3D representation via Score Distillation Sampling (SDS).
NeRF (Mildenhall et al., 2021). The key technique of such methods is the score distillation sampling (SDS) proposed by Poole al. (2023), where the diffusion priors are used as score functions to supervise the optimization of 3D representation. Concurrent with Dreamfusion, SJC (Wang et al., 2023a) proposed a similar technique using the publicly available stable-diffusion model (Rombach et al., 2022). Following works have been further improving the 3D representations (Lin et al., 2023a; Tsalicoglou et al., 2023; Tang et al., 2023a; Chen et al., 2023), sampling schedules (Huang et al., 2023), and loss design (Wang et al., 2023c). Although these methods can generate photo-realistic and arbitrary types objects without training on any 3D data, they are known to suffer from the multi- view consistency problem, discussed in Section 1. In addition, discussed in (Poole et al., 2023), each generated 3D model is individually optimized by tuning prompts and hyper-parameters parameters to avoid generation failures. Nevertheless, in MVDream, we significantly improve the generation robustness, and are able to produce satisfactory results with a single set of parameters without individual tuning.
3 METHODOLOGG
3.1 MULTI-VIEW DIFFUSION MODEL
To mitigate the multi-view consistency issue in 2D-lifting methods, typical solution is to improve its viewpoint-awarenee For example, Poole et al. (2023) add viewpoint descriptions to text conditions. A more sophisticated method would be incorporating exact camera parameters like in novel view synthesis methods (Liu et al., 2023). However, we hypothesize that even perfect camera-conditions model is not sufficient to solve the problem and the content in different views still could mismatch. For example, an eagle might look to its front from front view while looking to its right from back view, where only its body is complying with the camera condition.
Our next inspiration draws from video diffusion models. Since humans do not have a real 3D sensor, the typical way perceive 3D object observe it from all possible perspectives, which is similar to watching turnaround video. Recent works on video generation demonstrate the possibility of adapting image diffusion models to generate temporally consistent content (Ho et al., 2022; Singer et al., 2022; Blattmann et al., 2023; Zhou et al., 2022). However, adapting such video models to our problem is non-trivial as geometric consistency could be more delicate than temporal consistency. Our initial experiment shows that content drift could still happen between frames for video diffusion models when there is a large viewpoint change. Moreover, video diffusion models are usually trained on dynamic scenes, which might suffer from domain gap when applied to generating static scenes.
With such observations, we found it important to directly train a multi-view diffusion model, where we could utilize 3D rendered datasets to generate static scenes with precise camera parameters. Fig, (2) shows an illustration of our text-to-min multi-view diffusion model. We leverage the 3D datasets to render consistent multi-view images to supervise the diffusion model training. Formally, given set of noisy image X, E IR' Fx X xw a text prompt as condition y, and a set of extrinsic camera parameters IR Fx16 multi-view diffusion model is trained generate set of images Xy T xxxxxxxxxx
3

---

Multi-view Diffusion Training
Multi-view Dreambooth
Image Diffusion Loss
Image Diffusion Loss
Imaga
French cat
Robot
a. [v] doy
Multi-view
Parameter
Diffusion
Preservation Loss
Training of multi-view diffusion with two modes; image mode with 2D attention (upper) and multi-view mode with 3D attention and camera embeddings (lower). Right: Training of DreamBooth, where the pre-trained multi-view diffusion model is fine-tuned with the image mode of 2D attention and preservation loss.
Figure The training pipeline of MVDream. Left:
of the same scene from F different view angles. After the training, the model be used multi-view prior for 3D generation with techniques such as Score Distillation Sampling (SDS).
To inherit the generalizability of the 2D diffusion models, would like keep their architecture as much as possible for fine-tuning, However, such models can only generate one image at time and do not take camera conditions inputs. So the main questions here are: (1) how generate of consistent images from the same text prompt (Sec. 3.1.1), (2) how to add the camera pose control (Sec. 3.1.2) and (3) how to maintain the quality and generalizability (Sec. 3.1.3).
3.1.1 MULTI-VIEW CONSISTENT GENERATION WITH INFLATED 3D SELF-ATTENTION
Similar to video diffusion models (Ho et al., 2022; Singer et al., 2022; Blattmann et al., 2023; Zhou et al., 2022), we would like to adapt the attention layers model the cross-view dependency while keeping the remaining network as 2D model that only operates within a single image. However, found that simple temporal attention fails to learn multi viel consistency and content drift still happens even if we fine-tune the model on a 3D rendered dataset. Instead, we choose to use a 3D attention. Specifically, different views in self-attention (See Fig. 3), which we found able to generate rather consistent images even when the view gap very large. Specifically, given a tensor of shape the we format it as B x FHW x C' for self-attention, where the second dimension is the sequence dimension representing the tokens. In such a way, we could also inherit all the module weights from original 2D seff-attention. Note that we also experimented with incorporating new 3D self-attention layer rather than modifying the existing 2D one. However, we found that such design compromised the generation quality of multi-view images.
inflate the original 2D self-attention layer into 3D by connecting all
F x W C.
3.1.2 CAMERA EMBEDDINGS
Like video diffusion models, position encoding is necessary for our model to distinguish between different views. To this end, we compared relative position encoding (Singer et al., 2022), rotary embeddings (Su et al., 2021), and absolute camera parameters. We found that embedding camera parameters with a 2-layer MLP leads to the most satisfying image quality with distinguishable view differences. Specifically, we consider two methods inject camera parameters: (1) adding camera embeddings to time embeddings as residuals, and (2) appending camera embeddings to text embeddings for cross attention. Our experiment shows that both methods work but the former turns out to be more robust possibly because the camera embeddings would be less entangled with the text.
3.1.3 TRAINING LOSS FUNCTION
We found that the details in data curation and training strategy are important to the image generation quality as well. For the space limit, we refer the readers to the Appendix for the details on data processing. For the training,
fine-tune our model from the Stable Diffusion v2.1 base model (512x512 resolution), where we keep their settings for the optimizers and e-prediction but reduce image size 256x 256. We found that joint training with larger scale text-to-image dataset is
4

---

helpful for the generalizability of the fine-tuned model, as illustrated in the left Fig. (3). Formally, given text-image dataset X' and multi-view dataset Xm.. for training samples {x, y. cj E U (c is empty for X), the multi-view diffusion loss is defined as
LMV (0, X, Xmv) = Ex,,,tttc [Ie - co c, 1)
where x; is the noisy image generated from random noise and images x, y is the condition, e the camera condition and the eg is the multi-view diffusion model. In practice, with a 30% chance we train the multi-view model as a simple 2D text-to-image model on a subset of LAION dataset (Schuhmann et al., 2022) by turning off the 3D attention and camera embeddings.
3.2 TEXT-TO-3D GENERATION
We recognize two ways to utilize multi-view diffusion model for 3D generation:
Using the generated multi-view images as inputs for few-shot 3D reconstruction method.
Using the multi-view diffusion model as a prior for Score Distillation Sampling (SDS).
Although more straightforward, 3D reconstruction requires robust few-shot 3D reconstruction method, which is not available at the moment during this project. Therefore, we focus our experiments on the latter, where we modify existing SDS pipelines by replacing Stable Diffusion model with our multi-view diffusion model. This leads to two modifications: (1) changing the camera sampling strategy and (2) feeding the camera parameters as inputs. Instead of using direction-annotatte prompts as in Dreamfusion (Poole et al., 2023), we use original prompts for extracting the text embeddings.
In spite that such multi-view SDS can generate consistent 3D models, the content richness and texture quality still fall short of those images directly sampled by the denoising diffusion process. Thus, we propose several techniques alleviate the issue. First, we linearly anneal the maximum and minimum time step for SDS during optimization. Second, to prevent the model from generating styles of low quality 3D models in the dataset, we add a few fixed negative prompts during SDS. Finally, to alleviate the color saturation from large classifier free guidance (CFG), we would like to apply clamping techniques such as dynamic thresholding (Saharia et al., 2022) or CFG rescale from (Lin et al., 2023b), Since these tricks only apply to No, we propose to use an xe-reconstrtction loss instead of original SDS formulation:
LSDS(Ø, x =(()) Er,., ||x
xo113
(2)
It can be shown that Eq. (2) is equivalent to the original SDS with w(t), hyper-parameter in SDS, equal to signal-to-noise ratio (See Appendix). Here x = (()) refers to rendered images from 3D representation ó and &n is the estimated Xo from e y, c, f). whose gradient is detached. Empirically, we found that xo-reconstruction loss performs similarly to the original SDS but it mitigates the color saturation after we apply the CFG rescale trick (Lin et al., 2023b) on &p.
We also use the point lighting (Poole et al., 2023) and soft shading (Lin et al., 2023a) to regularize the geometry. For regularization loss, we only use the orientation loss proposed by (Poole et al.. 2023). We note that both of these two techniques only help to smooth the geometry and have little effect on the content in our case. We do not use any sparsity loss to force the separation between foreground the background but instead achieve it by replacing background with random colors. It is also worth noting that MVDream is theoretically combinable with other SDS variants such as SJC (Wang et al., 2023a) and VSD(Wang et al., 2023c), but this is out of the scope of this paper.
3.3 MULTI-VIEW DREAMBOOTH FOR 3D GENERATION
As shown in the right of Fig. (3), after training the multi-view diffusion model, we consider extending it a DreamBooth model (Ruiz et al., 2023) for 3D DreamBooth applications. Thanks to the generalization of the multi-view diffusion model, we found its multi-view ability can be maintained after fine-tuning Specifically, we consider two types of loss, an image fine-tuning loss and parameter preservation loss. Formally, let Xt indicates the set of identity images, our loss for DreamBooth is
CDB Kaa) = LLDM((Ko) + 10 0
(3)
No
5

---

(a) Temporal Atlention
(b) Additional 3D Self-Attention module
(c) Inflated 3D Self-Attentonn
Figure 4: Comparison between using different types attention modules for learning multi-view consistency discussed in Sec 3.1.1. Same random seed used for testing.
where LIDM is the image diffusion loss (Rombach et al., 2022), On is the initial parameter of original multi-view diffusion, Ng is the number of parameters, and is balancing parameter set to See Appendix for more training details.
To obtain a 3D model, we follow the process Sec. 3.2 by replacing the diffusion model with the DrcamBooth model. Note that original DreamBooth3D (Raj al., 2023) used three-stage optimization: partial DreamBooth, multi-view data generation, and multi-view DreamBooth. In comparison, method capitalizes the consistency of diffusion model and streamlines the process by training a multi-view (MV) DreamBooth model directly followed by 3D NeRF optimization.
4 EXPERIMENTS
To evaluate the multi-view diffusion models, we fine-tune the open-sourced stable diffusion 2. model (sta) the Objaverse dataset (Deitke et al., 2023) and LAION dataset (Schuhmann al., 2022) for experiments. We refer the readers to the Sec. A.6 for more training details. We evaluate our method on three tasks: (I) multi-view image generation for evaluating image quality and consistency (Sec. 4.1), (2) 3D (NeRF) generation with multi-view score distillation as main downstream task (Sec. 4.2), and (3) DreamBooth for personalized 3D generation (Sec. 4.3).
4.1 MULWI-VIEW IMAGE GENERATION
We first compare the three choices of attention module for modeling cross-view consistency: (1) ID temporal self-attention that is widely used in video diffusion models, (2) a new 3D self-attention module that is added onto existing model, and (3) inflating existing 2D self-attention module for 3D attention, as explained in 1.1.1. To show the difference between these modules, we trained the model with frames across 90 degree view change, which is closer video setting. We keep the image resolution at x12 as the original SD model in this experiment. As shown in Fig. (4), we found that even with such limited view angle change on static scenes, temporal self-attention still suffers from content drift. We hypothesize that this is because temporal attention can only interchange information between the same pixel from different frames while the corresponding pixels could be far away from each other in different views. Adding new 3D attention, on the other hand, leads to severe quality degradation without learning consistency. We believe this is because learning new parameters from scratch takes more training data and time, which is unsuitable for case. The proposed strategy of inflated 2D self-attention achieves the best consistency among all without losing generation quality. We note that the difference between these modules would be smaller if reduce the image size to 256 and the number of views to 4. However, to achieve the best consistency, we keep choice for the following experiments based our initial observations.

---

Model
FID
IS
CLIP
Validation set
N/A
12.90 + 0.66
30.12 + 3.15
Multi-view Diffusion (3D data only)
40.38
12.33 + 0.63
29.69 + 3.36
Multi-view Diffusion (3D L.AION 2D data)
39.04
12.97 + 0.60
30.38 + 3.50
Table 1: Quantitative evaluation on image synthesis quality. DDIM sampler is used for testing.
Zombie bust, terror, 123dsculpt, bust, zombie
Battletech Zeus with sword!, tabletop. miniature, bat- tletech, miniatures, wargames, 3d asset
Medieval House, grass, medieval, vines, farm, middle- age, medieval-house, stone, house, home, wood, medieval-decor, 3d asset
Isometric Slowpoke Themed Bedroom, fanart, poke- bedroom, assignment, isometric, pokemon3d, isometric-room, room-low-poly, 3d asset
An astronaut riding a horse
A bald eagle carved out of wood
dog wearing a black pirate hat
a DSLR photo of ghost cating a hamburger
Figure 5: Example images generated by our model using training and testing prompts. The first two show the synthesized images from training prompts. The bottom four examples are images generated using unseen prompts. See Appendix for more results.
In Table I, conduct quantitative comparison generation quality and text-image consistency, We randomly choose 1,000 subjects from the held-out validation set and generate 4-view images using the given prompts and camera parameters. Frechet Inception Distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al., 2016) are calculated to measure the image quality while the CLIP score (Radford et al., 2021) is used to measure the text-image consistency, which averaged across all views and subjects. Overall, our models can achieve a similar IS and CLIP score the training set, indicating good image quality and text-image consistency. We found that fine-tuning with 3D data alone could lead to deteriorated image quality. On the other side, combining 3D data with 2D dataset (LAION) for joint training mitigates the problem. Please also refer to Sec. A.4 and Fig. (12) for more qualitative examples for comparison.
In Fig. (5), we show a few qualitative examples from our model. It can be seen that our multi-view model can generate images from unseen prompts that are possibly counterfactual and in different style from training prompts. Like training, we append the text with " 3d asset" for generation.
4.2 3D GENERATION WITH MULTI-VIEW SCORE DISTILLATION
We compare our multi-vien SDS with existing text-to-3D methods. We implement our method in threestudio (thr) and use reproduced baselines in this framework for comparison. We consider the following baselines: dreamfusion-IF (Poole al., 2023), magic3d-IF (Lin et al., 2023a), text2mesh- IF (Tsalicoglou al., 2023), and prolificdreamer (Wang et al., 2023c). Note that the implementation in

---

0o
45
90"
135
00
45
90"
135
DreamPusdonsl
Magie-D--i
Churs
Ours(Nomm
A bulldog wearing black pirate hat
beautiful, intricate butterfly
Figure 6: Comparison of text-to-3D (NeRF) generation. See Appendix for more results.
original SDS
+ timestep annealing
+ neg prompts
+ rescale
Figure 7: Effects of different techniques that we apply in multi-view SDS. The input text is "a DSLR photo of a frog wearing sweater".
threestudio could be different from the original papers. For example, dreamfusion-II: magic3d-IF and text2mesh-IF use DeepFloyd (dee) as their guidance model while original ones use Imagen (Saharia al., 2022) and eDiff-I (Balaji al., 2022), which are private. Original dreamfusion uses Mip- NeRF 360 (Barron et al., 2022) for 3D representation, but all methods here (including ours) use multi-resolution hash-grid (Müller al., 2022) (except Tex12Mesh that uses SDF). We believe that these baselines represent the best re- re-implementation we could find. To test the performance comprehensively. we collected 40 prompts from various sources, including prior studies, internet prompts for image generation, and user inputs from 3D generation website (lum). As shown in Fig. (6), all baselines suffer from severe multi-view inconsistency. Magic3D performs relatively better by using DMTet Shen al. (2021) for second stage refinement. Prolificdreamer shows good texture quality for every view, but jointly they do not appear to be a reasonable 3D object. In comparison, the proposed method generates high quality 3D assets in much more stable way. In terms of running time, dreamfusion-Il text2mesh-IF and MVDream takes about 2 hours on single V100 GPU. Magic3D-IF-SD takes about 3.5 hours and prolificdreamer takes more than 10 hours.
Fig. (7) shows an ablation study the proposed techniques for score distillation introduced in Sec. 3.2. Adding time step annealing helps make the shape more complete. Adding negative
8

---

Photo of a [v] dog
Figure 9: Illustration of M MVDrcamBooth Results. From the imputs, show the generated multi-vier images given a description prompt at bottom. On the right, we show comparison NeRF rendered results from ours with and DreamBooth3D (Raj al., 2023). Notice ours perform better the object details such as furry skin. See Appendix for more results.
prompts significantly improves the visual style. Adding the CFG rescale further makes the texture color more natural.
To further validate the stability and quality of our model, we conducted user study on the generated models from 40 prompts. Each user given 5 rendered videos and their corre- sponding text input and is asked to select preferred 3D model among all. 914 feedbacks from 38 users were collected and result is shown in Fig. (8). On average, 78% users prefer model over others. That is, our model is preferred over the best of all baselines in most cases. We believe this is strong proof of the robustness and quality of the proposed method. Please the supplementary materials for more visual results.
proposed
78%
preffiddreame
magic3iffs
Figure 8: User study.
4.3 MULTI- DREAMBOOTH.
In Fig. (9), we compare 3D models generated from MVDream and DreamBooth3D (Raj et al., 2023). We found that our results have higher quality with better object details such as the curly hair and fur texture the dog. This is because during the NeRF training process with SDS loss, our MV DreamBooth diffusion models produce higher geometry consistency. We provide additional results in project page and supplementary materials.
DISCUSSION AND CONCLUSION
Conclusion In this paper, present the first multi-view diffusion model that is able generate set of multi-view images of an object from any given text. By fine-tuning a pre- pre-trained eexttto-image diffusion model mixture of 3D rendered data and large scale text-to-image data, our model is able to maintain the generalizability of the base model while achieving multi-view consistent generation. We show that the multi-view diffusion model can serve as good 3D prior and can be applied to 3D generation via SDS, which leads to better stability and quality than current open-sourced 2D lifting methods. Finally, the multi-view diffusion model can also be trained in few-shot setting for personalized 3D generation (multi-view DreamBooth).
Limitation We observe the following limitations of our current multi-view diffusion model. For the moment, the model can only generate images at resolution of 256x256 256, which is smaller than the 512x512 of the original stable diffusion. Also, the generalizability of our current model seems to be limited by the base model itself. For both aforementioned problems, by increasing the dataset size and replacing the base model with a larger diffusion model, such as SDXL (SDX). Furthermore, are affected by the rendered dataset. Although it can be alleviated by adding more style text prompts, also indicates that a more diverse and realistic rendering is necessary learn better multi-view diffusion model, which could be costly.
expect them to be solved
do observe that the generated styles (lighting, texture) of model

---

ETHICS STATEMENT
The multi-view diffusion model proposed in this paper aims to facilitate the 3D generation task that widely demanded in gaming and media industry. We do note that it could be potentially applied to unexpected scenarios such as generating violent and sexual content by third-parte party fine-tuning. Built upon the Stable Diffusion model (Rombach et al., 2022), it might also inherit the biases and limitations to generate unwanted results. Therefore, we believe that the images models synthesized using our approach should be carefully examined and be presented as synthetic. Such generative models may also have the potential to displace creative workers via automation. That being said, these tools may also enable growth and improve accessibility for the creative industry.
7 REPRODUCIBILITY STATEMENT
Our MVDream model is built from publicly available models and datasets, such as Stable Diffu- sion (Rombach et al., 2022), LAION (Schuhmann al., 2022) and Objaverse (Deitke et al., 2023). With all the implementation details provided, we believe it should be straightforward to re-produce the algorithm. Besides, we will release our code as well as model checkpoints publicly after the paper submission.
REFERENCES
stableddiffusion----bbs---
https://huginngaace wttpa//..o//tabbiliiieeeoeaeaaee Accessed: 2023-08-29. 9
stable-ddiffasloon-l--essss Accesessss....ssede
Sketchfab. Deepfloyd. D http:p///uugginnffe...////esss/s Accessed: Accesss 2023-08-25, 8
Accessed: 2023-08-30.
Luma.i.. http:///twt////./aaaaaa//aaa/aaa Accesss. 2023-08-25.
Stable diffusion image variation.
stable-di wtw.aaoo---mma---aaaaaaiiiooso
Stable diffusion 2.1 base.
stable-dif fusifffuinn2222 2---bbaae Accessed: 2023-07-14, 6, 17
Threestudio
Accessed: 2023-08-25. 18
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. urXiv:2211.01324 2022.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022. 8
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023. 3,
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awen 3d generative adversarial networks. In CVPR, 2022. 2
Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. GeNVS: Generative novel view synthesis with 3D-aware diffusion models. In arXiv, 2023. 2
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv:2303.1 /3873, 2023.
10

---

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: universe of annotated 3d objects. In CVPR, pp. 13 13142-13153, 2023. 6, 10, 17
Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In CVPR, pp. 10673-10683, 2022. 2
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: generative model of high quality 3d textured shapes learned from images. NeurlPS, 2022. 2
Paul Henderson and Vittorio Ferrari. Learning single-image 3d reconstruction by generative modelling of shape, pose and shading. International Journal of Computer Vision, 2020.
Paul Henderson, Vagia Tsiminaki, and Christoph H Lampert. Leveraging 2d data to learn textured 3d
mesh generation. In CVPR, 2020. 2
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NewrIPS, 2017. 7
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv:2210.02303, 2022. 3,4
Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. Dreamtime: An improved optimization strategy for text-to-3d content creation. arXiv:2306. /2422, 2023. 3
Heewoo Jun and Alex Nichol.
Shap-e:
Generating conditional 3d implicit functions.
arXiv:2305.024633, 2023. 2
Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy Mitra. Holodiffusion: Training 3d diffusion model using 2d images. In CVPR, 2023. 2
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014. 18
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023a. 1, 3, 5, 18
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. wwXiv:2305.0889) 2023b. 5, 17
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; Zero-shot one image to 3d object. arXiv:2303.11328, 2023. 2, 14
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Trans. Graph., 2022.
Thu Nguyen-Phuoe, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
Thu Nguyen-Phuoe, Christian Richardt, Long Mai, Yongliang Yang, and Niloy Mitra. Blockgan: Learning 3d object-aware scene representations from unlabelled images. NeurIPS, 2020. 2
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXivv2212.0885., 2022. 2
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In CVPR, 2021.
11

---

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXtv:2307.01952, 2023. 14
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023. 1, 7,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. arXiv:2303. 13508, 2023. 6, 9, 18
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 2, 3, 6, 10
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning eext-to-image diffusion models for subject-driven generation. In CVPR, 2023. 2, 5
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022. 8, 17
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NeurlPS, 2016.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurlPS, 2022. 5, 6, 10, 17
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. In NeurlPS, 2021.
J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In CVPR, 2023. 2
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv:2209. 2022. 3,4
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d-structtue-awwrt neural scene representations. NeurlPS, 32, 2019. 2
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv:2104.09864 2021. 4
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Viewset diffusion:(0-)) image- conditioned 3d generative models from 2d data, 2023. 2
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it- 3d: High-fidelity 3d creation from single image with diffusion prior. arXiv:2303. 14184, 2023a.
Shitao Tang, Fuayng Zhang, Jiacheng Chen, Peng Wang, and Furukawa Yasutaka. Mvdiffusion: Enabling holistic multi-view image generation with correspondence-awant diffusion. arXiv: 2307.01097, 2023b. 2
Christina Tsalicollou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv:2304 /2439, 2023. 3,
12

---

Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, 2023a. 3,
Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In CVPR, 2023b.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: figh-fidelity and diverse text-to-3d generation with variational score distillation. 12Xi::3305.16213, 2023c. 7,
Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. In ICLR, 2023. 2
Chao-Yuse Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari. Multiview compressive coding for 3d reconstruction. In CVPR, 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv:2211.1101 2022. 3,4
Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. In CVPR, 2023.
13

---

bulldog wearing black pirate hat A APPENDIX
A.1 COMPARISON WITH IMAGE-TO-3D METHOD
In Fig. (10), we compare our results with an state-of-the-art image-to-3D method, namely Zero123- XL (Liu al., 2023). In particular, we first generate image using the same prompt with SDXL (Podell et al., 2023), segment the object to remove background, and then provide the image as input to the image-to-3D system using Zero123-X: 23-XL in threestudio. We notice drawbacks of such pipeline compared to our text-to-min text-o-mulltivieee diffusion model:
1. ZeroI I 23-XI. is trained on 3D datasets only, whose objects are mostly aligned with simple poses (e.g. T-pose for characters). So the image-to-3D results tend be distorted if the input image has complicated poses and view angles that are different from 3D dataset. Specifically, when the object is geometrically spanned in the direction of the depth, the generated 3D tends to be too flat. This can be seen from the examples of eagle.
2. Zero 123-XL can hardly generate geometrically consistent and richly textured novel-view images, and therefore the score distilled objects tend to have less details, yielding a smooth, blurred and relatively uniform appearance the other sides different from the input view, as shown with the example of bulldog.
A.2 EFFECT OF NUMBER OF VIEWS
In Fig. (11), compare the SDS with diffusion models trained with different number of views. All the models are trained with the same settings except the number of views. Camera embeddings is used for all models. We found that the 1-view model, in spite of its camera-awarenest still suffer
Zeno13-X XX
hald cagle carved out of wond
Zati23-XL
Ours
Zerol
Ouns
beautiful, intricate butterfly
Figure 10: Comparison with sox-to-image-to-3D using SDXL Zero Zero123XX 123XL.
14

---

view
2 view
4 view (proposed)
an astronaut riding a horse
baby yoda in the style of Mormookiee
Figure 11: Comparison between SDS with diffusion models trained with different number of views. All models have camera embeddings as inputs.
3D data only
3D 2D (LAION) data
a bulldog wearing a black pirate hat
astronaut riding horse
Figure 12: Qualitative comparison between different multi-view diffusion models in Table 1. The same random seed is used for each row for comparison.
15

---

from Janus problem. partially due to imperfect camera annotations. The 2-view model greatly reduces the muulti-face problem, but it still could happen to some extent. Finally, the view model barely suffers from any multi- view consistency issues.
A.3 TRAINING WITH 2D DATA
Fig. (12) shows the qualitative comparison between training with and without 2D data for multi-view diffusion model. Empirically, we found that adding the 2D data has a clear improvement for the model generalizability. leading better image quality and text-image correspondence.
A.4 TRAINING WITH RANDOM CAMERA
In the main paper, model is trained for orthogonal views with the same clevation angle. Thus, question arises: can the method generalize to scenarios involving random camera views rather than orthogonal ones? We conduct such experiment and found that the proposed 3D self-attention network can indeed generate random multi-view images if trained on such data. In spite of worse quality compared to the orthogonal model, an interesting finding is that such random-view model able to generate much larger number of views during inference, as shown in Fig. (13), In our experiments, found it able generate 64 views and potentially more, even though it trained on 4 views.
Figure 13: Multi-view diffusion model trained on random views is able to generate more a larger number views than training (40 views for example here).
A.5 RECONSTRUCTION Loss FOR SDS
The original SDS loss does not have an explicit form, instead it defined by its gradient:
Ox VoLsDs(O, x g( = Er,,, (o(t))(e((x)))))) c, - OΦ
where e is the additional camera condition for our specific model, x; = ax TrE the sampled noisy image, and w(t) is a hyper-parameter that controls the loss weight at different time steps. a and oe are the signal and noise scale controlled by the noise schedule, where of + of Let us denote (x y. c, f) as for simplicity. Given estimated co predicted by the model, we can reversely estimate & by:
X, - -
Xn
04
0;X + OrE
0,60
(5)
Qi
0
X +
(e E9)
Ot
16

---

Algorithm I: Pseudocode for MVDream training
Data: X, Kmx for i 4- 1 to n - do
sample mode ~ (0, 1):
if mode 2 0.7 then
select a random 3D sample from Xmoi
4 random orthogonal views out of 32 views;
c t camera extrinsics;
else
< 4 random images from X';
c - Ø
y f text descriptions:
sample & ~ (0, 1000);
X, f- aadd_noise(x; 1);
forward_and _nd baddd_awddwd( x, X,,
So for the reconstruction loss Lsns Br,,,r (1100 x x1 is given by:
if we ignore dhe like in SDS, its gradient
0x
TaLsps(0,x=gll)) =(())) Er,,,e 2(xx 8t0)
(6)
20t (co - c) 0x
Et,c,
at
06.
which is equivalent to the original SDS with wit) 20, Thus, we can further adjust the 8o here with other tricks such as dynamic threshold (Saharia et al., 2022) or CFG rescale (Lin et al., 2023b).
A.6
IMPLEMENTATION DETAILS
A. DATA PREPARATION AND DIFFUSION MODEL TRAINING
We use the public Objaverse dataset (Deitke et al., 2023) as our 3D rendered dataset, which was the largest 3D dataset available by the time of this project. We simply use the names and tags of the objects as their text descriptions. Since this dataset is rather noisy, we filter the dataset with CLIP score to remove the objects whose rendered images are not relevant its name, which leads to about 350K objects at the end. For each object, we first normalize the object to be at the center within a bounding box between 0.5, 0.5]. and then choose a random camera fov between [15, 60) and elevation between [0, 30). The camera distance is chosen the object size (0.5) times NDC focal length with a random scaling factor between [0.9, 1.1]. ]. A random HDRI from blender used for lighting. 32 uniformly distributed azimuth angles used for rendering, starting from 0 degree. To increase the number of training examples, we render each object twice with different random settings. The rendered images are saved in RGBA format where the background is filled with random grayscale color during training.
During the training, we sample data batch with 70% chance from the 3D dataset and 30% chance from a subset of LAION dataset (Schuhmann et al., 2022). For the 3D dataset, 4 random views which are orthogonal to each other are chosen from the 32 views for training. The 4 camera parameters are normalized onto the sphere for inputs, i.c. it only represents the rotation. We fine-tune our model from the Stable Diffusion v2.1 base model (512x512 resolution) (sta), where we keep their settings for the optimizer and c-prediction. We use reduced image size of 256 256 and total batch size of 1,024 (4,096 images) for training and fine-tune the model for 50,000 steps. The training takes about 3 days on 32 Nvidia Tesla A 100 GPUs. To compensate for the visual style difference between the Objaverse and LAION dataset, we append the text " 3d asset" to the 3D data if the keyword "3d" is not in the prompt. For DreamBooth fine-tuning, learning-rate rate as 2e - 6, weight decay as 0.01 and a batch size of 4. A podedo-code for training is provided in Algorithm. 1.
train the model around for 600 steps with
17

---

A.6.2 SDS OPTIMIZATION
For multi-view SDS, implement our multi-view diffusion guidance in the threestudio (thr) library, which has implemented most state-of-the-art methods for text-to-3D generation under unified framework. We use the implicit-volume implementation threestudio as our 3D representation, which includes a multi-resolution hash-grid and MLP to predict density and RGB. For camera views, we sample the camera the same way as how we render the 3D dataset. See Sec. A.6.1 for more details. The 3D model is optimized for 10,000 steps with A optimizer (Kingma & Ba, 2014) at learning rate of 0.01. For SDS, the maximum and minimum time steps are decreased from 0.98 to 0.5 and 0.02, respectively, over the first 8,000 steps. We use rescale factor of 0.5 for the CFG rescale trick. The rendering resolution starts at 64 and is increased to x after the 5,000 steps. We also turn soft shading in (Lin et al., 2023a) after 5,000 steps. The background replaced with 50% chance. The SDS process takes about 1.5 hour on a Tesla V100 GPU with shading and hour without shading.
A.7 ADDITIONAL RESULTS
In Fig. (14), we show additonal image generation results from our model using the prompts from the training dataset. In Fig. (15) and (16), we show additional comparison results 3D generation using our method and baseline. In Fir. (17) we show additional results of DreamBooth using our method and DreamBooth3D (Raj et al., 2023).
18

---

Zombie bust, terror, 123dsculpt, bust, zombie
Battletech Zeus sword!, tabletop, miniature, tletech, miniatures, wargames
Medieval House, grass, medieval, vines, farm, middle- age. medieval-house stone, house, home, wood, medieval-docom
Isometric Slowpoke Themed Bedroom, fanart, poke- mon. bedroom, assignment, isometric, pokemon3d, isometric-room room-low-poly
Old Boot (Left), cloth, boot, old, substancepainter, sub- stance, character
Old Treasure Chest, chest, vintage, treasure, old, sub- stancepainter, substance
UCSF's Bear Mascot Sculpture, hear, lihrary, sanfran- cisco, photogramm, uesf kalmanovitz, bufano
DSLR Camera, photography, dslr, camera, noobie, box- modeling, maya
"Legolas" Custom Outfit, custom, free-model, fortnite-
Digital Wristwatch, wristwatch, timepiece, digital, watch
Figure 14:
Example images generated by our multiview diffusion model using prompts in training set. In each example, the top and bottom row are the real and synthesized images, respectively.
19

---

MapicAD-F-SD Dramnunsoo-FF
ProtiiDreanne Trutaah
Oun(Norma))
bulldog wearing a black pirate
DreanFosce--F
Magic5D-BB B0-3DD-BF-SS SD
RosMah
ProlifiDeaanaz
Ours(Normal)
Viking axe, fantasy, weapon, blender, 8k, HD
Figure 15: Comparison of 3D generation between basclines and our method.
20

---

MapicID-F-SD DrainFusioe-
ProtisDramne TouMaa
OuroNormal)
beautiful, intricate butterfly
DreamFossonIFF
MagicDD-BB B---5-----S-- SD
ProlifiDeaamaz
Ours(Normal)
mecha vampire girl chibi
Figure 16: Comparison of 3D generation between basclines and method.
21

---

Photo of a dog
wearing an umbrella
Draambewriitt
sitting an rainbow
Figure 17: MVDream DreamBooth Results.
22


Reviews:
[{"review_id": "k290mquWlr", "reviewer": "Reviewer_nQgK", "summary": "The paper proposes a multi-view diffusion model that is able to generate consistent multi-view images based on a given text prompt. Trained on both 2D and 3D data, the proposed method can have generalizability of 2D diffusion models and consistency of 3D renderings. Furthermore, the method can be applied to 3D generation via score distillation sampling.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to understand. The paper focuses on the consistency problem in using 2D diffusion models for 3D generation, which is neglected in previous methods. The proposed method is among the first to adopt the multi-view image generation setting instead of generating views separately. The experiential result seems promising.", "weaknesses": "1. How to ensure consistency across multi-view images? There is no particular 3D prior learned in the proposed method.\n\n2. what does re-using 2D self-attention mean? Does it mean inheriting the weights from a pre-trained model or just adopting the same architecture?\n\n3. The description of the training set should be included in the main paper rather than the appendix. \n\n4. Regarding multi-view consistency, it is also important to train the model only on the 3D dataset to see the effect of the utilization of two datasets.\n\n5. In Table 1, are the chosen 1000 subjects from the held-out testing set or a subset of the training set? Moreover, in terms of image quality, it would be better if compared with the current popular method such as zero-123.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8SOMRARgS3", "reviewer": "Reviewer_QGG2", "summary": "The paper proposed a way to finetune large 2D image diffusion models. The goal of finetuning is to allow the model to understand 3D objects.   Thus when we use the finetuned model with SDS loss, we can avoid the multi-face Janus issue. The training set is a mix of 3D multiview data and 2D image data. In the end, the authors show two main applications: 1) finetuning the model with dreambooth; 2) text-to-3d with SDS. The paper showed excellent results.", "questions": "See the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposed a multiview diffusion model. To reuse existing large 2d image datasets, the model is designed to be able to accept both single images and multiview images as input. \n1. The advantage of using both 3D and 2D data is also ablated in the appendix. This is an important observation. It also aligns with our conjecture: even with objaverse, the scale of 3D data is still limited. Using 2D datasets has a significant positive effect when training 3D networks.\n2. The trained model will be a valuable asset to this community. The authors proved this by showing how to apply SDS and dreambooth with the method. Thus I encourage the authors to release the rendering program and the training code in the future.", "weaknesses": "1. Did the authors consider rendering the images with large elevation? For example, what if we have camera poses at the top or bottom of the object? Will this give more information to score distillation?\n2. The main comparison in Fig. 6 is not fair. All other methods are optimization-based by reusing 2d image diffusion. However, the proposed method used additional datasets. A fair comparison would be zero123+SDS?\n3. Following the above comment, the trained model should be able to combine with SJC, VDS, and maybe other variants of SDS. This should be discussed somewhere in the main paper or the appendix.\n4. Some description of the training algorithm is still confusing. For example, \n+ what is the so-called \"3d self-attention\"? I assume this is done by reshaping FxHxWxC to (FxHxW)xC and then applying self-attention.\n+ 32 views are rendered during data preprocessing. How do the authors sample 4 views during training? Uniformly?\n+ I hope there will be an algorithm to describe the training process.\n5. Optimization time comparison. I would assume the time cost to generative a 3d model is similar to dreamfusion. But maybe with additional 3D information, the optimization would be faster? Or the same?\n6. Following the above comment, the method is trained with a large 3d dataset, however, the generating process still relies on an optimization process which is slow. This limits the potential usage of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LqIocTSXm8", "reviewer": "Reviewer_XNxK", "summary": "This paper approaches the task of text-conditioned 3D object generation. It addresses the issues of prior work, namely the multi-face Janus problem and content drift problem; while adding DreamBooth-style results. The method supervises SDS against multiple views, which are rendered from a video diffusion-inspired text-to-multi-view model, which is trained upon large scale 3D-text data. The results are significantly better than prior work, but the experiments are difficult to follow.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "The method shows a distinctive improvement over the state of the art.\n- It clearly reduces the multi-face janus problem. In several shown examples, it eliminates this problem.\n- It is selected as better than previous work at an extremely high rate: 78% vs. 4 other methods, combined.\n- Objects are clearly consistent across 360degree views, and do not suffer from content drift\n- Exhaustive results upon the project webpage make these conclusions clear\n\nUtilizing 3D assets to train a text-to-3D model for score distillation is a creative and effective strategy. The paper even shows impressive DreamBooth-style results", "weaknesses": "Summary: the qualitative performance of the methods in this paper are impressive, but the experimental presentation in writing is poor. I think many changes are needed, but the results are quite impressive so I am still positive about this paper.\n\nExperimental presentation is poor\n- What is the point of each experiment? There is no overview section to clarify this. After multiple reads, I think I can infer, but reading was challenging.\n- It is often very hard to figure out the point of an experiment or the details. For instance, upon first read, I was unclear at why Sec 4.1 was before 4.2, and if the main contribution of the paper was 4.1 and the distinction with 4.2. \n- It is interesting to use a text-to-multi-view model to train a text-to-3D model. How well would the multi-view diffusion model work to generate 3D directly?\n- There are generally lots of experimental details that need to be expanded. For instance, the attention module choice experiment is difficult to understand. Section 4.1 has the overview to the experiment and results in the same paragraph. The experiment isn’t fully specified, missing details such as what the caption is for Figure 4, or details of e.g. the 3D attention. The extent of this experiment is also one qualitative example, so it is hard to draw any confident conclusion.\n- Why is the train set used for evaluation in Table 1, as opposed to a held-out set? What is the significance of evaluating a change in batch size? The results don’t change significantly, and increasing batch size isn’t a technical contribution. It was also hard to understand what data was used until after more than one read of the paper. The following was also confusing: “Adding text-to-image dataset (LAION) further improves the CLIP score and IS.” Is the difference between “no 2D data” and “proposed” training on LAION? This isn’t clarified anywhere. Why is this a contribution? Metrics change very little.\n- Qualitative results in Figure 5 is a part of the same paragraph as the experiment in Table 4. Again, why is the training set shown? Is this the text-to-multi-view model or the text-to-3D model?\n- What is the significance of adding negative prompts, or CFG rescaling? The reasoning for these isn’t clarified anywhere\n- Dreambooth results are touched on briefly and seem disconnected from the rest of the paper. Why is this connected? Why does it work?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kejENBFwdl", "reviewer": "Reviewer_HRvB", "summary": "The paper proposes a novel method for performing text-to-3D generation using the score-distillation loss with a 2D diffusion model. The key insight is that using a diffusion model which is fine-tuned with multi-view data, and thus has some notion of view consistency, is able to provide better gradients in optimization and result in a better 3D representation consistent with the text prompt, avoiding classic text-to-3D artifacts such as the multi-face or Janus problem. The paper trains the score distillation diffusion model with multi-view data by augmenting it to produce 4 multi-view images of an object rather than a single image and uses rendered images from synthetic data as supervision. Using this method, the paper then demonstrates that using this diffusion model in score distillation enables higher quality generations, and importantly, removes the Janus problem as the gradients from each score-distillation rendered image are not biased towards the canonical \"front\" view of an object.", "questions": "1. Is the view dependent prompting used in DreamFusion completely removed in favor of using the camera position embedding in the multi-view diffusion model? Or is there still some notion of directionality included in the text prompt?\n\nOverall, I think that the paper is meriting of acceptance. The description of the method is clear, and the proposed contribution makes intuitive sense and seems to resolve the Janus problem and increase the quality of the text-to-3D representations. The evaluation is very thorough, compares to other state-of-the-art methods in this field with a number of qualitative results and metrics, including a user study. I think that deeply studying the robustness of the method, and showing that the proposed contribution fixes the Janus problem 100% of the time consistently would elevate the paper to be significantly stronger, and I'd be happy to increase my review score in that case.\n\n**Update after the author response**\n\nThank you for the additional clarifications. I do not have any more questions and choose to keep my score: weaknesses regarding hyperparameters and robustness have been addressed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "In my opinion, the strengths of the paper are:\n1. The paper is extremely clear, and the method is justified intuitively. I found the discussion on using the 2D diffusion model to generate images and then using these images directly to train a 3D representation to be very insightful, as this is a common baseline that naturally follows from the contribution of making the diffusion model partially 3D-aware.\n2. The generated results are extensive, and qualitatively, the quality of the generations is very high and it seems like the results avoid the Janus problem.\n3. The evaluations are comprehensive, using multiple sets of metrics, ablating potential baselines, and including a user study.", "weaknesses": "In my opinion, the weaknesses of the paper are:\n1. One potential major weakness is that the robustness of the method does not seem to be tested. In terms of the generated it results, it would be good to see multiple versions generated from the same prompt (i.e., corresponding to something like variance in quantitative metrics from various trained models). This is very important because text-to-3D is notoriously brittle, and it's important to understand if the proposed contribution removes the Janus problem 5% of the time, 50% of the time, or 100% of the time. Without this sort of comparison and extensive evaluation, it's impossible to judge the magnitude of the contribution.\n2. A minor weakness is that there seem to be a large number of hyperparameters where it's not entirely clear why their values were chosen. Why use a multi-view diffusion model outputting only 4 views instead of more? The justification for how camera parameters are embedded also seems a bit ad-hoc. I don't see this as a major weakness as they lead to good performance, but I wonder about how brittle the method is as a result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)