CAT3D: Create Anything in 3D with Multi-View Diffusion Models

Authors: ruiqi  gao
aleksander  hoły ński
philipp  henzler
arthur  brussee
ricardo  martin-brualla
pratul  srinivasan
jonathan t barron
ben  poole
google  deepmind
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 21:18:24

Abstract: Advances in 3D reconstruction have enabled high-quality 3D capture, but require a user to collect hundreds to thousands of images to create a 3D scene. We present CAT3D, a method for creating anything in 3D by simulating this real-world capture process with a multi-view diffusion model. Given any number of input images and a set of target novel viewpoints, our model generates highly consistent novel views of a scene. These generated views can be used as input to robust 3D reconstruction techniques to produce 3D representations that can be rendered from any viewpoint in real-time. CAT3D can create entire 3D scenes in as little as one minute, and outperforms existing methods for single image and few-view 3D scene creation.

Main paper: 
 introduction: 
 The demand for 3D content is higher than ever, since it is essential for enabling real-time interactivity for games, visual effects, and wearable mixed reality devices. Despite the high demand, high-quality 3D content remains relatively scarce. Unlike 2D images and videos which can be easily captured with consumer photography devices, creating 3D content requires complex specialized tools and a substantial investment of time and effort.Fortunately, recent advancements in photogrammetry techniques have greatly improved the accessibility of 3D asset creation from 2D images. Methods such as NeRF [1] , Instant-NGP [2] , and Gaussian Splatting [3] allow anyone to create 3D content by taking photos of a real scene and optimizing a representation of that scene's underlying 3D geometry and appearance. The resulting 3D representation can then be rendered from any viewpoint, similar to traditional 3D assets. Unfortunately, creating detailed scenes still requires a labor-intensive process of capturing hundreds to thousands of photos. Captures with insufficient coverage of the scene can lead to an ill-posed optimization problem, which often results in incorrect geometry and appearance and, consequently, implausible imagery when rendering the recovered 3D model from novel viewpoints.Reducing this requirement from dense multi-view captures to less exhaustive inputs, such as a single image or text, would enable more accessible 3D content creation. Prior work has developed specialized solutions for different input settings, such as geometry regularization techniques targeted for sparse-view reconstruction [4, 5] , feed-forward models trained to create 3D objects from single images [6] , or the use of image-conditioned [7] or text-conditioned [8] generative priors in the optimization process-but each of these specialized methods comes with associated limitations in quality, efficiency, and generality.In this paper, we instead focus on the fundamental problem that limits the use of established 3D reconstruction methods in the observation-limited setting: an insufficient number of supervising views. Rather than devising specialized solutions for different input regimes [8, 9, 7] , a shared solution is to instead simply create more observations-collapsing the less-constrained, under-determined 3D creation problems to the fully-constrained, fully-observed 3D reconstruction setting. This way, we reformulate a difficult ill-posed reconstruction problem as a generation problem: given any number of input images, generate a collection of consistent novel observations of the 3D scene. Recent video generative models show promise in addressing this challenge, as they demonstrate the capability to synthesize video clips featuring plausible 3D structure [10, 11, 12, 13, 14, 15] . However, these models are often expensive to sample from, challenging to control, and limited to smooth and short camera trajectories.Our system, CAT3D, instead accomplishes this through a multi-view diffusion model trained specifically for novel-view synthesis. Given any number of input views and any specified novel viewpoints, our model generates multiple 3D-consistent images through an efficient parallel sampling strategy. These generated images are subsequently fed through a robust 3D reconstruction pipeline to produce a 3D representation that can be rendered at interactive rates from any viewpoint. We show that our model is capable of producing photorealistic results of arbitrary objects or scenes from any number of captured or synthesized input views in as little as one minute. We evaluate our work across various input settings, ranging from sparse multi-view captures to a single captured image, and even just a text prompt (by using a text-to-image model to generate an input image from that prompt). CAT3D outperforms prior works for measurable tasks (such as the multi-view capture case) on multiple benchmarks, and is an order of magnitude faster than previous state-of-the-art. For tasks where empirical performance is difficult to measure (such as text-to-3D and single image to 3D), CAT3D compares favorably with prior work in all settings. 
related work: 
 Creating entire 3D scenes from limited observations requires 3D generation, e.g., creating content in unseen regions, and our work builds on the ever-growing research area of 3D generative models [16] . Due to the relative scarcity of 3D datasets, much research in 3D generation is centered on transferring knowledge learned by 2D image-space priors, as 2D data is abundant. Our diffusion model is built on the recent development of video and multi-view diffusion models that produce highly consistent novel views. We show that pairing these models with 3D reconstruction, similar to [17, 18] , enables efficient and high quality 3D creation. Below we discuss how our work is related to several areas of prior work.2D priors. Given limited information such as text, pretrained text-to-image models can provide a strong generative prior for text-to-3D generation. However, distilling the knowledge present in these image-based priors into a coherent 3D model currently requires an iterative distillation approach. DreamFusion [8] introduced Score Distillation Sampling (SDS) to synthesize 3D objects (as NeRFs) from text prompts. Research in this space has aimed to improve distillation strategies [19, 20, 21, 22 , 23], swap in other 3D representations [24, 25, 26, 27, 28] , and amortize the optimization process [29] . Using text-based priors for single-image-to-3D has also shown promise [30, 31, 32] , but requires a complex balancing of the image observation with additional constraints. Incorporating priors such as monocular depth models or inpainting models has been useful for creating 3D scenes, but tends to result in poor global geometry [33, 34, 35, 36] .2D priors with camera conditioning. While text-to-image models excel at generating visually appealing images, they lack precise control over the pose of images, and thus require a time-consuming 3D distillation process to encourage the 3D model to conform to the 2D prior. To overcome this limitation, several approaches train or fine-tune generative models with explicit image and pose conditioning [7, 37, 38, 39, 40, 41, 42] . These models provide stronger priors for what an object or scene should look like given text and/or input image(s), but they also model all output views independently. In cases where there is little uncertainty in what should appear at novel views, reasoning about generated views independently is sufficient for efficient 3D reconstruction [43] . But when there is some uncertainty exists, these top-performing methods still require expensive 3D distillation to resolve the inconsistencies between different novel views.Multi-view priors. Modeling the correlations between multiple views provides a much stronger prior for what 3D content is consistent with partial observations. Methods like MVDream [44] , ImageDream [9] , Zero123++ [45] , ConsistNet [46] , SyncDreamer [47] and ViewDiff [48] fine-tune text-to-image models to generate multiple views simultaneously. CAT3D is similar in architecture to ImageDream, where the multi-view dependency is captured by an architecture resembling video diffusion models with 3D self-attention. Given this stronger prior, these papers also demonstrate higher quality and more efficient 3D extraction.Video priors. Video diffusion models have demonstrated an astonishing capability of generating realistic videos [49, 50, 10, 12, 15, 13, 51] , and are thought to implicitly reason about 3D. However, it remains challenging to use off-the-shelf video diffusion models for 3D generation for a number of reasons. Current models lack exact camera controls, limiting generation to clips with only smooth and short camera trajectories, and struggle to generate videos with only camera motion but no scene dynamics. Several works have proposed to resolve these challenges by fine-tuning video diffusion models for camera-controled or multi-view generation. For example, AnimateDiff [52] LoRA fine-tuned a video diffusion model with fixed types of camera motions, and MotionCtrl [53] conditioned the model on arbitrary specified camera trajectories. ViVid-1-to-3 [54] combines a novel view synthesis model and a video diffusion model for generating smooth trajectories. SVD-MV [55] , IM-3D [17] and SV3D [55] further explored leveraging camera-controlled or multi-view video diffusion models for 3D generation. However, their camera trajectories are limited to orbital ones surrounding the center content. These approaches mainly focus on 3D object generation, and do not work for 3D scenes, few-view 3D reconstruction, or objects in context (objects that have not been masked or otherwise separated from the image's background).Feed-forward methods. Another line of research is to learn feed-forward models that take a few views as input, and output 3D representations directly, without an optimization process per instance [6, 56, 57, 58, 18, 59, 60] . These methods can produce 3D representations efficiently (within a few seconds), but the quality is often worse than approaches built on image-space priors.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of our method for certain categories, generalizing to different camera intrinsics, genearting large sets of consistent views, and requiremenet of manually constructed camera trajectories in the discussion section. We also describe limitations in the experiments section, e.g. image-to-3D for segmented objects having room for improvement. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 CAT3D is a two-step approach for 3D creation: first, we generate a large number of novel views consistent with one or more input views using a multi-view diffusion model, and second, we run a robust 3D reconstruction pipeline on the generated views (see Figure 3 ). Below we describe our multi-view diffusion model (Section 3.1), our method for generating a large set of nearly consistent novel views from it (Section 3.2), and how these generated views are used in a 3D reconstruction pipeline (Section 3.3). (2) run a robust 3D reconstruction pipeline on the observed and generated views. This decoupling of the generative prior and 3D reconstruction process results in efficiency improvements and reduced methodological complexity relative to prior work [7, 8, 42] , while also improving visual quality.c 7 G S B N G B R k Z a h g 5 q x V B f M z I 6 X j 2 L v h P P x O l q R S f z L I m J U c X g l Y U I + O h j 9 C d 7 y f 9 X r + V + K q S r p U E r O X 4 / G D 3 L 5 x I 3 H A i D G Z I 6 y L t 1 6 a 0 S B m K G X E R b D S p E Z 6 h C 1 J 4 V S B O d G n b V F 3 c 9 c g k r q T y R 5 i 4 R T c Z F n G t l 3 z s b 3 J k p v q y L 4 D X + j A S m L C t 1 + 0 c 6 a U P t g 0 G q p G S 6 W 1 4 U S m E L y V v q j e l p a J u D B F 4 l X v V s N j I O L Q y n l B F s G F L r y C s q C 8 / x l P k w x j f 8 C i C H M 0 I 8 j 9 i v B 1 B Q e Z Y c o 7 E x E L d 1 E R R X k t N X D E o C x i S k m p S 2 i S 1 r V G j l r d 5 M 7 c 2 y Z x N B s 7 L j f F y V 2 T l N p L 7 q M 5 T b + b k I Q v b j e I Y S o k Y v R C t 4 c 1 p R d k h 5 L m Z J m m S H b Y 2 x O o a 5 2 D T 6 b q R + 9 8 A a a a + / u 3 H / U g u t a / Y I K X k 3 B V p W Q z C E L W 1 K 8 J W z 8 P p W C 4 s V A 0 j B Z x Q T h a 1 g m F u g i 6 y D B q y M M H O j n p Z b a A n o k X o o b O 9 Y W 2 c W 0 X h M 6 L E 0 Z A 3 6 3 j h G z 3 J j pWe train a multi-view diffusion model that takes a single or multiple views of a 3D scene as input and generates multiple output images given their camera poses (where "a view" is a paired image and its camera pose). Specifically, given M conditional views containing M images I cond and their corresponding camera parameters p cond , the model learns to capture the joint distribution of N target images I tgt assuming their N target camera parameters p tgt are also given:EQUATIONModel architecture. Our model architecture is similar to video latent diffusion models (LDMs) [49, 11] , but with camera pose embeddings for each image instead of time embeddings. Given a set of conditional and target images, the model encodes every individual image into a latent representation through an image variational auto-encoder [61] . Then, a diffusion model is trained to estimate the joint distribution of the latent representations given conditioning signals. We initialize the model from an LDM trained for text-to-image generation similar to [62] trained on web-scale image data, with an input image resolution of 512 × 512 × 3 and latents with shape 64 × 64 × 8. As is often done in video diffusion models [50, 10, 11] , the main backbone of our model remains the pretrained 2D diffusion model but with additional layers connecting the latents of multiple input images. As in [44] , we use 3D self-attention (2D in space and 1D across images) instead of simple 1D self-attention across images. We directly inflate the existing 2D self-attention layers after every 2D residual block of the original LDM to connect latents with 3D self-attention layers while inheriting the parameters from the pre-trained model, introducing minimal amount of extra model parameters. We found that conditioning on input views through 3D self-attention layers removed the need for PixelNeRF [63] and CLIP image embeddings [64] used by the prior state-of-the-art model on few-view reconstruction, ReconFusion [7] . We use FlashAttention [65, 66] for fast training and sampling, and fine-tune all the weights of the latent diffusion model. Similar to prior work [10, 67] , we found it important to shift the noise schedule towards high noise levels as we move from the pre-trained image diffusion model to our multi-view diffusion model that captures data of higher dimensionality. Concretely, following logic similar to [67] , we shift the log signal-to-noise ratio by log(N ) towards the high signal-to-noise ratio region, where N is the number of target views. Similar shifts have been adopted empirically by [11, 10] . For training, latents of target images are noise perturbed while latents of conditional images are kept as clean, and the diffusion loss is defined only on target images. A binary mask is concatenated to the latents along the channel dimension, to denote conditioning vs. target images. To deal with multiple 3D generation settings, we train a single versatile model that can model a total of 8 conditioning and target views (N + M = 8), and randomly select the number of conditional views M to be 1 or 3 during training, corresponding to 7 and 5 target views respectively. The noise schedule is shifted based on 5 target views. See Appendix B for more model details.Camera conditioning. To condition on the camera pose, we use a camera ray representation ("raymap") that is the same height and width as the latent representations [38, 68] and encodes the ray origin and direction at each spatial location. The rays are computed relative to the camera pose of the first conditional image, so our pose representation is invariant to rigid transformations of 3D world coordinates. Raymaps for each image are concatenated channel-wise onto the latents for the corresponding image.Given a set of input views, our goal is to generate a large set of consistent views to fully cover the scene and enable accurate 3D reconstruction. To do this, we need to decide on the set of camera poses to sample, and we need to design a sampling strategy that can use a multi-view diffusion model trained on a small number of views to generate a much larger set of consistent views.Camera trajectories. Compared to 3D object reconstruction where orbital camera trajectories can be effective, a challenge of 3D scene reconstruction is that the views required to fully cover a scene can be complex and depend on the scene content. We empirically found that designing reasonable camera trajectories for different types of scenes is crucial to achieve compelling few-view 3D reconstruction. The camera paths must be sufficiently thorough and dense to fully-constrain the reconstruction problem, but also must not pass through objects in the scene or view scene content from unusual angles. In summary, we explore four types of camera paths based on the characteristic of a scene: (1) orbital paths of different scales and heights around the center scene, (2) forward facing circle paths of different scales and offsets, (3) spline paths of different offsets, and (4) spiral trajectories along a cylindrical path, moving into and out of the scene. See Appendix C for more details.Generating a large set of synthetic views. A challenge in applying our multi-view diffusion model to novel view synthesis is that it was trained with a small and finite set of input and output viewsjust 8 in total. To increase the total number of output views, we cluster the target viewpoints into smaller groups and generate each group independently given the conditioning views. We group target views with close camera positions, as these views are typically the most dependent. For single-image conditioning, we adopt an autoregressive sampling strategy, where we first generate a set of 7 anchor views that cover the scene (similar to [42] , and chosen using the greedy initialization from [69] ), and then generate the remaining groups of views in parallel given the observed and anchor views. This allows us to efficiently generate a large set of synthetic views while still preserving both long-range consistency between anchor views and local similarity between nearby views. For the single-image setting, we generate 80 views, while for the few-view setting we use 480-960 views. See Appendix C for details.Conditioning larger sets of input views and non-square images. To expand the number of views we can condition on, we choose the nearest M views as the conditioning set, as in [7] . We experimented with simply increasing the sequence length of the multi-view diffusion architecture during sampling, but found that the nearest view conditioning and grouped sampling strategy performed better. To handle wide aspect ratio images, we combine square samples from square-cropped input views with wide samples cropped from input views padded to be square.Our multi-view diffusion model generates a set of high-quality synthetic views that are reasonably consistent with each other. However, the generated views are generally not perfectly 3D consistent. Indeed, generating perfectly 3D consistent images remains a very challenging problem even for Input ReconFusion [7] CAT3D (ours) Ground Truth current state-of-the-art video diffusion models [70] . Since 3D reconstruction methods have been designed to take photographs (which are by definition perfectly consistent) as input, we modify the standard NeRF training procedure to improve its robustness to inconsistent input views.We build upon Zip-NeRF [71] , whose training procedure minimizes the sum of a photometric reconstruction loss, a distortion loss, an interlevel loss, and a normalized L2 weight regularizer. We additionally include a perceptual loss (LPIPS [72] ) between the rendered image and the input image. Compared to the photometric reconstruction loss, LPIPS emphasizes high-level semantic similarity between the rendered and observed images, while ignoring potential inconsistencies in low-level high-frequency details. Since generated views closer to the observed views tend to have less uncertainty and are therefore more consistent, we weight the losses for generated views based on the distance to the nearest observed view. This weighting is uniform at the beginning of the training, and is gradually annealed to a weighting function that more strongly penalizes reconstruction losses for views closer to one of the observed views. See Appendix D for additional details. 
experiments: 
 We trained the multi-view diffusion model at the core of CAT3D on four datasets with camera pose annotations: Objaverse [73] , CO3D [74] , RealEstate10k [75] and MVImgNet [76] . We then evaluated CAT3D on the few-view reconstruction task (Section 4.1) and the single image to 3D task (Section 4.2), demonstrating qualitative and quantitative improvements over prior work. The design choices that led to CAT3D are ablated and discussed further in Section 4.3.We first evaluate CAT3D on five real-world benchmark datasets for few-view 3D reconstruction. Among those, CO3D [74] and RealEstate10K [75] are in-distribution datasets whose training splits were part of our training set (we use their test splits for evaluation), whereas DTU [77] , LLFF [78] and the mip-NeRF 360 dataset [79] are out-of-distribution datasets that were not part of the training dataset. We tested CAT3D on the 3, 6 and 9 view reconstruction tasks, with the same train and eval splits as [7] .In Table 1 , we compare to the state-of-the-art for dense-view NeRF reconstruction with no learned priors (Zip-NeRF [71] ) and methods that heavily leverage generative priors such as ZeroNVS [42] Dataset [7] ).3-view 6-view 9-view Method PSNR ↑ SSIM ↑ LPIPS ↓ PSNR ↑ SSIM ↑ LPIPS ↓ PSNR ↑ SSIM ↑ LPIPS ↓ Harder Dataset Difficulty Easier ← ----------------------------------------→ RealEstate10K Zip-NeRF * [and ReconFusion [7] . We find that CAT3D achieves state-of-the-art performance across nearly all settings, while also reducing generation time from 1 hour (for ZeroNVS and ReconFusion) down to a few minutes. CAT3D outperforms baseline approaches on more challenging datasets like CO3D and mip-NeRF 360 by a larger margin, thereby demonstrating its value in reconstructing large and highly detailed scenes. Figure 4 shows the qualitative comparison. In unobserved regions, CAT3D is able to hallucinate plausible textured content while still preserving geometry and appearance from the input views, whereas prior works often produce blurry details and oversmoothed backgrounds.CAT3D supports the efficient generation of diverse 3D content from just a single input view. Evaluation in this under-constrained regime is challenging as there are many 3D scenes consistent with the single view, for example scenes of different scales. We thus focus our single image evaluation on qualitative comparisons (Figure 5 ), and quantitative semantic evaluations with CLIP [64] (Table 2 ). On scenes, CAT3D produces higher resolution results than ZeroNVS [42] and RealmDreamer [80] , and for both scenes and objects we better preserve details from the input image. On images with segmented objects, our geometry is often worse than existing approaches like ImageDream [9] and DreamCraft3D [81] , but maintains competitive CLIP scores. Compared to these prior approaches that iteratively leverage a generative prior in 3D distillation, CAT3D is more than an order of magnitude faster. Faster generation methods have been proposed for objects [6, 82, 83] , but produce significantly lower resolution results than their iterative counterparts, so they are not included in this comparison. IM-3D [17] achieves better performance on segmented objects with similar runtime, but does not work on scenes, or on objects in context.At the core of CAT3D is a multi-view diffusion model that has been trained to generate consistent novel views. We considered several model variants, and evaluated both their sample quality (on Input CAT3D (ours)RealmDreamer [80] ZeroNVS [42] ImageDream [9] DreamCraft3D [81] Baselines Figure 5 : 3D creation from single input images. Renderings of 3D models from CAT3D (middle row) are higher quality than baselines (bottom row) for scenes, and competitive for objects. Note that scale ambiguity amplifies the differences in renderings between methods. See supplemental website for additional comparisons. (numbers reproduced from [17] ). CAT3D produces competitive results to object-centric baselines while also working on whole scenes. (Note: shown images are 3D renders, and time for CAT3D was evaluated on 16 A100 GPUs.)in-domain and out-of-domain datasets) and few-view 3D reconstruction performance. We also compare important design choices for 3D reconstruction. Results from our ablation study are reported in Table 3 and Figure 6 in Appendix A and summarized below. Overall, we found that video diffusion architectures, with 3D self-attention (spatiotemporal) and raymap embeddings of camera pose, produce consistent enough views to recover 3D representations when combined with robust reconstruction losses.Image and pose. Previous work [7] used PixelNerf [63] feature-map conditioning for multiple input views. We found that replacing PixelNeRF with attention-based conditioning in a conditional video diffusion architecture using a per-image embedding of the camera pose results in improved samples and 3D reconstructions, while also reducing model complexity and the number of parameters. We found that embedding the camera pose as a low-dimensional vector (as in [37] ) works well for in-domain samples, but generalizes poorly compared to raymap conditioning (see Section 3.1).Increasing the number of views. We found that jointly modeling multiple output views (i.e., 5 or 7 views instead of 1) improves sample metrics -even metrics that evaluate the quality of each output image independently. Jointly modeling multiple outputs creates more consistent views that result in an improved 3D reconstruction as well.Attention layers. We found that 3D self-attention (spatiotemporal) is crucial, as it yields improved performance relative to factorized 2D self-attention (spatial-only) and 1D self-attention (temporalonly). While models with 3D self-attention in the finest feature maps (64 × 64) result in the highest fidelity images, they incur a significant computational overhead for training and sampling for relative small gain in fidelity. We therefore decided to use 3D self-attention only in feature maps of size 32 × 32 and smaller.Multi-view diffusion model training. Initializing from a pre-trained text-to-image latent diffusion model improved performance on out-of-domain examples. We experimented with fine-tuning the multi-view diffusion model to multiple variants specialized for specific numbers of inputs and outputs views, but found that a single model jointly trained on 8 frames with either 1 or 3 conditioning views was sufficient to enable accurate single image and few-view 3D reconstruction.3D reconstruction. LPIPS loss is crucial for achieving high-quality texture and geometry, aligning with findings in [17, 7] . On Mip-NeRF 360, increasing the number of generated views from 80 (single elliptical orbit) to 720 (nine orbits) improved central object geometry but sometimes introduced background blur, probably due to inconsistencies in generated content. 
discussion: 
 We present CAT3D, a unified approach for 3D content creation from any number of input images. CAT3D leverages a multi-view diffusion model for generating highly consistent novel views of a 3D scene, which are then input into a 3D multi-view reconstruction pipeline. CAT3D decouples the generative prior from 3D extraction, leading to efficient, simple, and high-quality 3D generation.Although CAT3D produces compelling results and outperforms prior works on multiple tasks, it has limitations. Because our training datasets have roughly constant camera intrinsics for views of the same scene, the trained model cannot handle test cases well where input views are captured by multiple cameras with different intrinsics. The generation quality of CAT3D relies on the expressivity of the base text-to-image model, and it performs worse in the cases where scene content is out of distribution for the base model (e.g. human faces, since the base model was trained on limited human data). The number of output views supported by our multi-view diffusion model is still relatively small, so when we generate a large set of samples from our model, not all views may be 3D consistent with each other (see Supplementary website). Finally, CAT3D uses manually-constructed camera trajectories that cover the scene thoroughly (see Appendix C), which may be difficult to design for large-scale open-ended 3D environments.There are a few directions worth exploring in future work to improve CAT3D. The multi-view diffusion model may benefit from being initialized from a pre-trained video diffusion model, as observed by [10, 17] . The consistency of samples could be further improved by extending the number of conditioning and target views handled by the model. Automatically determining the camera trajectories required for different scenes could increase the flexibility of the system.Here we conduct ablation studies over several important decisions that led to CAT3D. We consider several multi-view diffusion model variants, and evaluated them on novel-view synthesis with held-out validation sets (4k samples) from the training datasets (in-domain samples) and from the mip-NeRF 360 dataset (out-of-domain samples), as well as 3-view reconstruction on the mip-NeRF 360 dataset (out-of-domain renders). Unless otherwise specified, models in this section are trained with 3 input views and 5 output views, evaluated after 120k optimization iterations. Quantitative results are summarized in Table 3 . Then we discuss important 3D reconstruction design choices, with qualitative comparison in Figure 6 .Number of target views. We start from the setting where the model takes 3 conditional views as input and generates a single target view, identical to the ReconFusion baseline [7] . Results show that our multi-view architecture is favored when dealing with multiple input views, compared to the PixelNeRF used by [7] . We also show that going from a single target view to 5 target views results in a significant improvement on both novel-view synthesis and few-view reconstruction.Camera conditioning. As mentioned in Section 3.1, we compared two camera parameterizations, one is an 8-dimensional encoding vector fed through cross-attention layers that contains relative position, relative rotation quaternion, and absolute focal length. The other is camera rays fed through channel-wise concatenation with the input latents. We observe that the latter performs better across all metrics.Attention layers. An important design choice is the type and number of self-attention layers used when connecting multiple views. As shown in Table 3 , it is critical to use 3D self-attention instead of temporal 1D self-attention. However, 3D attention is expensive; 3D attention at the largest feature maps (64 × 64) is of 32k sequence length and this incurs a significant computation overhead during training and sampling with a marginal performance gain, especially for out-of-domain samples and renderings. We therefore chose to use 3D attention only for feature maps of size 32 × 32 and smaller.Multi-view diffusion model training. We compared the settings of training the multi-view diffusion models from scratch with initializing from a pre-trained text-to-image latent diffusion model. The latter performs better, especially for out-of-domain cases. We further trained the model for more iterations and observed a consistent performance gain up until 1M iterations. Then we fine-tuned the model for handling both cases of 1 conditional + 7 target views and 3 conditional + 5 target views (jointly), for another 0.4M iterations. We found this joint finetuning leads to better in-domain novel-view synthesis results with 3 conditional views, and out-of-domain results that are on-par with the previous model.3D reconstruction. We found perceptual distance (LPIPS) loss is crucial in recovering high-quality texture and geometry, similar to [17, 7] . We also compared the use of 80 views along one orbital path with 720 views along nine variably scaled orbital paths. In the Mip-NeRF 360 setting, increasing the number of views helps better regularize the geometry of central objects, but sometimes leads to blurrier textures in the background, due to inconsistencies in generated content.We initialize the multi-view diffusion model from a latent diffusion model (LDM) trained for text-toimage generation similar to [62] trained on web scale image datasets. See Figure 7 for a visualization of the model architecture. We modify the LDM to take multi-view images as input by inflating the 2D self-attention after every 2D residual blocks to 3D self-attention [44] . Our model adds minimal additional parameters to the backbone model: just a few additional convolution channels at the input layer to handle conditioning information. We drop the text embedding from the original model. Our latent diffusion model has 850M parameters, smaller than existing approaches built on video diffusion models such as IM-3D [17] (4.3B) and SV3D [55] (1.5B).We fine-tune the full latent diffusion model for 1.4M iterations with a batch size of 128 and a learning rate of 5 × 10 -5 . The first 1M iterations are trained with the setting of 1 conditional view and 7 as well as in the multi-view setting), we used the model with 3 conditional inputs and groups of 5 target outputs, selected by their indices in the camera trajectories.Anchor selection. For single-image conditioning, we first select a set of target viewpoints as anchors to ground the scene content. To select a set of anchor views that are spread throughout the scene and provide good coverage, we use the initialization strategy from [69] . This method greedily selects the camera whose position is furthest away from the already selected views. We found this to work well on a variety of trajectory-like view sets as well as random views that have been spread throughout a scene.Dealing with non-square images. While the multi-view latent diffusion model we trained only supports 512 × 512 square images, we found the model still performed well when padding non-square images to square. However, this method often reduces resolution, so we also run our model on a square-cropped version of the inputs, and then compose the square-cropped outputs with the edges from the padded outputs to create a different aspect ratio image.For the Zip-NeRF [71] baseline, we follow [7] to make a few modifications of the hyperparameters (See Appendix D in [7] ) that better suit the few-view reconstruction setting. We use a smaller view dependence network with width 32 and depth 1, and a smaller number of training iterations of 1000, which helps avoid overfitting and substantially speeds up the training and rendering. Our synthetic view sampling and 3D reconstruction process is run on 16 A100 GPUs. For few-view reconstruction, of the text-to-image model to be 3D self-attention (spatiotemporal). We remove the text embedding conditioning of the original model.we sample 128 × 128 patches of rays and train with a global batch size of 1M that takes 4 minutes to train. For single image to 3D, we use 32 × 32 patches of rays and a global batch size of 65k that takes 55 seconds to train. Learning rate is logarithmically decayed from 0.04 to 10 -3 . The weight of the perceptual loss (LPIPS) is set to 0.25 for single image to 3D and few-view reconstruction on RealState10K, LLFF and DTU datasets, and to 1.0 for few-view reconstruction on CO3D an MipNeRF-360 dataets.Distance based weighting. We design a weighting schedule that upweights views closer to captured views in the later stage of training to improve details. Specifically, the weighting is given by a Gaussian kernel: w ∝ exp -bs 2 , where s is the distance to the closest captured view and b is a scaling factor. For few-view reconstruction, b is linearly annealed from 0 to 15. We also anneal the weighting of generated views globally to further emphasize the importance of captured views. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We have not open sourced the code used in this work, but the datasets we used are all publicly available (Re10K, CO3D, MVImgNet, Objaverse).• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provide details on the diffusion model architecture in Appendix B, including learning rate and optimization hyperparameters, data mixtures, CFG masking probability, and more. The dataset splits were used and shared in previous work.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: For few-view reconstruction metrics, we follow prior work and do not report error bars. For single-image CLIP metrics we use standard deviation over prompts, as in prior work. Evaluation of 3D generative models is challenging, and better metrics are an active area of research.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We believe we have followed the Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: We currently discuss mostly the positive impacts of this work, but can add more acknowledgement of the potential for harm caused by 3D generative models if the reviewers believe it is useful. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: We do not plan to release these models and have not planned safeguards around the release. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the papers that created the datasets we used. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We produce 3D assets in the form of videos and 3D models that have been linked to in the supplementary material. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 
 
 Appendix:  


Decision: This paper was reviewed by four experts in the field.  Based on the reviewers' feedback, the decision is to recommend the paper for acceptance.  The reviewers did raise some valuable comments that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.    We congratulate the authors on the acceptance of their paper!

Reviews: review: paper_summary: paper_summary: The objective of this paper is to achieve single-view or few-view to 3D. The core of their method lies in a multi-image-based diffusion model that leverages 3D attention and raymap encoding for the camera poses.  This setup is different from concurrent work, IM-3D, which repurposes video generation model to achieve 3D, or ReconFusion, which iteratively refine novel view with diffusion model conditioned on PixelNeRF embeddings. Experiments are conducted against several competitive Baselines, like IM-3D and ReconFusion. Result-wise, their proposed method showcases a nice balance between quality and efficiency.
strength_weakness: strength_weakness: 1. It seems cumbersome to generate a large number of viewpoints(Line:174)  by doing anchor first, and enriching frames in between by repeated running CAT3D. Indeed, more view generated together from the multi-view diffusion is good. But running it iteratively will still produce inconsistencies over runs over different set of camera viewpoints, as is can be seen from Fig.6. 

2. The trajectory shape seems to influence the quality of the multi-view diffusion. This limits its generalizability to arbitrary/scattered image viewpoints towards the same scene target, which are common in daily life.
questions: questions: 1. What happens if we only use anchor views to run recon in step2? How the quality compared with the current setup?

2. What if we densify view through some existing video interpolation model (instead of re-run CAT3D)? Would be more efficient and better quality?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces CAT3D, a novel approach for generating 3D representations from a flexible number of input images. The authors tackle the challenge of limited input data, a common bottleneck for 3D reconstruction, by leveraging the power of multi-view diffusion models. Their method generates a collection of novel viewpoints consistent with the input, effectively transforming a sparse-view reconstruction problem into a more manageable dense-view scenario. These generated views are then fed into a robust 3D reconstruction pipeline based on a modified Zip-NeRF to produce the final 3D model.

The authors demonstrate impressive results on various benchmarks, showcasing CAT3D's ability to handle single images, sparse multi-view captures, and even text prompts as input. The method exhibits state-of-the-art performance on few-view 3D reconstruction tasks, outperforming existing methods in terms of both speed and accuracy on established datasets. While single-image 3D generation shows promise, the authors acknowledge the performance is not yet on par with leading methods specifically designed for that task, particularly for single objects. The paper presents a compelling advancement in 3D content creation by unifying different input modalities within a single framework and showcasing significant efficiency gains.
strength_weakness: strength_weakness: - While CAT3D aims to handle sparse inputs, its dependence on calibrated camera poses, presents a significant limitation. How does performance degrade with increasing sparsity and decreasing pose accuracy? 

- The paper relies on manually designed camera trajectories for novel view synthesis, which limits practicality and scalability. The authors briefly mention adapting trajectories based on scene characteristics but provide no concrete details. Developing an automated trajectory selection or optimization procedure, potentially guided by learned priors or scene understanding techniques, would significantly enhance the method's value and broader applicability.

- The paper, at times, overstates its accomplishments (e.g., "achieving state-of-the-art performance across nearly all settings") and does not adequately address its limitations. A more nuanced and critical self-assessment would strengthen the work.
questions: questions: - Could the authors provide a more quantitative assessment of how performance degrades with increasing pose noise or sparsity? 

- Have the authors considered incorporating an automatic trajectory optimization scheme within CAT3D? 

- While open-sourcing the code and trained models is ideal, could the authors at least elaborate on their plans for sharing their work and facilitating reproducibility? Providing more details about the training procedure and hyperparameters would also be beneficial.
limitations: limitations: The authors list several limitations, including the reliance on constant camera intrinsics during training, the expressiveness limits of the base text-to-image model, the small number of output views, and the need for manual camera trajectories. However, the discussion lacks concrete examples or quantifiable measures of the limitations.
paper_summary: paper_summary: In this paper, the authors propose a two-stage method for 3D creation. Specifically, they introduce a multi-view diffusion model to generate novel views given observed input views. Then using these views, they perform a robust 3D reconstruction using a Zip-NeRF variant. To generate consistent views, they also design a data-dependent sampling strategy. In addition, they also conduct extensive experiments to validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: - It is certain that using larger diffusion models can boost the performance. But it is interesting to showcase the improvement trend with increased diffusion models.
questions: questions: See the Weaknesses.
limitations: limitations: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: This paper proposed CAT3D, a pipeline that enabled the production of 3D representations from one or a few input views. CAT3D comprises a multi-view generation model to synthesize novel images from different viewpoints and a Zip-Nerf to achieve 3D reconstruction based on generated views. The 3D results shown in this paper are very impressive with high quality, and the authors provided sufficient experiments to verify the effectiveness of CAT3D.
strength_weakness: strength_weakness: 1. Most techniques have been proposed by previous works, including the raymap, Nerf with LPIPS loss (IM-3D). But I think this point is not the main issue of this paper, while CAT3D proves the scalability of combining all these techniques together.
2. Since ZIP-Nerf is utilized in CAT3D for dense 3D reconstruction, it is unclear whether the results shown in Figures 1, 2, 4, and 5 in the paper are derived from ZIP-Nerf or generated from the multi-view diffusion process. Are all these results exclusively from ZIP-Nerf? If not, it would be beneficial to provide more results directly from the diffusion model. This distinction is important, especially considering the potential inconsistencies noted in the limitations section.
3. It is not inappropriate to provide the inference efficiency with 16 A100 GPUs (Line573) only, which is not a regular setting for most users. The authors should provide the inference time with one GPU. More importantly, it is unclear whether the efficiency results from Table 2 are all fairly compared with one GPU or the same hardware condition.
4. CAT3D seems only being trained with 1cond+7target and 3cond+5target. Could it address the combination of arbitrary conditions and targets without fine-tuning? For example, 4cond+4target or 2cond+6target.
5. CAT3D enjoys good generalization, which is just trained on objaverse, CO3D, MVImgNet, and RealEstate10k. However, these datasets are all constrained (no background: objaverse; object-centric: CO3D, MVImgNet, most indoor scenes: RealEstate10k). How to confirm the generalization of full-model trainable StableDiffusion, especially for some text-to-image samples shown in the supplementary? This point is not mentioned in the main paper.
6. Unfortunately, the authors did not promise to release the code as shown in the checklist. Therefore, some implementation details should be further clarified as mentioned in the "Questions".
questions: questions: Some unclear implementation:

1. Lines 190-191 are unclear. How to deal with images with different aspect ratios during training and inference?
2. Missing details about the shift of the noise scheduler.
3. What is meant by "drop the text embedding"? Does it imply using an empty string "" as the input text for all samples, or does it completely remove all cross-attention layers? Additionally, how is stability maintained when all cross-attention layers are removed at the beginning of training?

Some other minor questions:
1. As mentioned in the paper, a mask channel is concated to the inputs to distinguish conditions and targets. Why not try to use SD-inpainting as the initialization to cover this task?
2. Could 3DGS be converged with fewer generated images? Generating 80/720 views is too costly in my opinion.
limitations: limitations: The authors discuss the limitations. However, no qualitative limitations are shown in the paper.

Comments: title: Thanks for the rebuttal
Thanks for the reply
Thanks for the reply
Thanks for the rebuttal
comment: Thanks for the clarification. After reading the rebuttal and other reviews, I decide to retain my score (accept). I believe the work deserves to be presented at NeurIPS.
Thanks for the detailed clarification! It is one solid work. I retained my score
Thank you very much!
Thank you very much!
Thanks for the rebuttal. My concerns have been addressed. So I retain my score.
Thanks for the reply. My concerns are all addressed. So I raise my score to 7.
Thanks for the reply. Please find our response below:

> It is interesting to see that CAT3D can be generalized to various condition and target views. Is this capability attributable to that no positional encoding is used in CAT3D? To my knowledge, video models with positional encoding fail to be extended to inference with arbitrary lengths.

While CAT3D does not use an embedding of time (e.g. positional encoding of the time index for each frame), it does use an embedding of camera pose (via the raymap). Unlike time embeddings where during training you only see a small but finite number of time embeddings, we see a much larger continuous set of pose embeddings which may aid in generalization. In video and language models, one still can get generalization depending on how you structure and interpolate the time embeddings (see e.g. [1, 2]).

[1] Chen, Shouyuan, et al. "Extending context window of large language models via positional interpolation."

[2] Kazemnejad, Amirhossein, et al. "The impact of positional encoding on length generalization in transformers."


> For the shift of the noise scheduler, I think the conclusion of using log(N) to shift the scheduler (N is the number of target images) is not convincing. Because when CAT3D is trained with 5 and 7 target views, it is just evaluated with N=5 in the scheduler. This only confirms that using large N helps multi-view training. The specific relationship between the number of views and the scheduler has not been thoroughly assessed in this paper, leaving the log(N) shifting here somewhat ambiguous. Furthermore, it's important to note that multi-view images should not simply be equated with high-resolution images, given that the overlap among multi-view images can vary greatly and is stochastic.

It is common practice in training video diffusion models to shift the noise schedule based on the number of target frames, to compensate for the amount of redundant information which may exist across pixels. This is similar to what is done when increasing a models spatial resolution. In reality, the amount of redundant information in a video is a function of the amount of camera and scene motion, i.e., how many pixels are similar or the same across frames, but the number of frames serves as a reasonable approximation. In practice, our model is adapted from a single image diffusion model, and therefore modifying the noise schedule from that base model is necessary, since supervising and predicting multiple frames strictly has more redundant information. And indeed, in practice, we found shifting noise (while training and sampling) improves the quality of results. It is true that we don’t dynamically adjust the schedule based on the number of target frames, we just use the same shift of log(5) for both 5 and 7 target frames, but we found that the difference between that shift in practice is small (e.g. the average LPIPS on the in-domain diffusion samples is 0.235 for shifting log(5) vs. 0.240 for shifting log(7)). The precise formula for log(N), while not explicitly defined in prior work, was something we derived approximately from the numerical noise schedule information provided in the video model literature [3,4]. We will add this detail to the paper. Future work on multi-view models may want to instead condition the model on logSNR and use different shifts when training and sampling with different numbers of frames.

[3] Blattmann, Andreas, et al. "Align your latents: High-resolution video synthesis with latent diffusion models."

[4] Blattmann, Andreas, et al. "Stable video diffusion: Scaling latent video diffusion models to large datasets." 


> I think that providing multi-view generation from diffusion before Nerf learning is very important to confirm the capacity limit of stablediffusion. Even if the model itself cannot be publicly released, sharing samples—such as 80 views generated from Stable Diffusion before NeRF optimization (in the gallery.html)—would greatly benefit the community. This would enable researchers to reproduce performance based on these multi-view images in their own NeRF optimizations.

Great point, we will include more samples alongside our NeRF results in the project page (and will also need to combine those videos with a serialized form of the camera pose trajectories).
Thanks for the rebuttal. Most concerns about implementation details have been addressed; however, I still have some questions that warrant further discussion.
1) It is interesting to see that CAT3D can be generalized to various condition and target views. Is this capability attributable to that no positional encoding is used in CAT3D? To my knowledge, video models with positional encoding fail to be extended to inference with arbitrary lengths.
2) For the shift of the noise scheduler, I think the conclusion of using log(N) to shift the scheduler (N is the number of target images) is not convincing. 
Because when CAT3D is trained with 5 and 7 target views, it is just evaluated with N=5 in the scheduler. This only confirms that using large N helps multi-view training. The specific relationship between the number of views and the scheduler has not been thoroughly assessed in this paper, leaving the log(N) shifting here somewhat ambiguous.
Furthermore, it's important to note that multi-view images should not simply be equated with high-resolution images, given that the overlap among multi-view images can vary greatly and is stochastic.
3) I think that providing multi-view generation from diffusion before Nerf learning is very important to confirm the capacity limit of stablediffusion. 
Even if the model itself cannot be publicly released, sharing samples—such as  80 views generated from Stable Diffusion before NeRF optimization (in the gallery.html)—would greatly benefit the community. This would enable researchers to reproduce performance based on these multi-view images in their own NeRF optimizations.