CAT3D: Create Anything in 3D with Multi-View Diffusion Models
Authors: Ruiqi Gao
Aleksander Holynski
Philipp Henzler
Arthur Brussee
Ricardo Martin Brualla
Pratul P. Srinivasan
Jonathan T. Barron
Ben Poole
Venue: NeurIPS 2024 oral (2024)
Keywords: 3D generation
Diffusion Models
3D reconstruction
Generative Models
Primary area: diffusion_based_models
TL;DR: CAT3D uses a multi-view diffusion model to create 3D scenes from any number of real or generated images.

Abstract: Advances in 3D reconstruction have enabled high-quality 3D capture, but require a user to collect hundreds to thousands of images to create a 3D scene. We present CAT3D, a method for creating anything in 3D by simulating this real-world capture process with a multi-view diffusion model. Given any number of input images and a set of target novel viewpoints, our model generates highly consistent novel views of a scene. These generated views can be used as input to robust 3D reconstruction techniques to produce 3D representations that can be rendered from any viewpoint in real-time. CAT3D can create entire 3D scenes in as little as one minute, and outperforms existing methods for single image and few-view 3D scene creation.

CAT3D: Create Anything in 3D with Multi-View Diffusion Models
Ruiqi Gao 1 Aleksander Holyúski'*
Philipp Henzler²
Arthur Brussee'
Ricardo Martin-
Pratul Srinivasan'
Jonathan T. Barron
Ben Poole
!Googe DeepMind
Google Research
*equal contribution
wabtin
Texi-loimmageto-3D
Real image to 3D
Sparse multí-view to 3D
Figure CAT3D enables 3D scene creation from any number of generated or real images.
Abstract
Advances in 3D reconstruction have enabled high-quality 3D capture, but require a user collect hundreds to thousands of images create a 3D scene. We present CAT3D, method for creating anything in 3D by simulating this real-world capture process with multi-view diffusion model. Given any number of input images and set of target novel viewpoints, our model generates highly consistent novel views a scene. These generated views can be used as input to robust 3D reconstruction techniques to produce 3D representations that can be rendered from any viewpoint in real-time. CAT3D can create entire 3D scenes in as little as one minute, and outperforms existing methods for single image and few-view 3D scene creation.
Introduction
The demand for 3D content is higher than ever, since it essential for enabling real-time interactivity for games, visual effects, and wearable mixed reality devices. Despite the high demand, high-qualie 3D content remains relatively scarce. Unlike 2D images and videos which can be easily captured with consumer photography devices, creating 3DD content requires complex specialized tools and substantial investment of time and effort.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Fortunately, recent advancements in photogrammetty techniques have greatly improved the accessibil- ity of 3D asset creation from 2D images. Methods such as NeRF [1]. Instant-NGP [2], and Gaussian Splatting [3] allow anyone to create 3D) content by taking photos of a real scene and optimizing rep- resentation of that scene's s underlying 3D geometry and appearance. The resulting 31D representation can then be rendered from any viewpoint, similar to traditional 3D assets. Unfortunately, creating detailed scenes still requires a labor-intensiie process of capturing hundreds to thousands of photos. Captures with insufficient coverage of the scene can lead to an ill-posed optimization problem, which often results in incorrect geometry and appearance and, consequently, implausible imagery when rendering the recovered 3D model from novel viewpoints.
Reducing this requirement from dense multi-view captures to less exhaustive inputs, such as a single image or text, would enable more accessible 3D content creation. Prior work has developed specialized solutions for different input settings, such as geometry regularization techniques targeted for sparse-view reconstruction [4, 5], feed-forward models trained to create 3D objects from single images [6], or the use of image-conditioned [7] or text-conditioned [8] generative priors in the optimization process- but each of these specialized methods comes with associated limitations in quality, efficiency, and generality.
In this paper, we instead focus on the fundamental problem that limits the use of established 3D reconstruction methods in the observation -emmiteto-- setting: an insufficient number of supervising views. Rather than devising specialized solutions for different input regimes [8, 9, 7], a shared solution is to instead simply create more observations envervations-cllapiiin collaassig the less- cosstranstai under-determined 3D creation problems to the fully-constrained, fully- fully-observed 3D) reconstruction setting. This way, we reformulate a difficult ill- ill-posed reconstruction problem as generation problem: given any number of input images, generate a collection of consistent novel observations of the 3D scene. Recent video generative models show promise in addressing this challenge, as they demonstrate the capability to synthesize video clips featuring plausible 3D structure [10, 11, 12, 13, 14, 15]. However, these models are often expensive to sample from, challenging to control, and limited to smooth and short camera trajectories.
Our system, CAT3D, instead accomplishes this through a multi-view diffusion model trained specifi- cally for novel-view synthesis. Given any number of input views and any specified novel viewpoints, our model generates multiple 31D-consistent images through an efficient parallel sampling strategy. These generated images are subsequently fed through a robust 3D reconstruction pipeline to produce a 3D representation that can be rendered at interactive rates from any viewpoint. We show that our model is capable of producing photorealistic results of arbitrary objects or scenes from any number of captured or synthesized input views in as little as one minute, We evaluate our work across various input settings, ranging from sparse multi-view captures to single captured image, and even just text prompt (by using a text-to-image model to generate an input image from that prompt). CAT3D outperforms prior works for measurable tasks (such as the multi-view capture case) on multiple benchmarks, and is an order of magnitude faster than previous state-of-the-ee For tasks where empirical performance is difficult to measure (such as text-to-3D and single image to 3D), CAT3D compares favorably with prior work in all settings.
2 Related Work
Creating entire 3D scenes from limited observations requires 3D generation, e.g., creating content in unseen regions, and work builds on the wver-growing research area of 3D generative models [16]. Due to the relative scarcity of 3D datasets, much research in 3D generation centered on transferring knowledge learned by 2D image-space priors, 2D data is abundant. Our diffusion model is built on the recent development of video and multi-view diffusion models that produce highly consistent novel views. We show that pairing these models with 3D reconstruction, similar to [17, 18), enables efficient and high quality 3D creation. Below we discuss how our work is related to several areas of prior work.
2D priors. Given limited information such as text, pretrained text-to-image models can provide a strong generative prior for text-to-3D generation. However, distilling the knowledge present in these image-based priors into coherent 3D model currently requires an iterative distillation approach. DreamFusion [8] introduced Score Distillation Sampling (SDS) to synthesize 3D objects (as NeRFs) from text prompts. Research in this space has aimed to improve distillation strategies [19, 20, 21, 22,

---

FPTI
Figure 2: Qualitative results (renders): CAT3D can create high-quality 3D objects or scenes from number of input modalities: input image generated by text-to-image model (top row), single captured real image (middle row), and multiple captured real images (bottom row).
3

---

23], swap in other 3D representations [24, 25, 26, 27, 28], and amortize the optimization process [29]. Using text-based priors for single-image-to-- has also shown promise [30, 31, 32], but requires complex balancing of the image observation with additional constraints. Incorporating priors such as monocular depth models or inpainting models has been useful for creating 3D seenes, but tends to result in poor global geometry [33, 34, 35, 36].
2D priors with camera conditioning. While text-to-image models excel generating visually appealing images, they lack precise control over the pose of images, and thus require a time-consuming 3D distillation process encourage the 3D model conform to the 2D prior. To overcome this limitation, several approaches train or fine-tune generative models with explicit image and pose conditioning [7, 37, 38, 39, 40, 41, 42]. These models provide stronger priors for what an object or scene should look like given text and/or input image(s), but they also model all output views independently. In cases where there is little uncertainty what should appear novel views, reasoning about generated views independently is sufficient for efficient 3D reconstruction [43]. But when there is some uncertainty exists, these op-performing methods still require expensive 3D distillation to resolve the inconsistencies between different novel views.
Multi-view priors. Modeling the correlations between multiple views provides much stronger prior for what 3D content is consistent with partial observations. Methods like MVDream [44], ImageDream [9]. Zerol23++ [45], ConsistNet [46], SyncDreamer [47] and ViewDiff [48] fine-tune -xt-tto-imaee models to generate multiple views simultaneously. CAT3D is similar in architecture to ImageDream, where the multi-view dependency is captured by architecture resembling video diffusion models with 3D self-attention. Given this stronger prior, these papers also demonstrate higher quality and more efficient 3D) extraction.
Video priors. Video diffusion models have demonstrated an astonishing capability of generating realistic videos [49, 50, 10, 12, 15, 13, 51], and are thought to implicitly reason about 3D. However, it remains challenging to use off-the-shelf video diffusion models for 3D generation for number of reasons. Current models lack exact camera controls, limiting generation to clips with only smooth and short camera trajectories, and struggle to generate videos with only camera motion but no scene dynamics. Several works have proposed to resolve these challenges by fine-tuning video diffusion models for camera-controled or multi-view generation. For example, AnimateDiff [52] LoRA fine-tuned a video diffusion model with fixed types of camera motions, and MotionCtrl [53]
conditioned the model on arbitrary specified camera trajectories. ViVid-I-tos [54] combines a novel view synthesis model and video diffusion model for generating smooth trajectories. SVD-MV [55],
IM-3D [17] and SV3D [55] further explored leveraging camera-controllle or multi-view video diffusion models for 3D generation. However, their camera trajectories
limited to orbital ones surrounding the center content. These approaches mainly focus on 3D object generation, and do not work for 3D scenes, few-view 3D reconstruction, objects in context (objects that have not been masked or otherwise separated from the image's 's background).
Feed-forward methods. Another line of research is to learn ferd-forward models that take a few views as input, and output 3D representations directly, without an optimization process per instance [6, 56, 57, 58, 18, 59, 60]. These methods can produce 3D representations efficiently (within a few seconds), but the quality often worse than approaches built on image-space priors.
3 Method
CAT3D two-step approach for 3D creation: first, we generate a large number novel views consistent with one or more input views using multi-view diffusion model, and second, we run a robust 3D reconstruction pipeline on the generated views (see Figure 3). Below we describe our multi-view diffusion model (Section 3.1), our method for generating a large set of nearly consistent novel views from it (Section 3.2), and how these generated views are used in a 3D reconstruction pipeline (Section 3.3).

---

Target
An Observed
Target View
Eneoder 0
Kay Chordinates
Baput Latents
Ray Coordinates Random Laicuts
Observed & Generated Views
and unobserved views
Multi-View Latent Diffusion Model
NeRF
Set of generated views
Geseruted Latenns
Imsge
Figure 3: Illustration of the method. Given many views, CAT3D creates 3D representation
of the scene in as little as one minute. CAT3D has two stages: (1) generate a large set synthetic views with a multi-view latent diffusion model conditioned the input views and target poses; (2) run robust 3D reconstruction pipeline on the observed and generated views. This decoupling of the generative prior and 3D reconstruction process results in efficiency improvements and reduced methodological complexity relative to prior work [7, 8, 42], while also improving visual quality.
3.1 Multi-View Diffusion Model
We train a multi-view diffusion model that takes single or multiple views of 3D scene input and generates multiple output images given their camera poses (where "a view" is paired image and its camera pose). Specifically, given M conditional views containing M images Iodd and their corresponding camera parameters p frodd the model learns capture the joint distribution N target images ['gt assuming their N target camera parameters s are also given:
p( (Iood/ cond
tgt
(I)
Model architecture. Our model architecture is similar to video latent diffusion models (LDMss) [49, 11], but with camera pose embeddings for each image instead of time embeddings. Given set of conditional and target images, the model encodes every individual image into a latent representation through an image variational auto-encoder [61]. Then, diffuusin model is trained estimate the joint distribution of the latent representations given conditioning signals. We initialize the model from LDM trained for rext-to-image generation similar to [62] trained web-scale image data, with an input image resolution of 512 x 512 x 3 and latents with shape 64 x 64 x As is often done in video diffusion models [50, 10, 11], the main backbone of our model remains the pretrained 2D diffusion model but with additional layers connecting the latents of multiple input images. As in [44], use 3D self-attention (2D in space and ID across images) instead of simple ID self-attention across images. We directly inflate the existing 2D self-attention layers after every 2D residual block of the original LDM connect latents with 3D self-attention layers while inheriting the parameters from the pre-trained model, introducing minimal amount of extra model parameters. We found that conditioning on input views through 3D self-attention layers removed the need for PixeINeRF [63] and CLIP image embeddings [64] used by the prior state-of-the-a model on few-view reconstruction, ReconFusion [7]. We use FlashAttention [65, 66] for fast training and sampling. and fine-tune all the weights of the latent diffusion model. Similar to prior work [10, 67], we found it important to shift noise schedule towards high noise levels as we move from the pre-trained image diffusion model to our multi-view diffusion model that captures data of higher dimensionality. Concretely, following the log signal-to-noise ratio by log( N) towards the high signal-to-noise ratio region, where N is the number of target views. Similar shifts have been adopted empirically
logic similar to [67], we shift
5

---

by [11, 10]. For training, latents of target images are noise perturbed while latents of conditional images are kept as clean, and the diffusion loss is defined only on target images. A binary mask is concatenated to the latents along the channel dimension, to denote conditioning vs. target images. To deal with multiple 3D generation settings, we train a single versatile model that can model a total of 8 conditioning and target views (N + M = 8), and randomly select the number of conditional views M to be 1 or 3 during training, corresponding to 7 and 5 target views respectively. The noise schedule is shifted based on 5 target views. See Appendix B for more model details.
Camera conditioning. To condition on the camera pose, we use a camera ray representation ("raymap") that is the same height and width as the latent representations [38, 68] and encodes the ray origin and direction at each spatial location. The rays are computed relative to the camera pose of the first conditional image, so our pose representation is invariant to rigid transformations of 3I) world coordinates. Raymaps for each image are concatenated channel-wise onto the latents for the corresponding image.
3.2 Generating Novel Views
Given a set of input views, our goal is to generate a large set of consistent views to fully cover the scene and enable accurate 3D reconstruction. To do this, we need to decide on the set of camera poses sample, and we need to design sampling strategy that can use multi-view diffusion model trained on a small number of views to generate much larger set of consistent views.
Camera trajectories. Compared to 3D object reconstruction where orbital camera trajectories can be effective, a challenge of 3D scene reconstruction is that the views required to fully cover scene can be complex and depend on the scene content. We empirically found that designing reasonable camera trajectories for different types of scenes is crucial to achieve compelling few-view 3D reconstruction. The camera paths must be sufficiently thorough and dense fully-constrain the reconstruction problem, but also must not pass through objects in the scene or view scene content from unusual angles. summary, we explore four types of camera paths based on the characteristic of a scene: (1) orbital paths of different scales and heights around the center scene, (2) forward facing circle paths of different scales and offsets, (3) spline paths of different offsets, and (4) spiral trajectories along a cylindrical path, moving into and out of the scene. See Appendix C for more details.
Generating a large set of synthetic views. A challenge in applying our multi-view diffusion model to novel view synthesis that it was trained with small and finite set of input and output views- just 8 in total. To increase the total number of output views, we cluster the target viewpoints into smaller groups and generate each group independently given the conditioning views. We group target views with close camera positions, these views are typically the most dependent. For single-image conditioning, we adopt an autoregressive sampling strategy, where we first generate a set of 7 anchor views that cover the scene (similar to [42]. and chosen using the greedy initialization from [69]). and then generate the remaining groups of views in parallel given the observed and anchor views. This allows us to efficiently generate a large set of synthetic views while still preserving both long-range consistency between anchor views and local similarity between nearby views. For the single-image setting, we generate 80 views, while for the few-view setting we 480-960 views. See Appendix C for details.
Conditioning larger sets of input views and non-square images. To expand the number of views we can condition on, we choose the nearest M views as the conditioning set, as in [7]. We experi- mented with simply increasing the sequence length of the multi-view diffusion architecture during sampling, but found that the nearest view conditioning and grouped sampling strategy performed better. To handle wide aspect ratio images, combine square samples from square-cropeed input views with wide samples cropped from input views padded to be square.
3.3 Robust 3D reconstruction
Our multi-view diffusion model generates a set of high-quality synthetic views that are reasonably consistent with each other. However, the generated views are generally not perfectly 3D consistent. Indeed, generating perfectly 3D consistent images remains very challenging problem even for

---

Input
ReconFusion [7]
CAT3D (ours)
Ground Truth
Figure 4: Qualitative comparison, few-view reconstruction (renders). A comparison of rendered reconstructions on scenes from mip-NeRF 36 (top row) and CO3D) (bottom row), given input captured views. Compared to ReconFusion [7], CAT3D better aligns with ground-truth in seen regions, while hallucinating plausible content in unseen regions. See supplemental website for additional comparisons.
current state-of-the-art video diffusion models [70]. Since 3D reconstruction methods have been designed to take photographs (which are by definition perfectly consistent) as input, we modify the standard NeRF training procedure improve its robustness inconsistent input views.
We build upon Zip-NeRF [71], whose training procedure minimizes the sum photometric reconstruction loss, distortion loss, an interlevel loss, and normalized L2 weight regularizer. We additionally include a perceptual loss (LPIPS [72]) between the rendered image and the input image. Compared the photometric reconstruction loss, LPIPS emphasizes high-level semantic similarity between the rendered and observed images, while ignoring potential inconsistencies in low-level high-frequency details. Since generated views closer to the observed views tend to have less uncertainty and are therefore more consistent, we weight the losses for generated views based on the distance to the nearest observed view. This weighting uniform the beginning of the training, and is gradually annealed to a weighting function that more strongly penalizes reconstruction losses for views closer to of the observed views. See Appendix D for additional details.
Experiments
We trained the multi-view diffusion model at the core of CAT3D on four datasets with camera pose annotations: Objaverse [73], CO3D [74), RealEstate 10k [75] and MVImgNet [76]. We then evaluated CAT3D on the few-view reconstruction task (Section 4.1) and the single image to 3D task (Section 4.2), demonstrating qualitative and quantitative improvements over prior work. The design choices that led to CAT3D are ablated and discussed further in Section 4.3.
4.1 Few- View 3D Reconstruction
We first evaluate CAT3D on five real-world benchmark datasets for few-view 3D reconstruction. Among those, CO3D [74] and RealEstate10K [75] are in-distribution datasets whose training splits were part of our training set (we use their test splits for evaluation), whereas DTU [77], LLFF [78] and mip-NeRF 360 dataset [79] are out-of-distribution datasets that were not part the training dataset. We tested CAT3D on the 3, 6 and view reconstruction tasks, with the same train and eval splits [7]. In Table 1, we compare to the state-of-the-art for dense-view NeRF reconstruction with no learned priors (ZiR-NeRF) [71]) and methods that heavily leverage generative priors such as ZeroNVS [42]

---

Dataset Method
3-view
6-view
9-view
PSNR
SSIM t LPIPS PSNR
SSIM
LPIPS +
PSNR t
SSIM LPIPS 4
RealEstateeOK
Easier
Zip-NeRF" [71]
20.77
0.774
0.332
27.34
0.906
0.180
31.56
0.947
0.118
ZeroNVS [42]
19.11
0.675
0.422
22.54
0.744
0.374
23.73
0.766
0.358
ReconFusion [7]
25.84
0.910
0.144
29.99
0.951
0.103
31.82
0.961
0.092
CAT3D (ours)
26.78
0.917
0.132
31.07
0.954
0.092
32.20
0,963
0.082
L.FFF
Zip-NeRF [71]
17.23
0.574
0.373
20.71
0.764
0.221
23.63
0.830
0.166
ZeroNVS [42]
15.91
0.359
0.512
18.39
0,449
0.438
18.79
0,470
0.416
Dataser Difficulty
ReconFusion [7]
21.34
0.724
0.203
24.25
0.815
0.152
25.21
0.848
0.134
CAT3D (ours)
21.58
0.731
0.181
24.71
0.833
0.121
25.63
0,860
0.107
DTU
Zip-NeRF [71]
9.18
0.601
0.383
8.84
0.589
0.370
9.23
0.592
0.364
ZeroNVS [42]
16.71
0.716
0.223
17.70
0.737
0.205
17.92
0.745
0.200
ReconFusion [7]
20.74
0.875
0.124
23.62
0.904
0.105
24.62
0.921
0.094
CAT3D (ours)
22.02
0.844
0.121
24.28
0.899
0.095
25.92
0.928
0.073
CO3D
Zip-NeRF' [71]
14.34
0.496
0.652
14.48
0.497
0.617
14.97
0.514
0.590
7croNVS [42]
17.13
0.581
0.566
19.72
0.627
0.515
20.50
0.640
0.500
ReconFusion [7]
19.59
0.662
0.398
21.84
0.714
0.342
22.95
0.736
0.318
CAT3D (ours)
20.57
0.666
0.351
22.79
0.726
0.292
23.58
0.752
0.273
Mip-NeRF 360
Harder
Zip-NeRF" [71]
12.77
0.271
0.705
13.61
0,284
0.663
14.30
0.312
0.633
ZeroNVS" [42]
14.44
0.316
0.680
15.51
0.337
0.663
15.99
0.350
0.655
ReconFusion [7]
15.50
0.358
0.585
16.93
0.401
0.544
18.19
0.432
0.511
CAT3D (ours)
16.62
0.377
0.515
17.72
0.425
0.482
18.67
0.460
0.460
Table Quantitative comparison few-view reconstruction. CAT3D outperforms baseline approaches across nearly all settings and metrics (modified baselines denoted with taken from [7]).
and ReconFusion [7]. We find that CAT3D achieves of-te-of-the-ar performance across nearly all settings, while also reducing generation time from hour (for ZeroNVS and ReconFusion) down to few minutes. CAT3D outperforms baseline approaches on more challenging datasets like CO3D and mip-NeRF 360 by a larger margin, thereby demonstrating its value in reconstructing large and highly detailed scenes. Figure shows the qualitative comparison. In unobserved regions, CAT3D is able to hallucinate plausible textured content while still preserving geometry and appearance from the input views, whereas prior works often produce blurry details and oversmoothed backgrounds.
4.2 Single image to 3D
CAT3D supports the efficient generation of diverse 3D content from just a single input view. Evalua- tion this under-constrained regime challenging there are many 3D scenes consistent with the single view, for example scenes of different scales. We thus focus our single image evaluation on qualitative comparisons (Figure 5), and quantitative semantic evaluations with CLIP [64] (Table 2). On scenes, CAT3D produces higher resolution results than ZeroNVS [42] and RealmDreamer [80], and for both scenes and objects we better preserve details from the input image. segmented objects, our geometry is often worse than existing approaches like ImageDream [9] and DreamCraft3D [81], but maintains competitive CLIP scores. Compared to these prior approaches that iteratively leverage generative prior in 3D distillation, CAT3D is more than order of magnitude faster. Faster generation methods have been proposed for objects [6, 82, 83], but produce significantly lower resolution results than their iterative counterparts, they not included in this comparison. IM-3D [17] achieves better performance on segmented objects with similar runtime, but does not work scenes, or objects in context.
images with
4.3 Ablations
At the core of CAT3D a multi-view diffusion model that has been trained to generate consistent novel views. We considered several model variants, and evaluated both their sample quality (on

---

Input
CAT3D (ours)
RealmDreamer(80]
ZeroNVS [42]
ImageDream [9]
DreamCraft3D [81]
Baselines
Figure 5: 3D creation from single input images. Renderings of 3D models from CAT3D (middle row) are higher quality than baselines (bottom row) for scenes, and competitive for objects. Note that scale ambiguity amplifies the differences in renderings between methods. See supplemental website for additional comparisons.
Model
Time (min)
CLIP (Image)
Table 2: Evaluating image-to- 3DD quality with CLIP image scores on examples from [9] (numbers reproduced from [17]). CAT3D
ImageDream [9]
120
83.77 +:: 5,2
One2345++ [84]
0.75
83.78 + 6.4
IM-3D (NeRF) [17]
40
87.37 + 5.4
produces competitive results to object-centric baselines while also working on whole scenes. (Note: shown images 3D ren- ders, and time for CAT3D was evaluated on 16 A100 GPUs.)
IM-3D [17]
3
91.40 + 5.5
CAT3D (ours)
1
88.54
1 8.6
in-domain and out-of-domain datasets) and few-view 3D reconstruction performance. We also compare important design choices for 3D reconstruction. Results from our ablation study are reported in Table and Figure 6 in Appendix A and summarized below. Overall, we found that video diffusion architectures, with 3D self-attention (spatiotemporal) and raymap embeddings of camera pose, produce consistent enough views to recover 3D representations when combined with robust reconstruction losses.
Image and pose. Previous work [7] used PixelNerf [63] feature-map conditioning for multiple input views. We found that replacing PixeINeRF with attention-based conditioning in conditional video diffusion architecture using a per-image embedding of the camera pose results in improved samples and 3D reconstructions, while also reducing model complexity and the number of parameters. We found that embedding the camera pose as a low-dimensional vector (as in [37]) works well for in-domain samples, but generalizes poorly compared raymap conditioning (see Section 3.1).
Increasing the number of views. We found that jointly modeling multiple output views (i.e., 5 or views instead of I) improves sample metrics even metrics that evaluate the quality of each output image independently. Jointly modeling multiple outputs creates more consistent views that result in an improved 3D reconstruction well.

---

Attention layers. We found that 3D self- attention (spatiotemporal) is crucial, as it yields improved performance relative factorized 2D self-attention (spatial-only) and ID self-attention (temporal- only). While models with 3D self-attention in the finest feature maps (64 x 64) result in the highest fidelity images, they incur a significant computational overhead for training and sampling for relative small gain in fidelity. We therefore decided to use 3D self-attention only in feature maps of size 32 x 32 and smaller.
Multi-view diffusion model training. Initializing from a pre-trained text-to-image latent diffusion model improved performance on out-of-domain examples. We experimented with fine-tuning the mueti-view diffusion model to multiple variants specialized for specific numbers of inputs and outputs views, but found that single model jointly trained on frames with either 1 or conditioning views was sufficient to enable accurate single image and few-view 3D reconstruction.
3D reconstruction, LPIPS loss is crucial for achieving high-quality texture and geometry, aligning with findings in [17, 7]. On Mip-NeRF 360, increasing the number of generated views from 80 (single elliptical orbit) 720 (nine orbits) improved central object geometry but sometimes introduced background blur, probably due to inconsistencies in generated content.
Discussion
We present CAT3D, a unified approach for 3D content creation from any number of input images. CAT3D leverages multi-view diffusion model for generating highly consistent novel views of 3D scene, which are then input into a 3D multi-view reconstruction pipeline. CAT3D decouples the generative prior from 3D extraction, leading to efficient, simple, and high-quality 3D generation.
Although CAT3D produces compelling results and outperforms prior works on multiple tasks, has limitations. Because our training datasets have roughly constant camera intrinsics for views of the same scene, the trained model cannot handle test cases well where input views are captured by multiple cameras with different intrinsics. The generation quality of CAT3D relies on the expressivity of the base eext-to-image model, and performs worse in the cases where scene content out of distribution for the base model (e.g. human faces, since the base model was trained on limited human data). The number of output views supported by our multi-view diffusion model is still relatively small, so when we generate large set of samples from our model, not all views may be 3D consistent with each other (see Supplementary website). Finally, CAT3D uses trajectories that cover the scene thoroughly (see Appendix C), which may be difficult to design for large-scale open-ended 3D environments.
camera
There are a few directions worth exploring in future work to improve CAT3D. The multi-view diffusion model may benefit from being initialized from a pre-trained video diffusion model, as observed by [10, 17]. The consistency of samples could be further improved by extending the number of conditioning and target views handled by the model. Automatically determining the camera trajectories required for different scenes could increase the flexibility of the system.
Acknowledgements We would like thank Daniel Watson, Rundi Wu, Jason Y. Zhang, Richard Tucker, Jason Baldridge, Michael Niemeyer, Rick Szeliski, Dana Roth, Jordi Pont-Tusee Tuset, Andeep Torr, Irina Blok, Doug Eck, and Henna Nandwani for their valuable contributions to this work. We also extend our gratitude to Shlomi Fruchter, Kevin Murphy, Mohammad Babaeizadeh, Han Zhang and Amir Hertz for training the base text-to-image latent diffusion model.
References
[1] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi. and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV, 2020.
[2] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. SIGGRAPH, 2022.
[3] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splatting for Teal-Time Radiance Field Rendering. SIGGRAPH, 2023.
10

---

[4] Jiawei Yang, Marco Pavone, and Yue Wang. FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization. CVPR, 2023.
[5] Nagabhushan Somraj, Adithyan Karanayil, and Rajiv Soundararajan. SimpleNeRE: Regulariz- ing Sparse Input Neural Radiance Fields with Simpler Solutions. SIGGRAPII Asia, 2023.
[6] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large Reconstruction Model for Single Image to
3D. arXiv:231 1.04400. 2023.
[7] Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Holynski. Reconfusion: 3d reconstruction with diffusion priors, 2023.
[8] Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using
2D Diffusion. ICLR, 2022.
[9] Peng Wang and Yichun Shi. Imagedream: Image- progeeprrmptttt multi-view diffusion for 3d generation. arXiv:2312.02201, 2023.
[10] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv:2311.15122 2023.
[11] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. CVPR, 2023.
[12] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu Video: Factorizing Text-to-Vides Generation by Explicit Image Conditioning. arXiv:2311.10009, 2023.
[13] Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv, 2024.
[14] Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models, 2023.
[15] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing. David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024.
[16] Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T Barron, Amit H Bermano, Eric Ryan Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al. State of the art on diffusion models for visual computing. arXiv:2310.07200 2023.
[17] Luke Melas- Kyrrazi, Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos. IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation, 2024.
[18] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model. arXiv:2311.06214, 2023.
[19] Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. Dream- Time: An Improved Optimization Strategy for Text-to-3D Content Creation. arXiv, 2023.
[20] Peihao Wang, Zhiwen Fan, Dejia Xu, Dilin Wang. Sreyas Mohan, Forrest landola, Rakesh Ranjan, Yilei Li. Qiang Liu, Zhangyang Wang, et al. SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identity. arXiv, 2023.
[21] Subin Kim, Kyungmin Lee, June Suk Choi, Jongheo Jeong, Kihyuk Sohn, and Jinwoo Shin. Collaborative score distillation for consistent visual editing. NeurlPS, 36, 2024.

---

[22] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao. Chongxuan Li, Hang Su, and Jun Zhu. ProlifieDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. NeurlPS, 2023.
[23] Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 19740-19750, 2023.
[24] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. 2023.
[25] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yo Yu Liu, and Tsung- Yi Lin. Magic3D: High-Resolution Text-to-3D Content Creation. CVPR, 2023.
[26] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv:2309.16633 16653, 2023.
[27] Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text 3d gaussian splatting with point cloud priors. vrXiv:2310008525 2023.
[28] Dave Epstein, Ben Poole, Ben Mildenhall, Alexei A Efros, and Aleksander Holynski. Disentan- gled 3d scene generation with layout learning. arXiv preprint 1.Xiv:22402.693666 2024.
[29] Jonathan Lorraine, Kevin Xie, Xiaohui Zeng. Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung- Yi Lin, Ming-Yo Yu Liu, Sanja Fidler, and James Lucas. ATT3D: Amortized Text-to-3D Object Synthesis. ICCV, 2023.
[30] Luke Kyriaz,y Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruction of any object from a single image. CVPR, 2023.
[31] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors. arXiv:2306. 17.3006.77444,. 2023.
[32] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-It-3D: High-Fidelity 3D Creation from Single Image with Diffusion Prior. ICCV, 2023.
[33] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models. ICCV, 2023.
[34] Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J Fleet. Monocular depth estimation using diffusion models. arXiv:2302. 14816, 2023.
[35] Hong-Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William Freeman, Forrester Cole, Deqing Sun, Noah Snavely, Jiajun Wu, et al. WonderJourney: Going from Anywhere to Everywhere. arXiv:2312.0388, 2023.
[36] Ethan Weber, Aleksander Hołyúski, Varun Jampani, Saurabh Saxena, Noah Snavely, Abhishek Kar, and Angjoo Kanazawa. Nerfiller: Completing scenes via generative 3d inpainting. arXiv preprint 232iv:2312.045000 2023.
[37] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; Zero-Shot One Image to 3D Object. arXiv, 2023.
[38] Daniel Watson, William Chan, Ricardo Martin Bruaala, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. arXiv:2210004628, 2022.
[39] Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Ben Mildenhall, Nataniel Ruiz, Shiran Zada, Kfir Aberman, Michael Rubenstein, Jonathan Barron, Yuanzhen Li, and Varun Jampani. DreamBooth3D: Subject-Driven Text-to-3D Generation. ICCV, 2023.
12

---

[40] Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. NeriDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion. ICMI., 2023.
[41] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. GeNVS: Generative novel view synthesis with 3D-aware diffusion models. arXiv, 2023.
[42] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, et al. ZeroNVS: Zero-Shot 360-Degree View Synthesis from Single Image. CVPR, 2024.
[43] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. 0-e-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization. arXiv, 2023.
[44] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. MVDream: Multi-view Diffusion for 3D Generation. arXiv, 2023.
[45] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. 12++:23+:: a single image to consistent multi-view diffusion base model, 2023.
[46] Jiayu Yang, Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong Li. ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion. arXiv:2310. 10343, 2023.
[47] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. SyncDreamer: Generating Multiview-consistent Images from a Single-view Image. arXiv, 2023.
[48] Lukas Höllein, Aljaž Božič, Norman Müller, David Novotmy, Hung- Yu Tseng, Christian Richardt, Michael Zollhöfer, and Matthias Nießner. Viewdiff: 3d-consistent image generation with text-to-image models, 2024.
[49] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv:2204.03455, 2022.
[50] Jonathan Ho William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv:2210.02303. 2022.
[51] Siddhant Jain, Daniel Watson, Eric Tabellion, Aleksander Holyúski, Ben Poole, and Janne Kontkanen. Video interpolation with diffusion models. arXiv preprint arXiv:2404.01200 2024.
[52] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Ani- matediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023.
[53] Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionetrl: A unified and flexible motion controller for video generation. arXiv preprint arXiv:23/2.0336 2023.
[54] Jeong-gi Kwak, Erqun Dong, Yuhe Jin, Hanseok Ko, Shweta Mahajan, and Kwang Moo Yi. ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models. 1rXiv:2312.01330 305, 2023.
[55] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel Multi-view Synthesis and 3D Generation from Single Image using Latent Video Diffusion, 2024.
[56] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oğuz. 3DGen: Triplane Latent Diffusion for Textured Mesh Generation. 0rXiv:2303.05537 2023.
[57] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Viewset Diffusion: (0- Jmmage-Conditioned 3D Generative Models from 2D Data. ICCV, 2023.
13

---

[58] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang. Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang. DMV3D: Denoising Multi- View Diffusion using 3D Large Reconstruction Model, 2023.
[59] Stanislaw Szymanowicz. Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. arXiv:2312.13150, 2023.
[60] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting. #xXiv2244.119722 2024.
[61] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
[62] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. CVPR, 2022.
[63] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural Radiance Fields from One or Few Images. CVPR, 2021.
[64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. ICML, 2021.
[65] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory--fficient exact attention with io-awareness. NeurIPS, 35, 2022.
[66] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv:2307.08691, 2023.
[67] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images. ICML, 2023.
[68] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry-fre novel view synthesis through set-latent scene representations. CVPR, 2022.
[69] David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In ACM-SIAM Symposium on Discrete Algorithms, 2007.
[70] Ayush Sarkar, Hanlin Mai, Amitabh Mahapatra, Svetlana Lazebnik, David Forsyth, and Anand Bhattad. Shadows Don't Lie and Lines Can't Bend! Generative Models don't ton't know Projective Geometry... for now, arXiv:2311.171:3 2023.
[71] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. ZiR-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields. ICCV, 2023.
[72] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. CVPR, 2018.
[73] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. CVPR, 2023.
[74] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny, Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction. ICCV, 2021.
[75] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. SIGGRAPH, 2018.
[76] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et MVImgget: A Large-scale Dataset of Multi-view Images. CVPR, 2023.
14

---

[77] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. CVPR, 2014.
[78] Ben Mildenhall, Pratul Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng. and Abhishek Kar. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. SIGGRAPH, 2019.
[79] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. CVPR, 2022.
[80] Jaidev Shriram, Alex Trevithick, Lingjie Liu, and Ravi Ramamoorthi. RealmDreamer; Text- Driven 3D Scene Generation with Inpainting and Depth Diffusion, 2024.
[81] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior. arXiv, 2023.
[82] Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Bong-Hai Zhang. Triplane Meets Gaussian Splatting: Fast and Generalizable Single- View 3D Reconstruction with Transformers, 2023.
[83] Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from single image. 2024.
[84] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng. Jiayuan Gu, and Hao Su. 2-3-2-3-4 Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv:2311.07885, 2023.
[85] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. 125.5,, 2022.
arXiv preprint
[86] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
15

---

A
Ablations
Here we conduct ablation studies over several important decisions that led to CAT3D. We consider several multi-view diffusion model variants, and evaluated them novel-view synthesis with held-out validation sets (4k samples) from the training datasets (in-domain samples) and from the mip-NeRI 360 dataset (out-of-doman) samples), as well as 3-veew reconstruction the mip-NeRF 360 dataset (out-of-domant renders). Unless otherwise specified, models in this section are trained with 3 input views and 5 output views, evaluated after 120k optimization iterations. Quantitative results are summarized in Table 3. Then we discuss important 3D reconstruction design choices, with qualitative comparison in Figure 6.
Number of target views. We start from the setting where the model takes 3 conditional views as input and generates a single target view, identical to the ReconFusion baseline [7]. Results show that our niet architecture is favored when dealing with multiple input views, compared to the PixeINeRF used by [7]. We also show that going from single target view to target views results in significant improvement both novel-view synthesis and few-view reconstruction.
Camera conditioning. As mentioned in Section 3.1, we compared two camera parameterizations, one is an 8-dimensional encoding vector fed through cross-attention layers that contains relative position, relative rotation quaternion, and absolute focal length. The other is camera rays fed through channel-wise concatenation with the input latents. We observe that the latter performs better across all metrics.
Attention layers. An important design choice is the type and number of self atteettention layers used when connecting multiple views. As shown in Table 3, it is critical to use 3D self-attention instead of temporal ID self-attention. However, 3D attention is expensive; 3D attention at the largest feature maps (64 x 64) is of 32k sequence length and this incurs a significant computation overhead during training and sampling with a marginal performance gain, especially for out-of-domain samples and renderings. We therefore chose to use 3D attention only for feature maps of size 32 x 32 and smaller.
Multi-view diffusion model training. We compared the settings of training the multi-view diffusion models from scratch with initializing from pre-trained text-to-image latent diffusion model. The latter performs better, especially for out-of-domain cases. We further trained the model for more iterations and observed a consistent performance gain up until 1M iterations. Then we fine-tuned the model for handling both cases of 1 conditional + 7 target views and 3 conditional 5 target views (jointly), for another 0.4M iterations. We found this joint finetuning leads to better in-domain novel-view synthesis results with 3 conditional views, and out-of-domain results that are on-par with the previous model.
3D reconstruction. We found perceptual distance (LPIPS) loss crucial in recovering high-quality texture and geometry, similar to [17, 7]. We also compared the use of 80 views along one orbital path with 720 views along nine variably scaled orbital paths. In the Mip-NeRF 360 setting, increasing the number of views helps better regularize the geometry of central objects, but sometimes leads to blurrier textures in the background, due to inconsistencies in generated content.
B Details of View Diffusion Model
We initialize the multi-view diffusion model from a latent diffusion model (LDM) trained for text-to- image generation similar to [62] trained on web scale image datasets. See Figure 7 for visualization of the model architecture. We modify the LDM to take multi-view images as input by inflating the 2D self-attention after every 2D residual blocks to 3D self-attention [44]. Our model adds minimal additional parameters to the backbone model: just a few additional convolution channels at the input layer to handle conditioning information. We drop the text embedding from the original model. Our latent diffusion model has 850M parameters, smaller than existing approaches built on video diffusion models such IM-3D [17] (4.3B) and SV3D [55] (1.5B).
We fine-tune tune the full latent diffusion model for 1.4M iterations with a batch size of 128 and learning rate of 5 x 10-5 The first 1M iterations are trained with the setting of conditional view and 7
16

---

In-domain diffusion samples
Out-of-domaan
diftusion samples
NeRF renderings
Setting
PSNRT SSIM? LPIPS!
PSNRT SSIMT
LPIPS
PSNRT
SSIMT LPIPS_
Baseline
ReconFusion [7]
14.01
0.265
0.568
15.49
0.358
0.585
# target views
3 cond fgt
18.85
0.638
0.359
14.12
0.262
0.553
16.17
0.360
0.546
3 cond 5 Egt
21.66
0.733
0.277
14.63
0.278
0.515
16.29
0.368
0.530
Camera conditioning Low-dim vector Raymap
21.17
0.710
0.304
14.19
0.266
0.530
15.97
0.359
0.544
21.66
0.733
0.277
14.63
0.278
0.515
16.29
0.368
0.530
Attention layers
Temporal attention
18.62
0,653
0.362
13.41
0.250
0.582
15.03
0.330
0.595
3D attention until 16
21.41
0.730
0.281
14.23
0.274
0.530
16.21
0.364
0.54
3D attention until 32
21.66
0,733
0.277
14.63
0.278
0.515
16.29
0.368
0.530
3D attention until 64 (full)
22.83
0.783
0.235
14.64
0.274
0.509
16.35
0.367
0.528
Model training
From scratch, 3 cond 5 Egt
21.16
0.722
0.282
13.88
0.255
0.546
15.68
0.348
0.557
From pretrained, 3 cond 5 tgt
21.66
0.733
0.277
14.63
0.278
0.515
16.29
0.368
0.530
From pretrained, 3 cond 5 tgt, 1M iters
22.49
0.757
0.256
15.19
0.303
0.482
16.58
0.384
0,509
From pretrained, jointly, 1.4M iters
22.96
0.777
0.235
15.15
0.294
0.488
16.62
0.377
0.515
Table 3: Ablation study of multi-view diffusion models. A comparison of model variants and
parameters settings across in-domain sequences (a mixture of Objaverse, Co3D, RealEstate 10k and MVImgNet - all sequences in the corresponding eval sets of our training data) and out-of-domate sequences (examples from the mip-NeRF 360 dataset). We evaluate the quality of samples from the diffusion model, as well as renderings from the subsequently optimized NeRF.
target views, while the rest 0.4M iterations are trained with equal mixture of condition 7 target views and 3 conditional + 5 target views. Our model was trained for 16 days on 128 TPU-v4 chips. Following [7] we draw training samples with equal probability from the four training datasets. We enable classifier-fres guidance (CFG) [85] by randomly dropping the conditional images and camera poses with probability of 0.1 during training.
C Details of Generating Novel Views
We use DDIM [86] with 50 sampling steps and CFG guidance weight 3 for generating novel views. It takes seconds to generate 80 views 16 A100 GPUs. As mentioned in Section 3.2, selecting camera trajectories that fully cover the 3D scene important for high-quality 3D generation results. See Figure 8 for illustration of the camera trajectories use. For the single image-to-3D setting we use two different types of camera trajectories, each containing 80 views:
A spiral around cylinder-like trajectory that moves into and out of the scene.
An orbit trajectory for images with central object.
For few-view reconstruction, we create different trajectories based on the characteristies of different datasets:
RealEstate 10K: we create a spline path fitted from the input views, and shift the trajectories along the zzlpllane by certain offsets, resulting in 800 views.
LLFF and DTU: we create a forward-facing circle path fitted from all views in the training set, scale it, and shift along the z-axis by certain offsets, resulting in 960 and 480 views respectively.
CO3D: we create a spline path fitted from the input views, and scale the trajectories by multiple factors, resulting in 640 views.
Mip-NeRF 360: we create elliptical path fitted from all views in the training set scale it, and shift along the 2-axis by certain offsets, resulting in 720 views.
For generating anchor views in the single-image setting, we used the model with 1 conditional input and target outputs. For generating the full set of views (both in the single-image anchored setting,
17

---

(a) 720 generated views
(b) 80 generated views
(c) Without LPIPS loss
Figure 6: Qualitative comparison of 3D reconstruction design choices. Rendered images (left) and depth maps (right) of a Mip-NeRF 360 scene under different settings: (a) 720 generated views along multiple orbital paths, (b) 80 generated views on single orbital path, and (c) 720 views, without the perceptual (LPIPS) loss.
as well as in the nult setting), we used the model with conditional inputs and groups of target outputs, selected by their indices in the camera trajectories.
Anchor selection. For single-image conditioning, we first select a set of target viewpoints as anchory to ground the scene content. To select set of anchor views that are spread throughout the scene and provide good coverage, we use the initialization strategy from [69]. This method greedily selects the camera whose position is furthest away from the already selected views. We found this to work well on a variety of trajectory-likk view sets as well as random views that have been spread throughout scene.
Dealing with non-square images. While the multi-view latent diffusion model we trained only supports 512 x 512 square images, we found the model still performed well when padding non-square images to square. However, this method often reduces resolution, so we also run our model on square-cropped version of the inputs, and then compose the square-cropped outputs with the edges from the padded outputs to create a different aspect ratio image.
D Details of 3D Reconstruction
For the Zip-NeRF [71] baseline, we follow [7] to make a few modifications of the hyperparameters (See Appendix (7]) that better suit few-view reconstruction setting. We use smaller view dependence network with width 32 and depth 1, and a smaller number of training iterations of 1000, which helps avoid overfitting and substantially speeds up the training and rendering. Our synthetic view sampling and 3D reconstruction process is run 16 A100 GPUs. For fiew reconstruction.
18

---

3x512x512
Images
Output Images
Encoder
Decoder
Relative camera raymap Observed mask
2x
8x64x64
Downsampling Block
Skip
Upsampling
Spatial Conv2D Stack
Block
Spatiotemporal Joint
8x32x32
Downsampling Block 1/2x
Upsampling
Skip
Block
Joint Attention
8x 16x 16
Downsampling Block
Skip
Upsampling Block
BTHWC-> (THW)
Flash Attention
1/2x
2x
B BITHW/ C C
8x8x8
Middle Conv2D
Stack
Figure 7: Illustration the network. CAT3D builds on latent text-to-image diffusion model. The input images of size 512 are 8x downsampled to latents of size 64 64, which concatenated with the relative camera raymap and binary mask that indicates whether or not the image has been observed. A 2D U-Net with temporal connections is utilized for building the latent diffusion model. After each residual block with resolution < 32 x 32, we inflate the original 2D self-attention (spatial) of the text-to-image model to be 3D self-attention (spatiotemporal). We remove the text embedding conditioning of the original model.
we sample 128 x 128 patches of rays and train with global batch size of 1M that takes minutes to train. For single image to 3D, we use 32 x 32 patches of rays and global batch size of 65k that takes seconds to train. Learning rate is logarithmically decayed from 0.04 to 10-3, The weight of the perceptual loss (LPIPS) is set to 0.25 for single image to 3D and few-view reconstruction on RealState 10K, LLFF and DTU datasets, and to 1.0 for few-view reconstruction on CO3D an MipNeRF-360 dataets.
Distance based weighting. We design a weighting schedule that upweights views closer to captured views in the later stage of training to improve details. Specifically, the weighting is given by Gaussian kernel: w & exp(-bs"). where is the distance to the closest captured view and is a scaling factor. For few-view reconstruction, b is linearly annealed from 0 to 15. We also anneal the weighting of generated views globally to further emphasize the importance of captured views.
19

---

(a) Single image to 3D, forward
(b) Single image 3D, orbit
4v
(c) RealEstate 10K
(d) LLFF
(e) DTU
(f) CO3D
g Mip-NeRF 360
Figure 8: Camera trajectories for generating novel views. Within each panel, left shows the side view and right shows the top view of the trajectories, colored by indices of views. (a)-(b): two types of trajectories used by single image to 3D. Observed view is highlighted in red, while anchor views are highlighted in orange. (c)-(g): trajectories used by 3D reconstruction. input views are highlighted in red.
20

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims around performance, runtime, and comparison to existing methods are described in the experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our method for certain categories, generalizing different camera intrinsics, genearting large sets of consistent views, and requiremenet manually constructed camera trajectories in the discussion section. We also describe limitations the experiments section, e.g. image-to-3D for segmented objects having room for improvement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We aimed provide the full details necessary to reproduce these results, includ- ing architecture details and a diagram, description of the training datasets and procedures, evaluation protocols for 3D, and several detailed appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
22

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We have not open sourced the code used in this work, but the datasets we used are all publicly available (Re10K, CO3D, MVImgNet, Objaverse).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm Secii///uuide//ooeeeeee
cessonnnnlyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /niis. wwtppblicgguiieeecccccccccccccce http:/gguuddee/eeeuuuubbiiiiiiii ProdesubmiissooPoll.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details the diffusion model architecture in Appendix B, includ- ing learning rate and optimization hyperparameters, data mixtures, CFG masking probability, and more. The dataset splits were used and shared in previous work.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: For few-view reconstruction metrics, we follow prior work and do not report error bars. For stngleimmae CLIP metries standard deviation over prompts, in prior work. Evaluation of 3D generative models is challenging, and better metrics are an active area of research.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: For 3D reconstruction, Appendix D describes the compute requirements and
iterations. For diffusion model training, the compute requirements are specified in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::::///euipppe www.rrppsccppuuuue cc/public/Ethi www.hhhhhciiieineee.e..
Answer: [Yes]
Justification:
We believe have followed the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [No]

---

We currently discuss mostly the positive impacts of this work, but can add more acknowledgement of the potential for harm caused by 3D generative models the reviewers believe it is useful.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification: We do not plan to release these models and have not planned safeguards around the release.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the papers that created the datasets we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
25

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcodecoodee.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We produce 3D assets in the form of videos and 3D models that have been linked to in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Paper does involve crowdsourcing nor resarch with human subjects.
Guidelines:
26

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "EAvMW3Lar7", "reviewer": "Reviewer_1Rgc", "summary": "The objective of this paper is to achieve single-view or few-view to 3D. The core of their method lies in a multi-image-based diffusion model that leverages 3D attention and raymap encoding for the camera poses.  This setup is different from concurrent work, IM-3D, which repurposes video generation model to achieve 3D, or ReconFusion, which iteratively refine novel view with diffusion model conditioned on PixelNeRF embeddings. Experiments are conducted against several competitive Baselines, like IM-3D and ReconFusion. Result-wise, their proposed method showcases a nice balance between quality and efficiency.", "questions": "1. What happens if we only use anchor views to run recon in step2? How the quality compared with the current setup?\n\n2. What if we densify view through some existing video interpolation model (instead of re-run CAT3D)? Would be more efficient and better quality?", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The quality of the generated/reconstructed 3D scene is state-of-the-art, and it is more efficient compared with ReconFusion and other iterative methods.\n\n2. The proposed multi-view diffusion model is effective. In LRM related literature, image-based diffusion is tricky to generate multi-view consistent. The proposed CAT3D certainly gives better multi-view image generation quality without explicit maintaining a 3D representation, which can benefit many related tasks. \n\n3. The paper is clearly motivated, and the experiments are carefully designed and reported.", "weaknesses": "1. It seems cumbersome to generate a large number of viewpoints(Line:174)  by doing anchor first, and enriching frames in between by repeated running CAT3D. Indeed, more view generated together from the multi-view diffusion is good. But running it iteratively will still produce inconsistencies over runs over different set of camera viewpoints, as is can be seen from Fig.6. \n\n2. The trajectory shape seems to influence the quality of the multi-view diffusion. This limits its generalizability to arbitrary/scattered image viewpoints towards the same scene target, which are common in daily life.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and careful review of our work. Below we address your questions and weaknesses mentioned:\n\n\n> cumbersome to generate a large number of viewpoints\n\nWe agree that jointly generating all target frames from the multi-view diffusion model would enable more consistent samples. However, simultaneously generating a large number of frames with video diffusion-like architectures is still an active area of research, with many SOTA text-to-video models using autoregressive generation in time to produce a larger number of frames (while losing consistency due to limited context length). By splitting the generation into anchors and then independent blocks of generation at different camera locations, we can efficiently generate a large number of frames *in parallel*. Building efficient architectures that can support the joint generation of a larger number of frames is an exciting direction for future research. \n\n> trajectory influences quality\n\nOur multi-view model can generalize to arbitrary camera trajectories, and we show results for other input capture types, e.g., forward-facing scenes like LLFF which contain several images pointed towards the same scene target. It is true that the model performance varies for different camera trajectories, and this is likely due to some of the biases inherited from our relatively limited training dataset. Mixing in training data with more diverse camera distributions could be an interesting future direction.\n\n> only use anchor views for reconstruction\n\nOur model was only trained to produce at most 8 views total (conditioning + target), and the 3D reconstruction methods we use (i.e., Zip-NeRF) are seldom able to reconstruct plausible geometry from 8 views (as a reference, see results with Zip-NeRF and 9 *real* views which are strictly better than 8 generated views in Table 1). Including additional frames through AR generation is critical to yield the high-quality reconstruction and NVS results we present in the paper (see also Fig. 6 that compares 80 vs. 720 frames).\n\n> densify views through video interpolation\n\n\nUsing a video interpolation model to generate frames is an interesting idea. However, it is not clear how to use such a model to incorporate more than 2 frames, and how to leverage the resulting frames in 3D reconstruction without explicit camera control. It’s also not clear whether this approach would be more efficient or would yield better quality results: if using an architecture similar to ours, the compute cost would be similar, and one would also have to estimate camera poses for the resulting frames (which adds an additional computational expense that our pose-conditioned multi-view diffusion models do not incur).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tOod1Ua1uL", "reviewer": "Reviewer_d7U9", "summary": "This paper introduces CAT3D, a novel approach for generating 3D representations from a flexible number of input images. The authors tackle the challenge of limited input data, a common bottleneck for 3D reconstruction, by leveraging the power of multi-view diffusion models. Their method generates a collection of novel viewpoints consistent with the input, effectively transforming a sparse-view reconstruction problem into a more manageable dense-view scenario. These generated views are then fed into a robust 3D reconstruction pipeline based on a modified Zip-NeRF to produce the final 3D model.\n\nThe authors demonstrate impressive results on various benchmarks, showcasing CAT3D's ability to handle single images, sparse multi-view captures, and even text prompts as input. The method exhibits state-of-the-art performance on few-view 3D reconstruction tasks, outperforming existing methods in terms of both speed and accuracy on established datasets. While single-image 3D generation shows promise, the authors acknowledge the performance is not yet on par with leading methods specifically designed for that task, particularly for single objects. The paper presents a compelling advancement in 3D content creation by unifying different input modalities within a single framework and showcasing significant efficiency gains.", "questions": "- Could the authors provide a more quantitative assessment of how performance degrades with increasing pose noise or sparsity? \n\n- Have the authors considered incorporating an automatic trajectory optimization scheme within CAT3D? \n\n- While open-sourcing the code and trained models is ideal, could the authors at least elaborate on their plans for sharing their work and facilitating reproducibility? Providing more details about the training procedure and hyperparameters would also be beneficial.", "limitations": "The authors list several limitations, including the reliance on constant camera intrinsics during training, the expressiveness limits of the base text-to-image model, the small number of output views, and the need for manual camera trajectories. However, the discussion lacks concrete examples or quantifiable measures of the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper presents a novel approach to 3D content creation by reframing the challenge of sparse input as a view generation problem. This core idea of generating the data needed for robust reconstruction is a valuable contribution to the field.\n\n- The authors demonstrate the effectiveness of CAT3D through comprehensive experiments on established benchmarks. Their results on few-view 3D reconstruction tasks are particularly impressive, showcasing state-of-the-art performance on standard metrics and surpassing existing techniques in terms of both speed and accuracy. The ablation study is well-executed, providing valuable insights into the contribution of different components of their method.\n\n- The paper is generally well-written and easy to follow. The authors clearly motivate their work, provide sufficient background information, and describe their methodology in a structured manner. The figures are informative and complement the textual descriptions well.\n\n- The ability to generate high-quality 3D content from a flexible number of input images, as CAT3D aims to achieve, has substantial practical significance. This flexibility is highly desirable for various applications. The demonstrated speed improvements over existing iterative optimization methods further add to its potential impact by enabling more efficient workflows.", "weaknesses": "- While CAT3D aims to handle sparse inputs, its dependence on calibrated camera poses, presents a significant limitation. How does performance degrade with increasing sparsity and decreasing pose accuracy? \n\n- The paper relies on manually designed camera trajectories for novel view synthesis, which limits practicality and scalability. The authors briefly mention adapting trajectories based on scene characteristics but provide no concrete details. Developing an automated trajectory selection or optimization procedure, potentially guided by learned priors or scene understanding techniques, would significantly enhance the method's value and broader applicability.\n\n- The paper, at times, overstates its accomplishments (e.g., \"achieving state-of-the-art performance across nearly all settings\") and does not adequately address its limitations. A more nuanced and critical self-assessment would strengthen the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful reading and kind words. Below we address your questions and weaknesses mentioned:\n\n> performance with increasing sparsity and decreasing pose accuracy\n\nIn terms of performance while varying the number of input views with accurate camera poses, Table 1 includes qualitative results for 3, 6, and 9 view sparse reconstruction. The CAT3D model was not trained with missing or noisy poses, but some of our training data (CO3D) has imperfect pose which leads to some small amounts of robustness. To verify this, we conducted an experiment by perturbing the camera rotations of the input conditioning 3 views by a certain number of degrees, and measuring the error between the generated and ground truth target images. See Figure 1 in the rebuttal PDF for average PSNR across varying amounts of rotation perturbation on the MipNerf-360 dataset. Our model can handle small rotation perturbation. Fine-tuning the model with more perturbations on camera pose should lead to improved performance in this setting, which would be an interesting direction for future work.\n\n> manually designed camera trajectories\n\nWe agree with the reviewer that jointly learning a trajectory model or inferring trajectories given scene content is an exciting direction for future work. Empirically, we found a handful of simple heuristic trajectories worked well for our experiments, but ideally the trajectories should cover the scene without being placed inside of objects or walls as mentioned in line 167-169.\n\n> overstates its accomplishments\n\nWe are happy to revise language to better reflect limitations. The “state-of-the-art performance across nearly all settings” is referring to Table 1, where our method indeed exhibits stronger performance than all prior work. Were there any other passages of text that you would like us to alter? We discuss several limitations in the discussion section, and can expand for the camera ready (especially discussing the challenges we mentioned above regarding trajectory selection). \n\n> reproducibility\n\nIn the Appendix, we aimed to provide all details necessary to reproduce CAT3D on top of an open-source latent diffusion model. If there are any additional details that the reviewer thinks are missing, we are happy to include them in an updated draft.\n\n> constant camera intrinsics, other limitations\n\nIt’s worth noting that our model does not rely on constant camera intrinsics during training. This is an artifact of our current training dataset where the camera intrinsics are approximately fixed for each scene. If we were to capture and include additional training data with camera intrinsics varying within a scene, we expect our model would be able to perform camera intrinsic manipulation. As far as limitations in expressiveness of the base model, one notable example is that our model performs poorly on human faces, as the base model was not trained on much human data. A showcase of the limitation of producing a small number of output views can be seen in the Supplementary website, where the generated spin video is clearly not perfectly 3D consistent. The need for manual camera trajectories is shown in Fig. 8, by the fact that it was necessary to create different types of trajectories based on characteristics of different datasets. We will further emphasize these points in the paper text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0uk8jGyi1b", "reviewer": "Reviewer_xucG", "summary": "In this paper, the authors propose a two-stage method for 3D creation. Specifically, they introduce a multi-view diffusion model to generate novel views given observed input views. Then using these views, they perform a robust 3D reconstruction using a Zip-NeRF variant. To generate consistent views, they also design a data-dependent sampling strategy. In addition, they also conduct extensive experiments to validate the effectiveness of the proposed method.", "questions": "See the Weaknesses.", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ The proposed method is simple but effective. It also obtains a superior 3D creation on scenes.\n+ The experiments are exhaustive and validate each components comprehensively, making this paper solid.\n+ This paper is easy to follow and provides much details. Thus, it is easy to reproduce it.", "weaknesses": "- It is certain that using larger diffusion models can boost the performance. But it is interesting to showcase the improvement trend with increased diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback on our work. \n\n> larger diffusion models can boost the performance\n\nWe certainly expect that larger models will lead to improved performance and can generate more consistent novel views. One relevant piece of evidence: we experimented with different model variants (Table 3) and found that increasing the amount of computation (e.g., by adding 3D attention operations at every resolution layer of the UNet) can improve performance (albeit at the expense of efficiency in training and sampling). We chose one of the smaller models as our showcase result with the intention of striking a balance of efficiency and performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EHi0ylaT6n", "reviewer": "Reviewer_Twzm", "summary": "This paper proposed CAT3D, a pipeline that enabled the production of 3D representations from one or a few input views. CAT3D comprises a multi-view generation model to synthesize novel images from different viewpoints and a Zip-Nerf to achieve 3D reconstruction based on generated views. The 3D results shown in this paper are very impressive with high quality, and the authors provided sufficient experiments to verify the effectiveness of CAT3D.", "questions": "Some unclear implementation:\n\n1. Lines 190-191 are unclear. How to deal with images with different aspect ratios during training and inference?\n2. Missing details about the shift of the noise scheduler.\n3. What is meant by \"drop the text embedding\"? Does it imply using an empty string \"\" as the input text for all samples, or does it completely remove all cross-attention layers? Additionally, how is stability maintained when all cross-attention layers are removed at the beginning of training?\n\nSome other minor questions:\n1. As mentioned in the paper, a mask channel is concated to the inputs to distinguish conditions and targets. Why not try to use SD-inpainting as the initialization to cover this task?\n2. Could 3DGS be converged with fewer generated images? Generating 80/720 views is too costly in my opinion.", "limitations": "The authors discuss the limitations. However, no qualitative limitations are shown in the paper.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. CAT3D achieves impressive results of 3D generation.\n2. The overall pipeline of multi-view generation is straightforward, yet effective with good robustness as verified in the experiments.\n3. Although CAT3D is trained with constrained multi-view images (no background: objaverse; object-centric: CO3D, MVImgNet, most indoor scenes: RealEstate10k), it still enjoys good generalization. \n4. This paper can be seen as evidence to verify that scaling up 3D generation through multi-view synthesis is feasible.\n5. The ablation study is convincing and sufficient.", "weaknesses": "1. Most techniques have been proposed by previous works, including the raymap, Nerf with LPIPS loss (IM-3D). But I think this point is not the main issue of this paper, while CAT3D proves the scalability of combining all these techniques together.\n2. Since ZIP-Nerf is utilized in CAT3D for dense 3D reconstruction, it is unclear whether the results shown in Figures 1, 2, 4, and 5 in the paper are derived from ZIP-Nerf or generated from the multi-view diffusion process. Are all these results exclusively from ZIP-Nerf? If not, it would be beneficial to provide more results directly from the diffusion model. This distinction is important, especially considering the potential inconsistencies noted in the limitations section.\n3. It is not inappropriate to provide the inference efficiency with 16 A100 GPUs (Line573) only, which is not a regular setting for most users. The authors should provide the inference time with one GPU. More importantly, it is unclear whether the efficiency results from Table 2 are all fairly compared with one GPU or the same hardware condition.\n4. CAT3D seems only being trained with 1cond+7target and 3cond+5target. Could it address the combination of arbitrary conditions and targets without fine-tuning? For example, 4cond+4target or 2cond+6target.\n5. CAT3D enjoys good generalization, which is just trained on objaverse, CO3D, MVImgNet, and RealEstate10k. However, these datasets are all constrained (no background: objaverse; object-centric: CO3D, MVImgNet, most indoor scenes: RealEstate10k). How to confirm the generalization of full-model trainable StableDiffusion, especially for some text-to-image samples shown in the supplementary? This point is not mentioned in the main paper.\n6. Unfortunately, the authors did not promise to release the code as shown in the checklist. Therefore, some implementation details should be further clarified as mentioned in the \"Questions\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and careful review of our work. Below we address your questions and weaknesses mentioned:\n\n> Most techniques have been proposed.\n\nWe agree with you that CAT3D leverages existing techniques for individual components of the system. The innovation of CAT3D lies in the effective integration and scalability of those components: a multi-view diffusion model, efficient sampling strategies, and a robust 3D reconstruction pipeline, resulting in a powerful and practical system for 3D content creation given any number of input views. CAT3D decouples the generative prior from the 3D extraction process, which not only contributes to its efficiency and simplicity, but also allows for potential improvements in either component without affecting the other in future research.\n\n> Results from the 3D reconstruction / multi-view diffusion. \n\nAll results in the main text are renderings of optimized mip-NeRF models, but the Appendix also includes quantitative results of images produced by the multi-view diffusion model (Table 3), and the anonymous website in the supplementary contains qualitative samples from the multi-view diffusion model. We will clarify this in the revised draft, and are happy to include more results from the diffusion model in the main text if the reviewer deems this to be useful.\n\n> Run time on one GPU.\n\nThank you very much for pointing out the differences in timings. We have added a note to the Table indicating that our timings take place on 16xA100 GPUs, and are working to evaluate the timing for a single GPU. One advantage of our approach is that it can benefit from parallelism when generating novel views, while other methods built around distillation and feedforward prediction do not.\n\n> Support arbitrary conditions and targets.\n\nInterestingly, we find our model can work on test settings different from what the model is trained for to some extent. For example, we find that doubling the target views or tripling the conditional views still works (see Figures 2 and 3 in the rebuttal PDF). We still anticipate training or fine-tuning the model on the actual test settings would lead to better results. \n\n> Full-model generalization. \n\nThis is a good question and a valid concern. We tried to verify the generalization ability of our model empirically by, e.g., testing our model on a wide variety of captured or generated images that are far out-of-distribution from what the model is trained on (see gallery.html in the supplementary). In Table 3, we observe that the model initialized from the pretrained text-to-image model performs better than the model initialized from scratch quantitatively (suggesting that the pre-trained priors are still present or useful in some capacity). To further maintain the generalization ability, a possible future direction is to jointly train the model on our datasets mixed with the text-to-image data that the model is pretrained on. \n\n> Images with different aspect ratios\n\nWhile the multi-view latent diffusion model we trained only supports 512x512 images, we found the model still performed well when padding non-square images to square. However, this method often reduces resolution, so we also run our model on a square-cropped version of the inputs, and then compose the square-cropped outputs with the edges from the padded outputs to create a different aspect ratio image. We will add these details to the Appendix.\n\n\n> shift of the noise scheduler. \n\nAs we describe in the text (lines 143-145), we add $\\log(N)$ to the log signal-to-noise ratio, where $N$ is the number of target images. More concretely, assuming for the base text-to-image model each time step $t$ is mapped to a log signal-to-noise ratio $\\log \\lambda_t$; then for our model, $t$ is mapped to $\\log \\lambda_t - \\log(N)$. In settings with a mixed number of target views (e.g. 5 and 7 target views), we pick $N$ to be the smallest number of target views (i.e., 5). The logic is similar to [67] where the noise schedule is shifted for higher resolution generation (see Eqn (4) in [67]).\n\n> Drop the text embedding.\n\nIt is removed completely for model simplicity. While architecture changes like this can impact the stability of fine-tuning, we found that a learning rate warmup is sufficient to mitigate potential instabilities. \n\n> Mask and SD inpainting.\n\nThe mask we used is to specify conditioning vs. target images. It is constant per image (i.e., not spatially varying), whereas the mask in SD inpainting is used for indicating unknown pixels within an image, which may not align with our task. \n\n> 3DGS. \n\nAs far as we know, all radiance field models including 3DGS and NeRFs are data hungry and 3DGS doesn’t necessarily require fewer captured (or generated) images. It’s worth noting that generating novel views is not the main computational bottleneck in the whole system. For example, in the single-image-to-3D setting, it takes 5 seconds to generate 80 views while it takes 55 seconds to run 3D scene extraction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time and critical feedback to improve our work. We appreciate that the reviewers found our method simple, effective, and efficient, leading to a “compelling advancement in 3D content creation.” We address individual questions below, but first highlight some shared concerns.

One of the biggest limitations of our work is the need to specify a desired camera trajectory, along which to generate novel views that supervise 3D reconstruction. Choosing the correct trajectory can be challenging for complex scenes, and we believe this is an exciting direction for future work. We found that simple orbit trajectories and forward-facing explorative trajectories can be useful across a wide variety of inputs (see Supplementary website), but for more complicated scenes a more automated strategy for bespoke trajectory selection would be useful. 

Regarding our autoregressive sampling strategy, we found that first producing anchors and then generating sets of views in parallel produced mostly consistent views while drastically accelerating sampling time. This strategy is distinct from video diffusion models where all frames are generated at once, and allows us to produce 3D scenes faster. 

We’ve aimed to ensure all details needed to reproduce this work are included in the text, and we are happy to iterate with reviewers to ensure this is the case.

Meta-review: This paper was reviewed by four experts in the field.  Based on the reviewers' feedback, the decision is to recommend the paper for acceptance.  The reviewers did raise some valuable comments that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.    We congratulate the authors on the acceptance of their paper!
Decision: Accept (oral)