Title: CAT3D: Create Anything in 3D with Multi-View Diffusion Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D generation

Diffusion Models

3D reconstruction

Generative Models
Abstract: Advances in 3D reconstruction have enabled high-quality 3D capture, but require a user to collect hundreds to thousands of images to create a 3D scene. We present CAT3D, a method for creating anything in 3D by simulating this real-world capture process with a multi-view diffusion model. Given any number of input images and a set of target novel viewpoints, our model generates highly consistent novel views of a scene. These generated views can be used as input to robust 3D reconstruction techniques to produce 3D representations that can be rendered from any viewpoint in real-time. CAT3D can create entire 3D scenes in as little as one minute, and outperforms existing methods for single image and few-view 3D scene creation.

CAT3D: Create Anything in 3D with Multi-View Diffusion Models

Abstract

Advances in 3D reconstruction have enabled high-quality 3D capture, but require a user to collect hundreds to thousands of images to create a 3D scene. We present CAT3D, a method for creating anything in 3D by simulating this real-world capture process with a multi-view diffusion model. Given any number of input images and a set of target novel viewpoints, our model generates highly consistent novel views of a scene. These generated views can be used as input to robust 3D reconstruction techniques to produce 3D representations that can be rendered from any viewpoint in real-time. CAT3D can create entire 3D scenes in as little as one minute, and outperforms existing methods for single image and few-view 3D scene creation.

1 Introduction

The demand for 3D content is higher than ever, since it is essential for enabling real-time interactivity for games, visual effects, and wearable mixed reality devices. Despite the high demand, high-quality 3D content remains relatively scarce. Unlike 2D images and videos which can be easily captured with consumer photography devices, creating 3D content requires complex specialized tools and a substantial investment of time and effort.

Fortunately, recent advancements in photogrammetry techniques have greatly improved the accessibility of 3D asset creation from 2D images. Methods such as NeRF [1], Instant-NGP [2], and Gaussian Splatting [3] allow anyone to create 3D content by taking photos of a real scene and optimizing a representation of that scene's underlying 3D geometry and appearance. The resulting 3D representation can then be rendered from any viewpoint, similar to traditional 3D assets. Unfortunately, creating detailed scenes still requires a labor-intensive process of capturing hundreds to thousands of photos. Captures with insufficient coverage of the scene can lead to an ill-posed optimization problem, which often results in incorrect geometry and appearance and, consequently, implausible imagery when rendering the recovered 3D model from novel viewpoints.

Reducing this requirement from dense multi-view captures to less exhaustive inputs, such as a single image or text, would enable more accessible 3D content creation. Prior work has developed specialized solutions for different input settings, such as geometry regularization techniques targeted for sparse-view reconstruction [4, 5], feed-forward models trained to create 3D objects from single images [6], or the use of image-conditioned [7] or text-conditioned [8] generative priors in the optimization process-but each of these specialized methods comes with associated limitations in quality, efficiency, and generality.

In this paper, we instead focus on the fundamental problem that limits the use of established 3D reconstruction methods in the observation-limited setting: an insufficient number of supervising views. Rather than devising specialized solutions for different input regimes [8, 9, 7], a shared solution is to instead simply create more observations-collapsing the less-constrained, under-determined 3D creation problems to the fully-constrained, fully-observed 3D reconstruction setting. This way, we reformulate a difficult ill-posed reconstruction problem as a generation problem: given any number of input images, generate a collection of consistent novel observations of the 3D scene. Recent video generative models show promise in addressing this challenge, as they demonstrate the capability to synthesize video clips featuring plausible 3D structure [10, 11, 12, 13, 14, 15]. However, these models are often expensive to sample from, challenging to control, and limited to smooth and short camera trajectories.

Our system, CAT3D , instead accomplishes this through a multi-view diffusion model trained specifically for novel-view synthesis. Given any number of input views and any specified novel viewpoints, our model generates multiple 3D-consistent images through an efficient parallel sampling strategy. These generated images are subsequently fed through a robust 3D reconstruction pipeline to produce a 3D representation that can be rendered at interactive rates from any viewpoint. We show that our model is capable of producing photorealistic results of arbitrary objects or scenes from any number of captured or synthesized input views in as little as one minute. We evaluate our work across various input settings, ranging from sparse multi-view captures to a single captured image, and even just a text prompt (by using a text-to-image model to generate an input image from that prompt). CAT3D outperforms prior works for measurable tasks (such as the multi-view capture case) on multiple benchmarks, and is an order of magnitude faster than previous state-of-the-art. For tasks where empirical performance is difficult to measure (such as text-to-3D and single image to 3D), CAT3D compares favorably with prior work in all settings.

2 Related Work

Creating entire 3D scenes from limited observations requires 3D generation, e.g. , creating content in unseen regions, and our work builds on the ever-growing research area of 3D generative models [16]. Due to the relative scarcity of 3D datasets, much research in 3D generation is centered on transferring knowledge learned by 2D image-space priors, as 2D data is abundant. Our diffusion model is built on the recent development of video and multi-view diffusion models that produce highly consistent novel views. We show that pairing these models with 3D reconstruction, similar to [17, 18], enables efficient and high quality 3D creation. Below we discuss how our work is related to several areas of prior work.

2D priors. Given limited information such as text, pretrained text-to-image models can provide a strong generative prior for text-to-3D generation. However, distilling the knowledge present in these image-based priors into a coherent 3D model currently requires an iterative distillation approach. DreamFusion [8] introduced Score Distillation Sampling (SDS) to synthesize 3D objects (as NeRFs) from text prompts. Research in this space has aimed to improve distillation strategies [19, 20, 21, 22,

23], swap in other 3D representations [24, 25, 26, 27, 28], and amortize the optimization process [29]. Using text-based priors for single-image-to-3D has also shown promise [30, 31, 32], but requires a complex balancing of the image observation with additional constraints. Incorporating priors such as monocular depth models or inpainting models has been useful for creating 3D scenes, but tends to result in poor global geometry [33, 34, 35, 36].

2D priors with camera conditioning. While text-to-image models excel at generating visually appealing images, they lack precise control over the pose of images, and thus require a time-consuming 3D distillation process to encourage the 3D model to conform to the 2D prior. To overcome this limitation, several approaches train or fine-tune generative models with explicit image and pose conditioning [7, 37, 38, 39, 40, 41, 42]. These models provide stronger priors for what an object or scene should look like given text and/or input image(s), but they also model all output views independently . In cases where there is little uncertainty in what should appear at novel views, reasoning about generated views independently is sufficient for efficient 3D reconstruction [43]. But when there is some uncertainty exists, these top-performing methods still require expensive 3D distillation to resolve the inconsistencies between different novel views.

Multi-view priors. Modeling the correlations between multiple views provides a much stronger prior for what 3D content is consistent with partial observations. Methods like MVDream [44], ImageDream [9], Zero123++ [45], ConsistNet [46], SyncDreamer [47] and ViewDiff [48] fine-tune text-to-image models to generate multiple views simultaneously. CAT3D is similar in architecture to ImageDream, where the multi-view dependency is captured by an architecture resembling video diffusion models with 3D self-attention. Given this stronger prior, these papers also demonstrate higher quality and more efficient 3D extraction.

Video priors. Video diffusion models have demonstrated an astonishing capability of generating realistic videos [49, 50, 10, 12, 15, 13, 51], and are thought to implicitly reason about 3D. However, it remains challenging to use off-the-shelf video diffusion models for 3D generation for a number of reasons. Current models lack exact camera controls, limiting generation to clips with only smooth and short camera trajectories, and struggle to generate videos with only camera motion but no scene dynamics. Several works have proposed to resolve these challenges by fine-tuning video diffusion models for camera-controled or multi-view generation. For example, AnimateDiff [52] LoRA fine-tuned a video diffusion model with fixed types of camera motions, and MotionCtrl [53] conditioned the model on arbitrary specified camera trajectories. ViVid-1-to-3 [54] combines a novel view synthesis model and a video diffusion model for generating smooth trajectories. SVD-MV [55], IM-3D [17] and SV3D [55] further explored leveraging camera-controlled or multi-view video diffusion models for 3D generation. However, their camera trajectories are limited to orbital ones surrounding the center content. These approaches mainly focus on 3D object generation, and do not work for 3D scenes, few-view 3D reconstruction, or objects in context (objects that have not been masked or otherwise separated from the image's background).

Feed-forward methods. Another line of research is to learn feed-forward models that take a few views as input, and output 3D representations directly, without an optimization process per instance [6, 56, 57, 58, 18, 59, 60]. These methods can produce 3D representations efficiently (within a few seconds), but the quality is often worse than approaches built on image-space priors.

3 Method

CAT3D is a two-step approach for 3D creation: first, we generate a large number of novel views consistent with one or more input views using a multi-view diffusion model, and second, we run a robust 3D reconstruction pipeline on the generated views (see Figure 3). Below we describe our multi-view diffusion model (Section 3.1), our method for generating a large set of nearly consistent novel views from it (Section 3.2), and how these generated views are used in a 3D reconstruction pipeline (Section 3.3).

<latexi sh

1_b

64="

JG

3Pn

2

L

uK

gpER8+

IB

>A

D/X

cdVN

M

HY

m

5

k

U

j7C

y

Wq

v

0

S

f

Q

T

O

w

o

Z

z

9

r

F

<latexi sh

1_b

64="

JG

3Pn

2

L

uK

gpER8+

IB

>A

D/X

cdVN

M

HY

m

5

k

U

j7C

y

Wq

v

0

S

f

Q

T

O

w

o

Z

z

9

r

F

Observed Views and Target Poses

Observed & Generated Views

NeRF

Decoder

Generated Latents

A Target View

Random Latents

Set of observed and

unobserved views

Set of generated views

An Observed View

Input Latents

Encoder

Generated Image

}

}

Multi-View Latent Diffusion Model

3.1 Multi-View Diffusion Model

We train a multi-view diffusion model that takes a single or multiple views of a 3D scene as input and generates multiple output images given their camera poses (where 'a view' is a paired image and its camera pose). Specifically, given M conditional views containing M images I cond and their corresponding camera parameters p cond , the model learns to capture the joint distribution of N target images I tgt assuming their N target camera parameters p tgt are also given:

Model architecture. Our model architecture is similar to video latent diffusion models (LDMs) [49, 11], but with camera pose embeddings for each image instead of time embeddings. Given a set of conditional and target images, the model encodes every individual image into a latent representation through an image variational auto-encoder [61]. Then, a diffusion model is trained to estimate the joint distribution of the latent representations given conditioning signals. We initialize the model from an LDM trained for text-to-image generation similar to [62] trained on web-scale image data, with an input image resolution of 512 × 512 × 3 and latents with shape 64 × 64 × 8 . As is often done in video diffusion models [50, 10, 11], the main backbone of our model remains the pretrained 2D diffusion model but with additional layers connecting the latents of multiple input images. As in [44], we use 3D self-attention (2D in space and 1D across images) instead of simple 1D self-attention across images. We directly inflate the existing 2D self-attention layers after every 2D residual block of the original LDM to connect latents with 3D self-attention layers while inheriting the parameters from the pre-trained model, introducing minimal amount of extra model parameters. We found that conditioning on input views through 3D self-attention layers removed the need for PixelNeRF [63] and CLIP image embeddings [64] used by the prior state-of-the-art model on few-view reconstruction, ReconFusion [7]. We use FlashAttention [65, 66] for fast training and sampling, and fine-tune all the weights of the latent diffusion model. Similar to prior work [10, 67], we found it important to shift the noise schedule towards high noise levels as we move from the pre-trained image diffusion model to our multi-view diffusion model that captures data of higher dimensionality. Concretely, following logic similar to [67], we shift the log signal-to-noise ratio by log( N ) towards the high signal-to-noise ratio region, where N is the number of target views. Similar shifts have been adopted empirically

by [11, 10]. For training, latents of target images are noise perturbed while latents of conditional images are kept as clean, and the diffusion loss is defined only on target images. A binary mask is concatenated to the latents along the channel dimension, to denote conditioning vs. target images. To deal with multiple 3D generation settings, we train a single versatile model that can model a total of 8 conditioning and target views ( N + M = 8 ), and randomly select the number of conditional views M to be 1 or 3 during training, corresponding to 7 and 5 target views respectively. The noise schedule is shifted based on 5 target views. See Appendix B for more model details.

Camera conditioning. To condition on the camera pose, we use a camera ray representation ('raymap') that is the same height and width as the latent representations [38, 68] and encodes the ray origin and direction at each spatial location. The rays are computed relative to the camera pose of the first conditional image, so our pose representation is invariant to rigid transformations of 3D world coordinates. Raymaps for each image are concatenated channel-wise onto the latents for the corresponding image.

3.2 Generating Novel Views

Given a set of input views, our goal is to generate a large set of consistent views to fully cover the scene and enable accurate 3D reconstruction. To do this, we need to decide on the set of camera poses to sample, and we need to design a sampling strategy that can use a multi-view diffusion model trained on a small number of views to generate a much larger set of consistent views.

Camera trajectories. Compared to 3D object reconstruction where orbital camera trajectories can be effective, a challenge of 3D scene reconstruction is that the views required to fully cover a scene can be complex and depend on the scene content. We empirically found that designing reasonable camera trajectories for different types of scenes is crucial to achieve compelling few-view 3D reconstruction. The camera paths must be sufficiently thorough and dense to fully-constrain the reconstruction problem, but also must not pass through objects in the scene or view scene content from unusual angles. In summary, we explore four types of camera paths based on the characteristic of a scene: (1) orbital paths of different scales and heights around the center scene, (2) forward facing circle paths of different scales and offsets, (3) spline paths of different offsets, and (4) spiral trajectories along a cylindrical path, moving into and out of the scene. See Appendix C for more details.

Generating a large set of synthetic views. Achallenge in applying our multi-view diffusion model to novel view synthesis is that it was trained with a small and finite set of input and output views just 8 in total. To increase the total number of output views, we cluster the target viewpoints into smaller groups and generate each group independently given the conditioning views. We group target views with close camera positions, as these views are typically the most dependent. For single-image conditioning, we adopt an autoregressive sampling strategy, where we first generate a set of 7 anchor views that cover the scene (similar to [42], and chosen using the greedy initialization from [69]), and then generate the remaining groups of views in parallel given the observed and anchor views. This allows us to efficiently generate a large set of synthetic views while still preserving both long-range consistency between anchor views and local similarity between nearby views. For the single-image setting, we generate 80 views, while for the few-view setting we use 480-960 views. See Appendix C for details.

Conditioning larger sets of input views and non-square images. To expand the number of views we can condition on, we choose the nearest M views as the conditioning set, as in [7]. We experimented with simply increasing the sequence length of the multi-view diffusion architecture during sampling, but found that the nearest view conditioning and grouped sampling strategy performed better. To handle wide aspect ratio images, we combine square samples from square-cropped input views with wide samples cropped from input views padded to be square.

3.3 Robust 3D reconstruction

Our multi-view diffusion model generates a set of high-quality synthetic views that are reasonably consistent with each other. However, the generated views are generally not perfectly 3D consistent. Indeed, generating perfectly 3D consistent images remains a very challenging problem even for

Input

ReconFusion [7]

CAT3D (ours)

Ground Truth

current state-of-the-art video diffusion models [70]. Since 3D reconstruction methods have been designed to take photographs (which are by definition perfectly consistent) as input, we modify the standard NeRF training procedure to improve its robustness to inconsistent input views.

We build upon Zip-NeRF [71], whose training procedure minimizes the sum of a photometric reconstruction loss, a distortion loss, an interlevel loss, and a normalized L2 weight regularizer. We additionally include a perceptual loss (LPIPS [72]) between the rendered image and the input image. Compared to the photometric reconstruction loss, LPIPS emphasizes high-level semantic similarity between the rendered and observed images, while ignoring potential inconsistencies in low-level high-frequency details. Since generated views closer to the observed views tend to have less uncertainty and are therefore more consistent, we weight the losses for generated views based on the distance to the nearest observed view. This weighting is uniform at the beginning of the training, and is gradually annealed to a weighting function that more strongly penalizes reconstruction losses for views closer to one of the observed views. See Appendix D for additional details.

4 Experiments

We trained the multi-view diffusion model at the core of CAT3D on four datasets with camera pose annotations: Objaverse [73], CO3D [74], RealEstate10k [75] and MVImgNet [76]. We then evaluated CAT3D on the few-view reconstruction task (Section 4.1) and the single image to 3D task (Section 4.2), demonstrating qualitative and quantitative improvements over prior work. The design choices that led to CAT3D are ablated and discussed further in Section 4.3.

4.1 Few-View 3D Reconstruction

We first evaluate CAT3D on five real-world benchmark datasets for few-view 3D reconstruction. Among those, CO3D [74] and RealEstate10K [75] are in-distribution datasets whose training splits were part of our training set (we use their test splits for evaluation), whereas DTU [77], LLFF [78] and the mip-NeRF 360 dataset [79] are out-of-distribution datasets that were not part of the training dataset. We tested CAT3D on the 3 , 6 and 9 view reconstruction tasks, with the same train and eval splits as [7]. In Table 1, we compare to the state-of-the-art for dense-view NeRF reconstruction with no learned priors (Zip-NeRF [71]) and methods that heavily leverage generative priors such as ZeroNVS [42]

and ReconFusion [7]. We find that CAT3D achieves state-of-the-art performance across nearly all settings, while also reducing generation time from 1 hour (for ZeroNVS and ReconFusion) down to a few minutes. CAT3D outperforms baseline approaches on more challenging datasets like CO3D and mip-NeRF 360 by a larger margin, thereby demonstrating its value in reconstructing large and highly detailed scenes. Figure 4 shows the qualitative comparison. In unobserved regions, CAT3D is able to hallucinate plausible textured content while still preserving geometry and appearance from the input views, whereas prior works often produce blurry details and oversmoothed backgrounds.

4.2 Single image to 3D

CAT3D supports the efficient generation of diverse 3D content from just a single input view. Evaluation in this under-constrained regime is challenging as there are many 3D scenes consistent with the single view, for example scenes of different scales. We thus focus our single image evaluation on qualitative comparisons (Figure 5), and quantitative semantic evaluations with CLIP [64] (Table 2). On scenes, CAT3D produces higher resolution results than ZeroNVS [42] and RealmDreamer [80], and for both scenes and objects we better preserve details from the input image. On images with segmented objects, our geometry is often worse than existing approaches like ImageDream [9] and DreamCraft3D [81], but maintains competitive CLIP scores. Compared to these prior approaches that iteratively leverage a generative prior in 3D distillation, CAT3D is more than an order of magnitude faster. Faster generation methods have been proposed for objects [6, 82, 83], but produce significantly lower resolution results than their iterative counterparts, so they are not included in this comparison. IM-3D [17] achieves better performance on segmented objects with similar runtime, but does not work on scenes, or on objects in context.

4.3 Ablations

At the core of CAT3D is a multi-view diffusion model that has been trained to generate consistent novel views. We considered several model variants, and evaluated both their sample quality (on

Input

CAT3D (ours)

RealmDreamer[80]

ZeroNVS [42]

ImageDream [9]

DreamCraft3D [81]

Baselines

±

in-domain and out-of-domain datasets) and few-view 3D reconstruction performance. We also compare important design choices for 3D reconstruction. Results from our ablation study are reported in Table 3 and Figure 6 in Appendix A and summarized below. Overall, we found that video diffusion architectures, with 3D self-attention (spatiotemporal) and raymap embeddings of camera pose, produce consistent enough views to recover 3D representations when combined with robust reconstruction losses.

Image and pose. Previous work [7] used PixelNerf [63] feature-map conditioning for multiple input views. We found that replacing PixelNeRF with attention-based conditioning in a conditional video diffusion architecture using a per-image embedding of the camera pose results in improved samples and 3D reconstructions, while also reducing model complexity and the number of parameters. We found that embedding the camera pose as a low-dimensional vector (as in [37]) works well for in-domain samples, but generalizes poorly compared to raymap conditioning (see Section 3.1).

Increasing the number of views. We found that jointly modeling multiple output views ( i.e. , 5 or 7 views instead of 1) improves sample metrics - even metrics that evaluate the quality of each output image independently . Jointly modeling multiple outputs creates more consistent views that result in an improved 3D reconstruction as well.

Attention layers. We found that 3D self-attention (spatiotemporal) is crucial, as it yields improved performance relative to factorized 2D self-attention (spatial-only) and 1D self-attention (temporalonly). While models with 3D self-attention in the finest feature maps ( 64 × 64 ) result in the highest fidelity images, they incur a significant computational overhead for training and sampling for relative small gain in fidelity. We therefore decided to use 3D self-attention only in feature maps of size 32 × 32 and smaller.

Multi-view diffusion model training. Initializing from a pre-trained text-to-image latent diffusion model improved performance on out-of-domain examples. We experimented with fine-tuning the multi-view diffusion model to multiple variants specialized for specific numbers of inputs and outputs views, but found that a single model jointly trained on 8 frames with either 1 or 3 conditioning views was sufficient to enable accurate single image and few-view 3D reconstruction.

3D reconstruction. LPIPS loss is crucial for achieving high-quality texture and geometry, aligning with findings in [17, 7]. On Mip-NeRF 360, increasing the number of generated views from 80 (single elliptical orbit) to 720 (nine orbits) improved central object geometry but sometimes introduced background blur, probably due to inconsistencies in generated content.

5 Discussion

We present CAT3D, a unified approach for 3D content creation from any number of input images. CAT3D leverages a multi-view diffusion model for generating highly consistent novel views of a 3D scene, which are then input into a 3D multi-view reconstruction pipeline. CAT3D decouples the generative prior from 3D extraction, leading to efficient, simple, and high-quality 3D generation.

Although CAT3D produces compelling results and outperforms prior works on multiple tasks, it has limitations. Because our training datasets have roughly constant camera intrinsics for views of the same scene, the trained model cannot handle test cases well where input views are captured by multiple cameras with different intrinsics. The generation quality of CAT3D relies on the expressivity of the base text-to-image model, and it performs worse in the cases where scene content is out of distribution for the base model (e.g. human faces, since the base model was trained on limited human data). The number of output views supported by our multi-view diffusion model is still relatively small, so when we generate a large set of samples from our model, not all views may be 3D consistent with each other (see Supplementary website). Finally, CAT3D uses manually-constructed camera trajectories that cover the scene thoroughly (see Appendix C), which may be difficult to design for large-scale open-ended 3D environments.

There are a few directions worth exploring in future work to improve CAT3D. The multi-view diffusion model may benefit from being initialized from a pre-trained video diffusion model, as observed by [10, 17]. The consistency of samples could be further improved by extending the number of conditioning and target views handled by the model. Automatically determining the camera trajectories required for different scenes could increase the flexibility of the system.

Acknowledgements We would like to thank Daniel Watson, Rundi Wu, Jason Y. Zhang, Richard Tucker, Jason Baldridge, Michael Niemeyer, Rick Szeliski, Dana Roth, Jordi Pont-Tuset, Andeep Torr, Irina Blok, Doug Eck, and Henna Nandwani for their valuable contributions to this work. We also extend our gratitude to Shlomi Fruchter, Kevin Murphy, Mohammad Babaeizadeh, Han Zhang and Amir Hertz for training the base text-to-image latent diffusion model.

References

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV , 2020.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. SIGGRAPH , 2022.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. SIGGRAPH , 2023.

Jiawei Yang, Marco Pavone, and Yue Wang. FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization. CVPR , 2023.

Nagabhushan Somraj, Adithyan Karanayil, and Rajiv Soundararajan. SimpleNeRF: Regularizing Sparse Input Neural Radiance Fields with Simpler Solutions. SIGGRAPH Asia , 2023.

Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large Reconstruction Model for Single Image to 3D. arXiv:2311.04400 , 2023.

Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Holynski. Reconfusion: 3d reconstruction with diffusion priors, 2023.

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D Diffusion. ICLR , 2022.

Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv:2312.02201 , 2023.

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv:2311.15127 , 2023.

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. CVPR , 2023.

Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning. arXiv:2311.10709 , 2023.

Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv , 2024.

Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models, 2023.

Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.

Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T Barron, Amit H Bermano, Eric Ryan Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al. State of the art on diffusion models for visual computing. arXiv:2310.07204 , 2023.

Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos. IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation, 2024.

Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model. arXiv:2311.06214 , 2023.

Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. DreamTime: An Improved Optimization Strategy for Text-to-3D Content Creation. arXiv , 2023.

Peihao Wang, Zhiwen Fan, Dejia Xu, Dilin Wang, Sreyas Mohan, Forrest Iandola, Rakesh Ranjan, Yilei Li, Qiang Liu, Zhangyang Wang, et al. SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identity. arXiv , 2023.

Subin Kim, Kyungmin Lee, June Suk Choi, Jongheon Jeong, Kihyuk Sohn, and Jinwoo Shin. Collaborative score distillation for consistent visual editing. NeurIPS , 36, 2024.

Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. NeurIPS , 2023.

Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19740-19750, 2023.

Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. ICCV , 2023.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3D: High-Resolution Text-to-3D Content Creation. CVPR , 2023.

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv:2309.16653 , 2023.

Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. arXiv:2310.08529 , 2023.

Dave Epstein, Ben Poole, Ben Mildenhall, Alexei A Efros, and Aleksander Holynski. Disentangled 3d scene generation with layout learning. arXiv preprint arXiv:2402.16936 , 2024.

Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung-Yi Lin, Ming-Yu Liu, Sanja Fidler, and James Lucas. ATT3D: Amortized Text-to-3D Object Synthesis. ICCV , 2023.

Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruction of any object from a single image. CVPR , 2023.

Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, HsinYing Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors. arXiv:2306.17843 , 2023.

Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior. ICCV , 2023.

Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models. ICCV , 2023.

Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J Fleet. Monocular depth estimation using diffusion models. arXiv:2302.14816 , 2023.

Hong-Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T Freeman, Forrester Cole, Deqing Sun, Noah Snavely, Jiajun Wu, et al. WonderJourney: Going from Anywhere to Everywhere. arXiv:2312.03884 , 2023.

Ethan Weber, Aleksander Hoły´ nski, Varun Jampani, Saurabh Saxena, Noah Snavely, Abhishek Kar, and Angjoo Kanazawa. Nerfiller: Completing scenes via generative 3d inpainting. arXiv preprint arXiv:2312.04560 , 2023.

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-Shot One Image to 3D Object. arXiv , 2023.

Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. arXiv:2210.04628 , 2022.

Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Ben Mildenhall, Nataniel Ruiz, Shiran Zada, Kfir Aberman, Michael Rubenstein, Jonathan Barron, Yuanzhen Li, and Varun Jampani. DreamBooth3D: Subject-Driven Text-to-3D Generation. ICCV , 2023.

Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion. ICML , 2023.

Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. GeNVS: Generative novel view synthesis with 3D-aware diffusion models. arXiv , 2023.

Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, et al. ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image. CVPR , 2024.

Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization. arXiv , 2023.

Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. MVDream: Multi-view Diffusion for 3D Generation. arXiv , 2023.

Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model, 2023.

Jiayu Yang, Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong Li. ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion. arXiv:2310.10343 , 2023.

Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. SyncDreamer: Generating Multiview-consistent Images from a Single-view Image. arXiv , 2023.

Lukas Höllein, Aljaž Božiˇ c, Norman Müller, David Novotny, Hung-Yu Tseng, Christian Richardt, Michael Zollhöfer, and Matthias Nießner. Viewdiff: 3d-consistent image generation with text-to-image models, 2024.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv:2204.03458 , 2022.

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv:2210.02303 , 2022.

Siddhant Jain, Daniel Watson, Eric Tabellion, Aleksander Hoły´ nski, Ben Poole, and Janne Kontkanen. Video interpolation with diffusion models. arXiv preprint arXiv:2404.01203 , 2024.

Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 , 2023.

Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.03641 , 2023.

Jeong-gi Kwak, Erqun Dong, Yuhe Jin, Hanseok Ko, Shweta Mahajan, and Kwang Moo Yi. ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models. arXiv:2312.01305 , 2023.

Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion, 2024.

Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas O˘ guz. 3DGen: Triplane Latent Diffusion for Textured Mesh Generation. arXiv:2303.05371 , 2023.

Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Viewset Diffusion: (0)Image-Conditioned 3D Generative Models from 2D Data. ICCV , 2023.

Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang. DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Model, 2023.

Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. arXiv:2312.13150 , 2023.

Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting. arXiv:2404.19702 , 2024.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 , 2013.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. CVPR , 2022.

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural Radiance Fields from One or Few Images. CVPR , 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. ICML , 2021.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. NeurIPS , 35, 2022.

Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv:2307.08691 , 2023.

Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images. ICML , 2023.

Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Luˇ ci´ c, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations. CVPR , 2022.

David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In ACM-SIAM Symposium on Discrete Algorithms , 2007.

Ayush Sarkar, Hanlin Mai, Amitabh Mahapatra, Svetlana Lazebnik, David A Forsyth, and Anand Bhattad. Shadows Don't Lie and Lines Can't Bend! Generative Models don't know Projective Geometry... for now. arXiv:2311.17138 , 2023.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields. ICCV , 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. CVPR , 2018.

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. CVPR , 2023.

Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction. ICCV , 2021.

Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. SIGGRAPH , 2018.

Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. MVImgNet: A Large-scale Dataset of Multi-view Images. CVPR , 2023.

Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. CVPR , 2014.

Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. SIGGRAPH , 2019.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. CVPR , 2022.

Jaidev Shriram, Alex Trevithick, Lingjie Liu, and Ravi Ramamoorthi. RealmDreamer: TextDriven 3D Scene Generation with Inpainting and Depth Diffusion, 2024.

Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior. arXiv , 2023.

Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers, 2023.

Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image, 2024.

Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv:2311.07885 , 2023.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

A Ablations

Here we conduct ablation studies over several important decisions that led to CAT3D. We consider several multi-view diffusion model variants, and evaluated them on novel-view synthesis with held-out validation sets ( 4 k samples) from the training datasets ( in-domain samples ) and from the mip-NeRF 360 dataset ( out-of-domain samples ), as well as 3-view reconstruction on the mip-NeRF 360 dataset ( out-of-domain renders ). Unless otherwise specified, models in this section are trained with 3 input views and 5 output views, evaluated after 120 k optimization iterations. Quantitative results are summarized in Table 3. Then we discuss important 3D reconstruction design choices, with qualitative comparison in Figure 6.

Number of target views. We start from the setting where the model takes 3 conditional views as input and generates a single target view, identical to the ReconFusion baseline [7]. Results show that our multi-view architecture is favored when dealing with multiple input views, compared to the PixelNeRF used by [7]. We also show that going from a single target view to 5 target views results in a significant improvement on both novel-view synthesis and few-view reconstruction.

Camera conditioning. As mentioned in Section 3.1, we compared two camera parameterizations, one is an 8-dimensional encoding vector fed through cross-attention layers that contains relative position, relative rotation quaternion, and absolute focal length. The other is camera rays fed through channel-wise concatenation with the input latents. We observe that the latter performs better across all metrics.

Attention layers. An important design choice is the type and number of self-attention layers used when connecting multiple views. As shown in Table 3, it is critical to use 3D self-attention instead of temporal 1D self-attention. However, 3D attention is expensive; 3D attention at the largest feature maps ( 64 × 64 ) is of 32 k sequence length and this incurs a significant computation overhead during training and sampling with a marginal performance gain, especially for out-of-domain samples and renderings. We therefore chose to use 3D attention only for feature maps of size 32 × 32 and smaller.

Multi-view diffusion model training. Wecompared the settings of training the multi-view diffusion models from scratch with initializing from a pre-trained text-to-image latent diffusion model. The latter performs better, especially for out-of-domain cases. We further trained the model for more iterations and observed a consistent performance gain up until 1 Miterations. Then we fine-tuned the model for handling both cases of 1 conditional + 7 target views and 3 conditional + 5 target views ( jointly ), for another 0 . 4 Miterations. We found this joint finetuning leads to better in-domain novel-view synthesis results with 3 conditional views, and out-of-domain results that are on-par with the previous model.

3D reconstruction. We found perceptual distance (LPIPS) loss is crucial in recovering high-quality texture and geometry, similar to [17, 7]. We also compared the use of 80 views along one orbital path with 720 views along nine variably scaled orbital paths. In the Mip-NeRF 360 setting, increasing the number of views helps better regularize the geometry of central objects, but sometimes leads to blurrier textures in the background, due to inconsistencies in generated content.

B Details of Multi-View Diffusion Model

We initialize the multi-view diffusion model from a latent diffusion model (LDM) trained for text-toimage generation similar to [62] trained on web scale image datasets. See Figure 7 for a visualization of the model architecture. We modify the LDM to take multi-view images as input by inflating the 2D self-attention after every 2D residual blocks to 3D self-attention [44]. Our model adds minimal additional parameters to the backbone model: just a few additional convolution channels at the input layer to handle conditioning information. We drop the text embedding from the original model. Our latent diffusion model has 850 Mparameters, smaller than existing approaches built on video diffusion models such as IM-3D [17] (4.3B) and SV3D [55] (1.5B).

We fine-tune the full latent diffusion model for 1.4M iterations with a batch size of 128 and a learning rate of 5 × 10 -5 . The first 1 Miterations are trained with the setting of 1 conditional view and 7

Table 3: Ablation study of multi-view diffusion models. A comparison of model variants and parameters settings across in-domain sequences (a mixture of Objaverse, Co3D, RealEstate10k and MVImgNet - all sequences in the corresponding eval sets of our training data) and out-of-domain sequences (examples from the mip-NeRF 360 dataset). We evaluate the quality of samples from the diffusion model, as well as renderings from the subsequently optimized NeRF.

target views, while the rest 0 . 4 Miterations are trained with an equal mixture of 1 condition + 7 target views and 3 conditional + 5 target views. Our model was trained for 16 days on 128 TPU-v4 chips. Following [7] we draw training samples with equal probability from the four training datasets. We enable classifier-free guidance (CFG) [85] by randomly dropping the conditional images and camera poses with a probability of 0 . 1 during training.

C Details of Generating Novel Views

We use DDIM [86] with 50 sampling steps and CFG guidance weight 3 for generating novel views. It takes 5 seconds to generate 80 views on 16 A100 GPUs. As mentioned in Section 3.2, selecting camera trajectories that fully cover the 3D scene is important for high-quality 3D generation results. See Figure 8 for an illustration of the camera trajectories we use. For the single image-to-3D setting we use two different types of camera trajectories, each containing 80 views:

A spiral around a cylinder-like trajectory that moves into and out of the scene.

An orbit trajectory for images with a central object.

For few-view reconstruction, we create different trajectories based on the characteristics of different datasets:

RealEstate10K: we create a spline path fitted from the input views, and shift the trajectories along the xz -plane by certain offsets, resulting in 800 views.

LLFF and DTU: we create a forward-facing circle path fitted from all views in the training set, scale it, and shift along the z -axis by certain offsets, resulting in 960 and 480 views respectively.

CO3D: we create a spline path fitted from the input views, and scale the trajectories by multiple factors, resulting in 640 views.

Mip-NeRF 360: we create a elliptical path fitted from all views in the training set , scale it, and shift along the z -axis by certain offsets, resulting in 720 views.

For generating anchor views in the single-image setting, we used the model with 1 conditional input and 7 target outputs. For generating the full set of views (both in the single-image anchored setting,

(c) Without LPIPS loss

as well as in the multi-view setting), we used the model with 3 conditional inputs and groups of 5 target outputs, selected by their indices in the camera trajectories.

Anchor selection. For single-image conditioning, we first select a set of target viewpoints as anchors to ground the scene content. To select a set of anchor views that are spread throughout the scene and provide good coverage, we use the initialization strategy from [69]. This method greedily selects the camera whose position is furthest away from the already selected views. We found this to work well on a variety of trajectory-like view sets as well as random views that have been spread throughout a scene.

Dealing with non-square images. While the multi-view latent diffusion model we trained only supports 512 × 512 square images, we found the model still performed well when padding non-square images to square. However, this method often reduces resolution, so we also run our model on a square-cropped version of the inputs, and then compose the square-cropped outputs with the edges from the padded outputs to create a different aspect ratio image.

D Details of 3D Reconstruction

For the Zip-NeRF [71] baseline, we follow [7] to make a few modifications of the hyperparameters (See Appendix D in [7]) that better suit the few-view reconstruction setting. We use a smaller view dependence network with width 32 and depth 1, and a smaller number of training iterations of 1000, which helps avoid overfitting and substantially speeds up the training and rendering. Our synthetic view sampling and 3D reconstruction process is run on 16 A100 GPUs. For few-view reconstruction,

1/2×

8×64×64

8×8×8

2×

Upsampling

Block

Middle Conv2D

Stack

2x

Spatial Conv2D

Stack

Spatiotemporal

Joint Attention

Joint Attention

B T H W C -> B (T H W) C

B (T H W)  C -> B T H W C

Flash Attention

Downsampling

Block

1/2×

8×32×32

2×

Upsampling

Block*

Downsampling

Block

1/2×

8×16×16

2×

Upsampling

Block*

Downsampling

Block

Input Images

Output Images

Encoder

Decoder

8×512×512

Relative camera raymap

Observed mask

Skip

Skip

Skip

we sample 128 × 128 patches of rays and train with a global batch size of 1 Mthat takes 4 minutes to train. For single image to 3D, we use 32 × 32 patches of rays and a global batch size of 65 k that takes 55 seconds to train. Learning rate is logarithmically decayed from 0 . 04 to 10 -3 . The weight of the perceptual loss (LPIPS) is set to 0 . 25 for single image to 3D and few-view reconstruction on RealState10K, LLFF and DTU datasets, and to 1 . 0 for few-view reconstruction on CO3D an MipNeRF-360 dataets.

Distance based weighting. We design a weighting schedule that upweights views closer to captured views in the later stage of training to improve details. Specifically, the weighting is given by a Gaussian kernel: w ∝ exp ( -bs 2 ) , where s is the distance to the closest captured view and b is a scaling factor. For few-view reconstruction, b is linearly annealed from 0 to 15 . We also anneal the weighting of generated views globally to further emphasize the importance of captured views.

(a) Single image to 3D, forward

(b) Single image to 3D, orbit

(c) RealEstate10K

(e) DTU

(f) CO3D

(g) Mip-NeRF 360

XO

0.2

0.3

0.4

1.5

0.5Z

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims around performance, runtime, and comparison to existing methods are described in the experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our method for certain categories, generalizing to different camera intrinsics, genearting large sets of consistent views, and requiremenet of manually constructed camera trajectories in the discussion section. We also describe limitations in the experiments section, e.g. image-to-3D for segmented objects having room for improvement.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: No theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We aimed to provide the full details necessary to reproduce these results, including architecture details and a diagram, description of the training datasets and procedures, evaluation protocols for 3D, and several detailed appendices.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We have not open sourced the code used in this work, but the datasets we used are all publicly available (Re10K, CO3D, MVImgNet, Objaverse).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide details on the diffusion model architecture in Appendix B, including learning rate and optimization hyperparameters, data mixtures, CFG masking probability, and more. The dataset splits were used and shared in previous work.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: For few-view reconstruction metrics, we follow prior work and do not report error bars. For single-image CLIP metrics we use standard deviation over prompts, as in prior work. Evaluation of 3D generative models is challenging, and better metrics are an active area of research.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: For 3D reconstruction, Appendix D describes the compute requirements and iterations. For diffusion model training, the compute requirements are specified in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We believe we have followed the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: We currently discuss mostly the positive impacts of this work, but can add more acknowledgement of the potential for harm caused by 3D generative models if the reviewers believe it is useful.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: We do not plan to release these models and have not planned safeguards around the release.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the papers that created the datasets we used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We produce 3D assets in the form of videos and 3D models that have been linked to in the supplementary material.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Paper does not involve crowdsourcing nor resarch with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed CAT3D, a pipeline that enabled the production of 3D representations from one or a few input views. CAT3D comprises a multi-view generation model to synthesize novel images from different viewpoints and a Zip-Nerf to achieve 3D reconstruction based on generated views. The 3D results shown in this paper are very impressive with high quality, and the authors provided sufficient experiments to verify the effectiveness of CAT3D.\n\nSTRENGTHS:\n1. CAT3D achieves impressive results of 3D generation.\n2. The overall pipeline of multi-view generation is straightforward, yet effective with good robustness as verified in the experiments.\n3. Although CAT3D is trained with constrained multi-view images (no background: objaverse; object-centric: CO3D, MVImgNet, most indoor scenes: RealEstate10k), it still enjoys good generalization. \n4. This paper can be seen as evidence to verify that scaling up 3D generation through multi-view synthesis is feasible.\n5. The ablation study is convincing and sufficient.\n\nWEAKNESSES:\n1. Most techniques have been proposed by previous works, including the raymap, Nerf with LPIPS loss (IM-3D). But I think this point is not the main issue of this paper, while CAT3D proves the scalability of combining all these techniques together.\n2. Since ZIP-Nerf is utilized in CAT3D for dense 3D reconstruction, it is unclear whether the results shown in Figures 1, 2, 4, and 5 in the paper are derived from ZIP-Nerf or generated from the multi-view diffusion process. Are all these results exclusively from ZIP-Nerf? If not, it would be beneficial to provide more results directly from the diffusion model. This distinction is important, especially considering the potential inconsistencies noted in the limitations section.\n3. It is not inappropriate to provide the inference efficiency with 16 A100 GPUs (Line573) only, which is not a regular setting for most users. The authors should provide the inference time with one GPU. More importantly, it is unclear whether the efficiency results from Table 2 are all fairly compared with one GPU or the same hardware condition.\n4. CAT3D seems only being trained with 1cond+7target and 3cond+5target. Could it address the combination of arbitrary conditions and targets without fine-tuning? For example, 4cond+4target or 2cond+6target.\n5. CAT3D enjoys good generalization, which is just trained on objaverse, CO3D, MVImgNet, and RealEstate10k. However, these datasets are all constrained (no background: objaverse; object-centric: CO3D, MVImgNet, most indoor scenes: RealEstate10k). How to confirm the generalization of full-model trainable StableDiffusion, especially for some text-to-image samples shown in the supplementary? This point is not mentioned in the main paper.\n6. Unfortunately, the authors did not promise to release the code as shown in the checklist. Therefore, some implementation details should be further clarified as mentioned in the \"Questions\".\n\nQUESTIONS:\nSome unclear implementation:\n\n1. Lines 190-191 are unclear. How to deal with images with different aspect ratios during training and inference?\n2. Missing details about the shift of the noise scheduler.\n3. What is meant by \"drop the text embedding\"? Does it imply using an empty string \"\" as the input text for all samples, or does it completely remove all cross-attention layers? Additionally, how is stability maintained when all cross-attention layers are removed at the beginning of training?\n\nSome other minor questions:\n1. As mentioned in the paper, a mask channel is concated to the inputs to distinguish conditions and targets. Why not try to use SD-inpainting as the initialization to cover this task?\n2. Could 3DGS be converged with fewer generated images? Generating 80/720 views is too costly in my opinion.\n\nLIMITATIONS:\nThe authors discuss the limitations. However, no qualitative limitations are shown in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-30"}

comments: SUMMARY:
In this paper, the authors propose a two-stage method for 3D creation. Specifically, they introduce a multi-view diffusion model to generate novel views given observed input views. Then using these views, they perform a robust 3D reconstruction using a Zip-NeRF variant. To generate consistent views, they also design a data-dependent sampling strategy. In addition, they also conduct extensive experiments to validate the effectiveness of the proposed method.

STRENGTHS:
+ The proposed method is simple but effective. It also obtains a superior 3D creation on scenes.
+ The experiments are exhaustive and validate each components comprehensively, making this paper solid.
+ This paper is easy to follow and provides much details. Thus, it is easy to reproduce it.

WEAKNESSES:
- It is certain that using larger diffusion models can boost the performance. But it is interesting to showcase the improvement trend with increased diffusion models.

QUESTIONS:
See the Weaknesses.

LIMITATIONS:
Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CAT3D, a novel approach for generating 3D representations from a flexible number of input images. The authors tackle the challenge of limited input data, a common bottleneck for 3D reconstruction, by leveraging the power of multi-view diffusion models. Their method generates a collection of novel viewpoints consistent with the input, effectively transforming a sparse-view reconstruction problem into a more manageable dense-view scenario. These generated views are then fed into a robust 3D reconstruction pipeline based on a modified Zip-NeRF to produce the final 3D model.\n\nThe authors demonstrate impressive results on various benchmarks, showcasing CAT3D's ability to handle single images, sparse multi-view captures, and even text prompts as input. The method exhibits state-of-the-art performance on few-view 3D reconstruction tasks, outperforming existing methods in terms of both speed and accuracy on established datasets. While single-image 3D generation shows promise, the authors acknowledge the performance is not yet on par with leading methods specifically designed for that task, particularly for single objects. The paper presents a compelling advancement in 3D content creation by unifying different input modalities within a single framework and showcasing significant efficiency gains.\n\nSTRENGTHS:\n- The paper presents a novel approach to 3D content creation by reframing the challenge of sparse input as a view generation problem. This core idea of generating the data needed for robust reconstruction is a valuable contribution to the field.\n\n- The authors demonstrate the effectiveness of CAT3D through comprehensive experiments on established benchmarks. Their results on few-view 3D reconstruction tasks are particularly impressive, showcasing state-of-the-art performance on standard metrics and surpassing existing techniques in terms of both speed and accuracy. The ablation study is well-executed, providing valuable insights into the contribution of different components of their method.\n\n- The paper is generally well-written and easy to follow. The authors clearly motivate their work, provide sufficient background information, and describe their methodology in a structured manner. The figures are informative and complement the textual descriptions well.\n\n- The ability to generate high-quality 3D content from a flexible number of input images, as CAT3D aims to achieve, has substantial practical significance. This flexibility is highly desirable for various applications. The demonstrated speed improvements over existing iterative optimization methods further add to its potential impact by enabling more efficient workflows.\n\nWEAKNESSES:\n- While CAT3D aims to handle sparse inputs, its dependence on calibrated camera poses, presents a significant limitation. How does performance degrade with increasing sparsity and decreasing pose accuracy? \n\n- The paper relies on manually designed camera trajectories for novel view synthesis, which limits practicality and scalability. The authors briefly mention adapting trajectories based on scene characteristics but provide no concrete details. Developing an automated trajectory selection or optimization procedure, potentially guided by learned priors or scene understanding techniques, would significantly enhance the method's value and broader applicability.\n\n- The paper, at times, overstates its accomplishments (e.g., \"achieving state-of-the-art performance across nearly all settings\") and does not adequately address its limitations. A more nuanced and critical self-assessment would strengthen the work.\n\nQUESTIONS:\n- Could the authors provide a more quantitative assessment of how performance degrades with increasing pose noise or sparsity? \n\n- Have the authors considered incorporating an automatic trajectory optimization scheme within CAT3D? \n\n- While open-sourcing the code and trained models is ideal, could the authors at least elaborate on their plans for sharing their work and facilitating reproducibility? Providing more details about the training procedure and hyperparameters would also be beneficial.\n\nLIMITATIONS:\nThe authors list several limitations, including the reliance on constant camera intrinsics during training, the expressiveness limits of the base text-to-image model, the small number of output views, and the need for manual camera trajectories. However, the discussion lacks concrete examples or quantifiable measures of the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe objective of this paper is to achieve single-view or few-view to 3D. The core of their method lies in a multi-image-based diffusion model that leverages 3D attention and raymap encoding for the camera poses.  This setup is different from concurrent work, IM-3D, which repurposes video generation model to achieve 3D, or ReconFusion, which iteratively refine novel view with diffusion model conditioned on PixelNeRF embeddings. Experiments are conducted against several competitive Baselines, like IM-3D and ReconFusion. Result-wise, their proposed method showcases a nice balance between quality and efficiency.\n\nSTRENGTHS:\n1. The quality of the generated/reconstructed 3D scene is state-of-the-art, and it is more efficient compared with ReconFusion and other iterative methods.\n\n2. The proposed multi-view diffusion model is effective. In LRM related literature, image-based diffusion is tricky to generate multi-view consistent. The proposed CAT3D certainly gives better multi-view image generation quality without explicit maintaining a 3D representation, which can benefit many related tasks. \n\n3. The paper is clearly motivated, and the experiments are carefully designed and reported.\n\nWEAKNESSES:\n1. It seems cumbersome to generate a large number of viewpoints(Line:174)  by doing anchor first, and enriching frames in between by repeated running CAT3D. Indeed, more view generated together from the multi-view diffusion is good. But running it iteratively will still produce inconsistencies over runs over different set of camera viewpoints, as is can be seen from Fig.6. \n\n2. The trajectory shape seems to influence the quality of the multi-view diffusion. This limits its generalizability to arbitrary/scattered image viewpoints towards the same scene target, which are common in daily life.\n\nQUESTIONS:\n1. What happens if we only use anchor views to run recon in step2? How the quality compared with the current setup?\n\n2. What if we densify view through some existing video interpolation model (instead of re-run CAT3D)? Would be more efficient and better quality?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}