Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text
Authors: Xinyang Li
Zhangyu Lai
Linning Xu
Yansong Qu
Liujuan Cao
Shengchuan Zhang
Bo Dai
Rongrong Ji
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-3D Scene Generation
Latent Diffusion Models
Primary area: diffusion_based_models
TL;DR: automatically generate realistic 3D scenes and camera trajectories from texts

Abstract: Recent advancements in 3D generation have leveraged synthetic datasets with ground truth 3D assets and predefined camera trajectories. However, the potential of adopting real-world datasets, which can produce significantly more realistic 3D scenes, remains largely unexplored. In this work, we delve into the key challenge of the complex and scene-specific camera trajectories found in real-world captures. We introduce Director3D, a robust open-world text-to-3D generation framework, designed to generate both real-world 3D scenes and adaptive camera trajectories. To achieve this, (1) we first utilize a Trajectory Diffusion Transformer, acting as the \emph{Cinematographer}, to model the distribution of camera trajectories based on textual descriptions. Next, a Gaussian-driven Multi-view Latent Diffusion Model serves as the \emph{Decorator}, modeling the image sequence distribution given the camera trajectories and texts. This model, fine-tuned from a 2D diffusion model, directly generates pixel-aligned 3D Gaussians as an immediate 3D scene representation for consistent denoising. Lastly, the 3D Gaussians are further refined by a novel SDS++ loss as the \emph{Detailer}, which incorporates the prior of the 2D diffusion model. Extensive experiments demonstrate that Director3D outperforms existing methods, offering superior performance in real-world 3D generation.

Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text
Xinyang Li Zhangyu Lai' Linning Xu² Yansong Qu'
Liujuan Cao Shengchuan Zhang' Bo Dai Rongrong Ji
Key Laboratory of Multimedia Trusted Perception and Efficient Computing,
Ministry of Education of China, Xiamen University
2 Shanghai Artificial Intelligence Laboratory.
The Chinese University of Hong Kong, University Hong Kong
Code: https: / httpg//tttuuuo..oiiaaaaeoe http://www..oooooiaaaaaannaaaaaa acinyagg/drrdettrrrrr
Dec C &
rocky
Cnematograhhe
remole mounhein pies
covered village night
Director3D
Detaler
Given textual descriptions, Director3D employs three key components: the Cinematogra- pher generates the camera trajectories, the Decorator creates the initial 3D scenes, and the Detailer refines the details.
Figure 1:
Abstract
Recent advancements in 3D generation have leveraged synthetic datasets with ground truth 3D assets and predefined camera trajectories. However, the potential of adopting real-world datasets, which can produce significantly more realistic 3D scenes, remains largely unexplored. In this work, we delve into the key challenge of the complex and scene-specific camera trajectories found in real-world captures. We introduce Director3D, a robust open-world text-to-3D generation framework, designed generate both real-world 3D scenes and adaptive camera trajectories. To achieve this, (1) we first utilize a Trajectory Diffusion Transformer, acting as the Cinematographer.
model the distribution of camera trajectories based on textual descriptions. (2) Next, Model serves as the Decorator, modeling the image sequence distribution given the camera trajectories and texts. This model, fine-tuned from a 2D diffusion model, directly generates pixel-aligned 3D Gaussians as an immediate 3D scene representation for consistent denoising. (3) Lastly, the 3D Gaussians are further refined by a novel SDS+ loss as the Detailer, which incorporates the prior of the 2D diffusion model. Extensive experiments demonstrate that Director3D outperforms existing methods, offering superior performance in real-world 3D generation.
couussian-drvven Multi-view Latent Diffusion
Work done during an internship of Xinyang Li with Shanghai Artificial Intelligence Laboratory 1 Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A collection fresh vegetables arranged wicker basket
A beld head of man
A snowy woodland path lined with frost-covered trees and
A cluster of tents pitched near
A stainless steel loaster sitting on marble countertop
paint-splatiered easel swan on tranquil lake
expansive badllands lerrain, vibrant markelplace early where jagged rock formations..
moriin....
faux-fur leopard print hat
derelict space station orbiting desolate planet...
sleep gorge carved by ancient rivers, with rocky walls and...
Figure 2: Multi-view image results rendered with the generated camera trajectories and 3D scenes.
Introduction
Generating 3D scenes from texts holds great promise for industries such as gaming, robotics, and VR/AR. Previous methods [2233 which use score distillation sampling (SDS) to optimize 3D representations such as Neural Radiance Fields (NeRFs) 51, involve lengthy and unstable optimiza- tion processes. In contrast, newer approaches employ feed-forward networks 6 7 e.g., diffusion and reconstruction models, to directly generate 3D representations from text or text-guided multi- view images, significantly enhancing generation speed. Moreover, advancements in 3D Gaussian Splatting [8] further accelerate training and rendering speeds, driving the next wave of progress in text-to-3D generation I0 12 [3]. However, most existing methods focus solely object-level 3D generation. Recently, preliminary works [14] [5 16 [7 [18] have begun addressing scene-level 3D generation. Despite these efforts, visual quality, generation speed, and generalization remain suboptimal due to reliance on only 2D priors or limited few-classes 3D datasets.
In this work, we leverage real-world datasets (e.g., MVImgNet [19] and DL3DV-1OK (20 to achieve realistic text-to-3D generation. However, real-world captures from in-the-wild scenes differ significantly from traditional object-level synthetic datasets, introducing new requirements for the text-to-3D generation framework. Firstly, real-world captures feature complex, unpredictable, and scene-specific camera trajectories. unlike the controlled and predefined settings in object-level synthetic datasets like Objaverse [21] shown in Fig. (Left.. Secondly, real-world scenes include unbounded backgrounds, complicating the of common bounded 3D representations such as Tri-planes [22). Lastly, the diversity and quantity of real-world captures are limited, potentially decreasing the generalization ability for open-world texts.
We address these challenges with novel framework, Director3D, illustrated in Fig. and Fig. (Right). Fig. 2 shows that our framework supports generating 3D scenes across various domains. In summary, our approach includes the following three key components:
Traj-DiT (Trajectory Diffusion Transformer) as Cinematographer: Generates dense-view camera trajectories from text. Camera parameters (intrinsics and extrinsics) are treated as temporal tokens, and Transformer model performs conditional denoising of the camera trajectory.
Uses a sparse-view subset of the camera trajectory for image sequence diffusion, generating pixel-aligned and unbounded 3D Gaussians intermediate 3D representations. This model, fine-tunels from 2D latent diffusion model, leverages strong priors and collaborative training with multi-view and single-view data to mitigate the limited diversity and quantity of real-world captures, enhancing generalization.
GM-LDM (Gaussian-driven Multi-view Latent Diffusion Model) Decorator:
SDS+ Loss as Detailer: Enhances visual quality of the 3D Gaussians by back- pack-propagates a novel SDS+ loss from images rendered at randomly interpolated cameras within the trajectory.

---

Dense-view Trajectory
Pixel-aligned
Gaussians
Cinemalogmphurr
Dasailer
LDMI
Synthetic Dalaset
Real-world Dalasel
Sparse-view
Continuer-view
Figure 3: Left: Comparison of the simplified camera trajectory distributions between synthetic and real-world multi-view datasets. Right: Pipeline and models of Director3D.
2 Related Works
Text-to-3D Generation. Since the introduction of DreamField 23 which combines the vision- language model CLIP (24) with NeRFs 44, there has been substantial progress in the area of text-to-3D generation. DreamFusion [T] and SJC [25] employ 2D image diffusion models to refine the 3D representation through Score Distillation Sampling (SDS). Subsequent methods [26 27 28
29 30] enhance the formulations to more effectively utilize 2D diffusion models for text-to-3D generation, achieving greater stability and improved visual quality. Some methods [3] B1 32 33
suggest incorporating an additional 3D prior into 2D diffusion models to enhance 3D consistency. Additionally, various methods (34 35
7 38 employ 3D diffusion models to directly produce 3D assets. However, most of these works primarily focus on object-level generation. For scene-
36
37
combine image inpainting 46 to progressively lift an image into a 3D scene with user-defined camera trajectory. Nonetheless, the multi-view inconsistency of image inpainting and monocular depth estimation can lead to undesirable geometry and texture artifacts. propose using panorama image diffusion models for 3D scene generation, which is limited to some specific types of scenes such as indoor rooms. Therefore, developing a text-to-3D scene generation method with open-wott generalization capabílities remains unsolved.
level 3D generation, preliminary works [15
14
39
40
41
16
42
models 43 44 and monocular depth estimation models 45
Also, some works 47 48
3D Gaussian Splatting (3DGS) S introduces parameterized 3D Gaussians as 3D representation and splatting-basee based rasterization technique for novel view synthesis based dense- view images, significantly reducing the rendering time compared to NeRF-based methods
4
50 51..
stabilizing the train- ing [55]. adapting it to dynamic scene modeling [56, 57 58 59 and applying it to different appli-
Current methods concentrate
improving the geometry quality [52
53
54
cations 660 61 62 63 64). Further, to train generalizable sparse-view reconstruction model with
3D Gaussians as the intermediate representation, some methods [65 66 67 38) I2] [ propose to convert image features into pixel-aligned 3D Gaussians and optimize reconstruction models through back prekpprppggg losses from rendered images.
Mcanwhile, DreamGaussian employs 3DGS for eext-to-3D generation, with SDS loss the optimiza- tion objective. DreasernDreamerr [IT] and GSGen [10] further enhance the generation quality and 3D
consistency by initializing the 3D Gaussians based point cloud diffusion models, Point-E [34
The generalizable view reconstruction model (e.g.. GRM (38], LGM 12), and GS-LRM 13]
for 3D Gaussians also facilitate text-to-3D generation, using multi-view diffusion models 3 68
to acquire sparse-view images as inputs. Specifically, GS-LRM takes a step forward by employing large video generation model (i.e.. SORA (69]) for text-to-3D scene generation. However, these works heavily rely on the 3D consistency of the multi-view images, which the 2D-based diffusion models can not guarantee. Distinctively, our GM-LDM employs alilnaligned 3D Gaussians as the intermediate 3D representation for rendering- multi-view diffusion, directly enforcing 3D consistency during the diffusion process and producing 3D representations.
3 Preliminary
Latent Diffusion Models (LDMss) 44 70 consist of two key components: an auto-encoder and latent denoising network. The autoencoder establishes bi-directional mapping from the space of the original data to low-resolution latent space: = E(x), xr = D(=), where and D are the encoder and decoder, respectively. The latent denoising network eg is trained to denoise noisy latent
3

---

given a specific timestep and condition Its training objective for e-prediction is defined as:
L L= L= ||e - ca (=t, ft,21 1)
where the noisy latent obtained by 2 = = vazE(x) +
ag is a monotonically decreasing noise schedule and e ~ (i0, 1) is random noise. During inference, a random noise is sampled as 2T (0, 1). By continuously denoising the random noise 27 with condition y (e.g,, text embedding), we can derive fully denoised latent E. Then, the denoised latent is fed into the latent decoder 'D to generate the high-resolution image = D()).
Multi-view Difftusion Models aim to model the distribution of multi-view images X' with 3D consistency, where each image is captured by distinct camera within the same static 3D scene. Its objective for To -o-prediction can be written as:
L Ex, [|XX Xo y. f))1
(2)
where C' represents the camera parameters for the different views. Early works 3 31 in this field are based on 2D LDMs. They fine-tune the 2D LDMs by integrating cross-view connections between the multi-view images into the original single-view 2D LDMs, using multi-viww data rendered from 3D datasets. These methods lack strict 3D consistency since there is no actual 3D representation during multi-view denoising. A more advanced approach, DMV3D 77]. employs a 3D reconstruction model to generate frese-free 3D representations and predict multi-view images from noisy multi-view inputs by rendering- based denoising process. This enables 3D generation tasks to be accomplished without per-asset optimization during inference.
Score Distillation Sampling (SDS) [ 2 uses a pretrained 2D diffusion model to optimize 3D representation. Considering a differentiable 3D representation parameterized by G and rendering function denoted as R. the rendered image produced for given camera pose can be expressed as R(g, c). SDS distills the prior of 2D LDM to optimize 3D representation & as follows:
VgLSDS = Et,,,c wx()) (2 c) @E(R(G,c)) ag
where is the ground truth noise, is the noise predicted by the 2D LDM with 2, as input for timestep and w(l) represents a weighting function that varies according to the timestep L. The SDS loss can be also converted into a reconstruction-liks objective [72:
LSDS Ersro w(t)
-1
||2
=11
where E = (Ze /1 -a,e\/Vaa.
(4)
VI
āt
4 Method
4.1
Problem Formulation and Overview of Director3D
We consider the multi-view dataset of real-world captures as a joint distribution of image sequences and camera trajectories conditioned on texts, denoted as (((('C)))) Here, X' = { represents the image sequence, C = {c,) MM denotes the camera trajectory, and M is the number of views. To model this joint distribution, we separately handle the conditional distributions p(C) y) and p(X'((, y)) (see Appendix [ for detailed discussions). Furthermore, we model each image in the sequence as rendered view of unified 3D scene representation under the corresponding camera, expressed as ct; R)) where R. is the 3D rendering function.
Director3D addresses this by incorporating three collaborative processes analogous roles in film production: the Cinematographer, the Decorator, and the Detailer. Firstly, the Trajectory Diffusion Transformer (Traj-DiT), serving as the Cinematographer, models the distribution of dense-view camera trajectories, as detailed in Sec. 4.2 For image sequences, directly modeling the dense-view distribution is complex and resource- essource-intensvee intensive. To address this, we use Gaussians-driver Multi-view Latent Diffusion Model (GM-LDM), acting as the Decorator, to model the image distribution through a sparse subset of dense views. This model utilizes pixel-aligned 3D Gaussians as the intermediate representation, described in Sec. 4.3 Finally, to improve the visual quality of the generated 3D scenes, we employ a novel SDS++ loss, functioning as the Detailer, to refine the 3D Gaussians through dense-camera interpolation rendering, presented in Sec. 4.4

---

An apple
DIT
Block
Cross Attention Layer
Self- Attention Layer
MLP Layer
Text
Timeslep
park
Figure 4: Left: Architecture of Traj-DiT. Right: Visualization of the predicted camera trajectory for different denoising timesteps.
4.2 Traj-DiT as Cinematographer
To model the trajectory distribution, we represent the camera trajectory as a set of camera parameters frr; tg. f., p. where r SO(3) and ~ IR3 are the rotation and translation of the camera poses, f R. is the focal lengths and p R² is the principle points. To ensure consistency and comparability across scenes, we normalize the trajectory for each scene in two steps: First, we convert all camera poses to be relative to the first one so that the first camera pose an identity matrix; Then, we re-scale the translation to make the distance from the first to the farthest camera to We adapt the architecture of the Diffusion Transformer (DiT) 73 to generate camera trajectories, as illustrated in Fig. (Left). The temporal order of real-world captures, akin to video sequences, necessitates learnable temporal embedding to differentiate between cameras of different frames. This embedding helps the model capture the sequential dependencies inherent in real-world data. Each DiT block includes a cross-attention layer to extract information from text embeddings encoded by the CLIP [24] text encoder. Additionally, the timestep t modulates the pre-layer normalization and post-layer output scalar, similar to the original DiT, allowing the model to learn temporal dynamics effectively. The model is trained to minimize the zo- prediction diffusion objective:
BC..) ||C Co ys t)11
where Co is the parameterized Traj-DiT model and is the noisy camera trajectory.
By leveraging the strengths of the DiT architecture, we aim to enhance the fidelity and coherence of the generated trajectories, instead of relying on pre-defined ones. We showcase two examples of predicted camera trajectories for different denoising steps Fig. (Right), demonstrating effectiveness of our model in generating smooth and accurate camera paths.
4.3 GM-LDM as Decorator
propose GM-LDM to model the image sequence distribution (C, y)) and generate immediate 3D Gaussians the joint 3D scene representation. The GM-LDM, fine-tuned from the Stable Diffusion model with a slightly modified architecture, leverages its image generation prior to enhance 3D scene generation. For efficiency, diffusion is applied to sparse-view subset of the camera trajectory, significantly reducing computational overhead. During training, sparse-view images are processed through the frozen latent encoder obtain multi-view latents 2 IR xxxxxxx where N < M. Noise is then added to these multi-view latents 2 to produce noisy latents Z,
2D-based Denoising. The noisy multi-view latents Z, fed into the latent denoising network Ze in parallel. For convenience, we modify the e-prediction of the original Stable Diffusion model to Po-prrecciion. The denoised multi-view latents are obtained as F) = Z,({Z,), C', y, t), where F represents additional multi-view features for enhanced 3D information. Sparse- -iew cameras C' are integrated into the network by combining the ray-maps (ox d, d) [ with the noisy latents, where and d denote the origin and direction of pixel-aligned rays, respectively. We replace self-attention blocks in the original 2D latent denoising network with cross-view self-attention blocks (3] to better capture multi view correlations. The denoised multi-view latents 2 are supervised using simple
5

---

at
D
c0
{ū. Us
Can8++
Z
Dg
E
2
Back-propagate
Figure 5: Left: Architecture of GM-LDM. The model is fine-tuned from 2D LDM with minor
modifications, performing rendering-ben based denoising for generating initial 3D Gaussians. Right: Pipeline of calculating SDS++ loss, which refines the 3D Gaussians with the original 2D LDM.
multi-view latent diffusion objective:
Caa Ex, | 2
(6)
Rendering-based Denoising. To generate 3D Gaussians for rendering-based denoising, the denoised multi-view latents 2 and additional features F are input into Gaussians decoder Dg. This decoder outputs Gaussian features {Te, Q. $,, Oti, c;} D3 (2, Fi), where Ta Q; 81, aN and c, represent the depth, rotation quaternion, scale, opacity, and spherical harmonics coefficients of 256 x 256 3D Gaussians for view i, respectively. The Gaussians decoder Da is initialized with the weights of the original latent decoder D, with re-initialized first and last convolutional layers to handle the additional features and specific Gaussian channels. The predicted depth is then converted into pixel-aligned Gaussian positions μ, o, + T,d,, The multi-view 3D Gaussians g = {14 I.s 8;, Ol.y C, J are concatenated jointly represent the 3D scene. During training, views are randomly sampled from the dinw camera trajectory to supervise the predicted 3D Gaussians in image space, ensuring consistent and accurate 3D scene representation:
L3 E.r.e [ [
where is the rendering function, -(:,-) -) is reconstruction loss penalizing the difference between images, and (x,c) c) E (X, C) is the ground truth of an image and camera pair from the dense views. We use combination of MSE loss and LPIPS 774) loss for the reconstruction loss É, similar the original reconstruction loss of the original auto encoder The total training loss is simply the sum of the above losses: Zza + L3a- This approach leverages the strengths of multi-view data and pixel-aligned Gaussian representations to enhance the fidelity and coherence of the generated 3D scenes. By fine-tuning from robust 2D LDM and using a sparse-view subset, we strike a balance between performance, efficiency, and generalizability. enabling the generation of coherent 3D scenes. As shown in Fig. 5 (Left), during inference, images are rendered with the input cameras C' and encoded by the latent encoder & to obtain Gaussian-driver denoised latents:
2g E(²"), where X R(G,C').
(8)
Inspired by Dual3D 75 GM-LDM inference can toggle between 2D-based and sendering-based denoising. The 2D-based denoising offers better generalization, aligning closely with the original Stable Diffusion, while rendering-basce denoising ensures superior 3D consistency due to its imme- diate joint 3D representation. The 3D Gaussians generated the final denoising step serve as the initial 3D scene for subsequent refinement.
Collecting and annotating real-world multi-view datasets is laborious, often resulting in limited diversity and quantity, which hinders generalization for open- open-world texts. To address this, we follow MVDream 3 and collaboratively train the GM-LDM using both multi-view and datasets to enhance generalization. We treat single-view images N and apply the same rendering process and training losses, which increases the diversity of training data, thereby improving the model's ability to generalize across diverse scenarios.
special case of multi-view images with

---

4.4 SDS++ Loss as Detailer
To enhance the details and visual quality the 3D Gaussians, we propose the SDS+ loss, leveraging the 2D diffusion prior for refinement. Our research on existing SDS- methods identifies three key points crucial for success, this includes: (1) Appropriate Target Distribution [26]: This ensures
that the rendered images align effectively with the textual conditions, avoiding over-smoothing and
over-saturation. (2) Adaptive Estimation of the Current Distribution [26 76 This provides a
counter optimization objective, pushing the rendered images away from the current distribution to
enhance details. (3) Latent-space and Image-space Objectives [72]: Combining these objectives
helps prevent noisy over-smoothing artifacts that may arise from using only one of them. During refining, we first render the 3D Gaussians 9 with randomly sampled camera e from the continuous interpolated camera trajectory to produce
R(G, c). This image is encoded into the latent space by (x) = 2. then disturbed with randomly sampled noise and timestep f to produce a noisy latent 2g. The 2D diffusion model Eg then predicts the denoised latent & from 2t. As illustrated in Fig. (Right), the proposed SDS+ loss can be formulated by:
image
LSDS++ Er,,C (()
Vāl
(Azll= 21B Au1|Er 2 .13)
VI at
where A and A are the weights for latent-space and image-space objectives, respectively, 2 is the predicted latent, and (2)) is the predicted image. The predicted latent can be derived by Eq. We use a compositional predictions as follows:
Cug Êuc + c.
(10)
Instead of setting Exc = c as in the standard SDS loss, we introduce a learnable source prediction é. for adaptive estimation of the current distribution:
EAc = ca (2e. f.t),
(11)
where c0(2,, 6) uses a learnable text embedding ÿ to efficiently estimate the current distribution. This approach leverages the original latent denoising network (76) and is trained by minimizing (2ce t) <ll along with the refining process. The target prediction employs classifier-free guidance for improved text alignment:
Eug Weig (co(2,, t) co(2e, o, t)) + co(2s, o,
(12)
where Weg is the classifier-free guidance scale. SDS++ loss integrates the above three key points, ensuring efficient and realistic refinement of 3D Gaussians.
5
Experiments
5.1
Implementation Details
We utilize the MVImgNet [19] for object-level and DL3DV 20 for scene-level real-world multi- view datasets. Text prompts for each scene are generated using the multi-modal large language model
IntemLMM-XCopposer [77]. To enhance generalization, we incorporate the 2D dataset LAION [78). For GM-LDM, we set the lengths of dense and sparse views to M 29 and N 8, respectively. The classifier-free guidance scale is 7.5 for 2D-based denoising and for rendering- based denoising to ensure 3D consistency. Following Dual3D 771, we balance 3D consistency and generalization by using 1/10 rendering- based denoising steps. Image and latent resolutions are set to 256 and 32, respectively. For SDS+ loss, the weights for latent-space and image-space losses are X 1 and >, = 0.01. Wefg is set to 7.5, with refining iterations set to 1000. Generating a scene takes approximately 5 minutes. Further implementation details are provided in Appendix A
5.2 Generation Results.
We show the generation results of camera trajectories and image sequences for various text prompts in Fig. 6 For object-level prompts, the generated camera trajectories typically circle and face the objects, aligning well with the distribution in MVImgNet. In contrast, scene-level prompts yield more diverse and complex camera trajectories, showcasing the effectiveness of our Traj-DiT model. Our method also generates realistic images across different types of prompts, demonstrating the effectiveness and generalization ability of GM-LDM and SDS++ loss. Additional generation results are available Appendix[D [ These results highlight the robustness of our approach in handling both object-level and scene-level prompts for 3D scene generation.
7

---

blue denim jacket
collection of fresh vegetables arranged in wicker basket
A decrepit amusement arcade on the boardwalk...
green paperback novel lying on a beige couch
An underground cave system with stalactites and stalagmites...
derelict factory with broken windows...
A bicycle leaning against wall
castle made of stone
A space observatory, its dome cracked and panels missing...
Figure 6: Generation results of Director3I) for both camera trajectories and image sequences.
Four ripe apples in basket
green wellington
An ornate fountain
crumbling fortress hill
blue and white with lid
lighthouse shore
A wooden bench in park and white lighthuuse cliff
Figure 7:
Qualitative comparison between Director3D and different baselines.
5.3 Qualitative Comparison.
We qualitatively compare our method with several baseline methods, as shown in Fig. 7 (1) GRM 38 feed-forward text-to-3D generation method for 3D Gaussians using multi-view images generated by 2D-based diffusion models. It supports only object-level 3D generation. Our comparisons using object descriptions show that GRM produces unrealistic 3D Gaussians limited objects, due its training on synthetic datasets. In contrast, our method generates high-quality 3D scenes with both objects and backgrounds. (2) GrussianDreemeer [TT] state-of-the-art SDS-based method for 3D Gaussians, integrating priors from both 2D and 3D diffusion models. While can generate objects with ground layers, tends to produce over-saturated textures. Our method, however, generates more realistic scenes with better handling of shadows, lighting, and material reflections. (3) DreamScene [17 uses Formation Pattern Sampling and strategic camera sampling for Gaussians. Since it is not open-sourcee source,, we use examples from its project page. Although it generates scene- wide consistent 3D scenes, the results are overly saturated and cartoonish. (4) LucidDreamer [16), based on Text2NeRF [14), uses 2D foundation models for 3D Gaussians. While it can generate photo-realistic textures, the multi-view consistency poor, with visible artifacts object edges due to inaccurate monocular depth estimation. It also struggles with excessive object generation from descriptive prompts due to reliance on single-view inpainting. These comparisons highlight the superior performance of the proposed Director3D for realistic 3D generation.

---

Table 1: Quantitative comparison of different object-centric models with text prompts in T3Bench.
Method
BRISQUE J
NIQE 4
CLIP-Score 1
DreamFusion
90.2
10.48
67.4
Magic3D [7]
92.8
11.20
72.3
LatentNerf 880
88.6
9.19
68.1
SJC 2
82.0
10.15
61.5
Fantasia3D
81
69.6
7.65
66.6
ProlificDDreamer 26
61.5
7.07
69.4
Ours w/o refining
37.1
6.41
80.0
Ours
32.3
4.35
85.5
Table 2: Quantitative comparison of scene-level models with 64 prompts
Method
NIQE 4
CLIP-Score f
Inference Time (min) +
GrussianDreamer
6.96
71.8
15
ZeroNVS 82)
9.84
67.2
90
LucidDreamer-LLFF [6
3.53
83.3
40
LacidDreamer--HccaBBagg
16
3.61
82.9
40
LucidDreamerr-BccoForr
[16
3.40
74.2
40
Ours
4.09
83.9
5.4 Quantitative Comparison.
We present a quantitative comparison between our framework and several baseline models in Tab. the
For this experiment, contains 100 prompts closely matching the descriptions in MVImgNet. The quantitative results evaluated using CLIP-Score [84), NIQE (85), and BRISQUE [86] metrics. For each 3D scene generated by different methods, we render a video and uniformly sample 36 frames to calculate the average score for each metric. For baselines without adaptive camera trajectories, videos are rendered by circling around the 3D representations at a fixed elevation. The BRISQUE and NIQE results indicate that our method significantly outperforms existing baseline models in terms of image quality. Additionally, the CLIP-Score shows our method's 's superior ability to align generated images with their textual descriptions, even without refining. These results underscore the robustness and effectiveness of our framework in generating high-quality, semantically aligned 3D scenes.
OhjjectwwithSSrrooee
set of T3Bench [3 which
We also conduct a quantitative comparison with three scene-level baselines for 32 object-centric prompts and 32 scene-level prompts in the Tab. 2 Our method achieves the highest CLIP-Score and the second-best NIQE Score with the shortest inference time. Without adaptively generated camera trajectories (Traj-DiT) and a high-performance multi-view diffusion model with immediate 3D representation (GM-LDM), ZeroNVS exhibits deteriorated visual quality from multiple viewpoints and requires time-consuming SDS optimization process from scratch for each scene. LucidDreamer attains the best NIQE Score; However, we observe that it is plagued by multi-view inconsistency, visible artifacts at edges, and excessive objects as shown in Fig. 7
5.5 User-specific Camera Trajectories
Director3D supports the utilization of both pre-generation and post-generation user-specific camera trajectories. For pre-generation user-specific camera trajectories, users are capable of employing user-specific camera trajectories instead the generated camera trajectories from Traj-DiT for 3D scene generation. For cost-generation user-specific camera trajectories, users can render novel views by providing novel cameras after generating the 3D scene. We develop an interactive demo for visualizing the generated camera trajectories and 3D Gaussians, which is also capable of rendering the 3D Gaussians with novel cameras, as shown in Fig. g

---

distance view
novel views
distance view
novel views
AEE
deep lake that mirrors the sky and cliffs around it...
mountain pass, where the wind howls through craggy
Figure 8: Screenshots of the interactive demo for visualizing generated camera trajectories and 3D Gaussians of Director3ID. The frames rendered with novel cameras.
5.6 Ablation Study
Director3D is formulated sequential process model the joint distribution of camera trajectories and image se- quences. The Traj-DiT component generates the base cam- era trajectory for each scene, while the GM-LDM provides the initial 3D scene. These ducing meaningful 3D scenes. Our ablation study primarily focuses the SDS+ loss to highlight its significance. As
full
A =0
A
models are crucial for pro-
shown in Fig. ? we conduct comprehensive experiments to analyze the impact of the SDS+ loss. First, we re- move the refining process entirely. Although the initial 3D Gaussians generated by GM-LDM match the input text, the visual quality is unsatisfactory, with missing details. This is expected due the limited diversity and quantity of the multi-view dataset used for training GM-LDM. Setting 6sr = degrades the SDS+ loss into SDS+ loss (2). The results show a significant decrease in visual quality and an over-smoothing issue, highlighting the importance adaptive estimation of the current distribution. Setting Wofe 1 turns the SDS+ loss into the LODS loss (76) with an additional image-space objective. This results in noisy details, as the conditional noise prediction alone does not provide clear optimization direction. By setting A 0 and A 0 respectively, observe that using only the latent-space objective leads to noisy details and artifacts, while using only the image-space objective results in missing details.
wlo refining
Welge=
delicious hamburger wooden table
Figure 9: Ablation of SDS++ loss
The full model, incorporating the proposed SDS++ loss, achieves the best visual quality with clear and realistic details. These findings underscore the importance of each component in the SDS++ loss and its role refining 3D Gaussians. Additional ablation studies, including qualitative and quantitative of more cases and those with randomly generated trajectories, are provided in Appendix
Conclusion
In this paper, we propose an open-world text-to-3D generation framework capable of generating real-world 3D scenes with adaptive camera trajectory, named Director3D. We first introduce Cinematographer (i.e., Traj-DiT) that can generate dense-view camera trajectories given texts. Then, a Decorator (i.e., GM-LDM) and a Detailer (i.e., SDS++ loss) are proposed for initial generation and further refining. respectively, with 3D Gaussians as the 3D scene representation. We demonstrate effectiveness our method with extensive experiments. We believe our work makes essential contributions to the text-to-3D generation community, especially in discovering the potential of leveraging real-world multi-view datasets for realistic 3D generation. Our future works include improving the generation scope, boosting model efficiency and quality, and leveraging more datasets.
10

---

7 Acknowledgements
This work was supported by National Science and Technology Major Project (No. 2022ZD01 18202), the National Science Fund for Distinguished Young Scholars (No.62025603) the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), and the Natural Science Foundation Fujian Province of China (No.2021J0100,, No.2022106001), This project is also funded in part by Shanghai Artificial Intelligence Laboratory.
References
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[2] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich, Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint urXiv:2212.00772 2022.
[3] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mydream: Multi-view diffusion for 3d generation. arXiv preprint arXi 16512, 2023.
[4] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[5] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1-15, 2022.
[6] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.044 144400, 2023.
[7] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, al. Dmv3d: Denoising multi-viww diffusion using 3d large reconstruction model. arXiv preprint arXiv:2311 1,00217, 2023.
[8] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (SIGGRAPII) 42(4), July 2023.
[9] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 16653, 2023.
[10] Zilong Chen, Feng Wang. Yikai Wang, and Huaping Liu. Text-to-3d using gaussian splatting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[11] Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from to 3d gaussians by hridging 2d and diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[12] Jiaxiang Tang. Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.0503 2024.
[13] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm; Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:2404. 19702, 2024.
[14] Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven 3d generation with neural radiance fields. arXiv preprint arXiv:2305. / 2023.
[15] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Mathhia NieBner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7909-7920, October 2023.
[16] Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:331113334, 2023.

---

[17] Haoran Li. Haolin Shi, Wenli Zhang, Wenjun Wu, Yong Liao, Lin Wang, Lik hang Lee, and Pengyuan Zhou. Dreamscene: 3d gaussian-basee text-to-3d scene generation via formation pattern sampling. arXiv preprint arXiv:2404.03575, 2024.
[18] Shijie Zhou, Zhiwen Fan, Dejia Xu, Haoran Chang, Pradyumna Chari, Suya Bharadwaj. Tejas You, Zhangyang Wang. and Achuta Kadambi. Dreamscene360: Unconstrained text-to-3d scene generation with panoramic gaussian splatting. arXiv preprint arXiv:2404.06903 2024.
[19] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong. Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9150-9161, 2023.
[20] Lu Ling, Yichen Sheng. Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D03dy-10k: A large-scale scene dataset for deep learning-basel baaeid 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22160-221666 2024.
[21] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13142 13153, 2023.
[22] Eric R Chan, Comnor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages (6133-1633,, 2022.
[23] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zera-shot text-guided object generation with dream fields. In Proceedings of the JEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 867-876, 2022.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning (ICML), pages 8748-8763.
PMLR, 2021.
[25] Haochen Wang, Xiaodan Du Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d difffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12619-12629, 2023.
[26] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. 16213, 2023.
[27] Oren Katzir, Or Patashnik, Daniel Cohen-Or, and Dani Lischinski. Noise-free score distillation. arXiv preprint arXiv:2310. 17590, 2023.
[28] ZinXin Zou, Weihao Cheng, Yan-Pei Cao, Shi-Sheng Huang, Ying Shan, and Song- Hai Zhang. Sparse3d: Distilling multiview-consistett diffusion for object reconstruction from sparse views. arXiv preprint arXiv:2308. /4078, 2023.
[29] Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-dd generation using hybrid score distillation sampling, arXiv preprini arXiv:231 1..223311.198.. 2023.
[30] Jinbo Wu, Xiaobo Gao, Xing Liu, Zhengyang Shen, Chen Zhao, Haocheng Feng, Jingtuo Liu, and Errui Ding. Hd-fusion: Detailed text-to-3d generation leveraging multiple noise estimation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 2202-3211, 2024.
[31] Zexiang Liu, Yangguang Li, Youtian Lin, Xin Yu, Sida Peng. Yan-Pei Cao. Xiaojuan Qi, Xiaoshui Huang. Ding Liang, and Wanli Ouyang. Unidream: Unifying diffusion priors for relightable text-to-3d 3d generation. arXiv preprint arXiv: 2023.
[32] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hat Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross domainn diffusion. arXiv preprint arxiv:2310.15008, 2023.
12

---

[33] Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synedreamer: Generating multiview-consisteet images from single-view image. arXiv preprint arXiv:2309.03455 2023.
[34] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint a.X17:22/2.08551, 2022.
[35] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463 2023.
[36] Zhicong Tang, Shuyang Gu, Chunyu Wang, Ting Zhang, Jianmin Bao, Dong Chen, and Baining Guo. Volumediifussio: Flexible text-to-3d generation with efficient volumetric encoder. arXiv preprint urXiv:2312.11459, 2023.
[37] Ryan Shue, Eric Ryan Chan, Ryan Po. Zachary Ankner. Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages
2023.
[38] Yinghao Xu, Zifan Shi, Wang Yifan, Sida Peng, Ceyuan Yang. Yujun Shen, and Wetzstein Gordon. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint arXiv: 2403. 14621, 2024.
[39] Jiabao Lei, Jiapeng Tang, and Kui Jia. Rgbd2: Generative scene synthesis via incremental view inpainting using rghd diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), June 2023.
[40] Rafail Fridman, Amit Abecasis. Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene generation. arXiv preprint urXiv:2302.01133 2023.
[41] Dana Cohen-Bal Bar, Elad Richardson, Gal Metzer, Raja Giryes, and Daniel Cohen-Ou Or. Set-the Set-the-scene: Global- local training for generating controllable nerf scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 2920-2929, October 2023.
[42] Jaidev Shriram, Alex Trevithick, Lingjic Liu, and Ravi Ramamoorthi. Realmdreamer: Text-driven 3d scene generation with inpainting and depth diffusion. arXiv preprint arXiv:2404.07199 2024.
[43] Jonathan Ho, Ajay Jain, and Picter Abbcel. Denoising diffusion probabilistic models. Advances In Neural Information Procensing Systems (NeurIPS), 33:6840-6851, 2020.
[44] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages (0684-10695, 2022.
[45] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large- scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[46] Rene Ranftl, Katrin Lasinger, David Hafner, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP:1-1, 08 2020.
[47] Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi- view image generation with correspondence-awart diffusion. arXiv preprint arxiv:2307. 000097 2023.
[48] Yikun Ma, Dandan Zhan, and Zhi Jin. Fastscene: Text-driven fast 3d indoor scene generation via panoramic gaussian splatting. arXiv preprint arXiv:2405.0577 2024.
[49] Peng Wang, Lingjic Liu, Yuan Liu, Christian Thcobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In 35th Conference on Neura! Information Processing Systems, pages 27 17171-27183. Curran Assoicates, Inc., 2021.
[50] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[51] Sara Fridovich- Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
13

---

[52] Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.12775,222 2023.
[53] Mulin Yu, Tao Lu, Linning Xu, Lihan Jiang, Yuanbo Xiangli, and Bo Dai. Gsdf: 3dgs meets sdf for improved rendering and reconstruction. arXiv preprint arXiv:2403. 16964, 2024.
[54] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. ACM Transuctions on Graphics (SIGGRAPHI). 2024.
[55] Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold-gs gs: Structured 3d gaussians for view-adaptive rendering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[56] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.
[57] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis, arXiv preprint arXiv:2312.168/1 2023.
[58] Zeyu Yang, Hongye Yang. Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. International Conference on Learning Representations (ICLR), 2024.
[59] Yuheng Jiang, Zhehao Shen, Penghao Wang, Zhuo Su, Yu Hong, Yingliang Zhang, Jingyi Yu, and Lan Xu. Hifi4g: High-fidelity human performance rendering via compact gaussian splatting, arXiv preprint urXiv:2312.03-66 2023.
[60] Jin Chuan Shi. Miao Wang, Hao- Bin Duan, and Shao-Hua Guan. Language embedded 3d gaussians for open- vey-vooabullry scene understanding. In Proceedings of the IEFF/CVP Conference Computer Vision and Pattern Recognition, pages 5333-5343, 2024.
[61] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaus- sian splatting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2005102000000 20060, 2024.
[62] Yansong Qu, Shaohui Dai, Xinyang Li, Jianghang Lin, Liujuan Cao, Shengehuan Zhang, and Rongrong Ji. Goi: Find 3d gaussians of interest with an optimizable open-vocabulary semantic-space hyperplane. In 2024.
Proceedings of the 32nd ACM International Conference on Multimedia, pages
[63] Jonas Kulhanek, Songyou Peng, Zuzana Kukelova, Mare Pollefeys, and Torsten Sattler. Wildgaussians: 3d gaussian splatting in the wild. arXiv preprint arXiv:2407.08447, 2024.
[64] Yuze Wang, Junyi Wang, and Yue Qi. We-gs: An wi-the-wild efficient 3d gaussian representation for unconstrained photo collections. arXiv preprint 20Xi::2466002007, 2024.
[65] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. arXiv preprint 24v:24003 14627, 2024.
[66] David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In The IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[67] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In The IRFE/CVP Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[68] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-vies generation and large reconstruction model. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[69] OpenAl. Sora: Creating video from text. https://openai.coo/svr.. 2024.
[70] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text to---t- image diffusion models with deep language understanding. Advances In Neural Information Processing Systems (NeurIPS), 01133677936999 2022.
14

---

[71] Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In 2nd International Conference on Learning Representations (ICLR) 2014, Banff. AB, Canada, 2014.
[72] Junzhe Zhu and Peiye Zhuang. Hifa: fideliiyyityy text-to-3d generation with advanced diffusion guidance. arXiv preprint arXiv:2305. 1876. 2023.
[73] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 4195 4205, October 2023.
[74] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR). pages 586-595, 2018.
[75] Xinyang Li, Zhangyu Lai, Linning Xu, Jianfei Guo, Liujuan Cao, Shengchuan Zhang, Bo Dai, and Rongrong Ji. Dual3d: Efficient and consistent text-to-3d generation with dual-mode multi-view latent diffusion, arXiv preprint urXiv:2405.09877 098744 2024.
[76] Xiaofeng Yang, Yiwen Chen, Cheng Chen, Chi Zhang. Yi Xu, Xulei Yang. Fayao Liu, and Guosheng Lin. Learn optimize denoising scores for 3d generation. arXiv preprint arXiv:2312. 04820, 2023.
[77] Pan Zhang, Xiaoyi Dong, Bin Wang. Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui Hc, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internim- xxcompse.r..r A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.1511 121 2023.
[78] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis. Mitchell Wortsman, et al. Laion-Sb: An open acrge-scale dataset for training next generation image-text models. Advances In Neural Information Processing Systems (NeurlPS), 35:25278 -25294, 2022.
[79] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the TEEE/CVF Confenence on Computer Vision and Pattern Recognition (CVPR), pages 300-309, 2023.
[80] Gal Metzer, Elad Richardson, Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape- guided generation of 3d shapes and textures. arXiv preprint arXiv:2211.0766 2022.
[81] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
[82] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Hermman, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, and Jiajun Wu. ZeroNVS: Zero-shot 360-degree view synthesis from single real image. CVPR, 2024, 2023.
[83] Yuze He, Yushi Bai, Matthieu Lin, Wang Zhao, Yubin Hu, Jenny Sheng, Ran Yi, Juanzi Li, and Yong-Jin Liu. h'ennhh: Benchmarking current progress in text-to-3d generation. arXiv preprint 10.09,33, 2023.
[84] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference- free evaluation metric for image captioning. In Conference Empirical Methods in Natural Language Processing (EMNLP). 2021.
[85] Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):20-221 212, 2013.
[86] Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assessment in the spatial domain. IEFF Transactions on Image Processing, 21(14))6695-4708, 2012.
[87] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
[88] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations (ICLR) 2021, Virtual Event, Austria, 2021.
[89] Johannes Lutz Schönberger and Jan Miihael Frahm. Structure- motion revisited, In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
15

---

[90] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In International Conference on Computer Vision (ICCV), 2021.
[91] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Transactions on Graphics (SIGGRAPH), 2018.
[92] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotater 3d reconstructions of indoor scenes. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattrnn Recognition (CVPR), 2017.
16

---

A Implemental Details
Datasets. For real-world multi-view datasets, we leverage MVImgNet [9] and DL3DVI0K 20 datasets for object-level and scene-level captures, respectively. The MVImgNet dataset is large-scale scale object-level dataset, comprising 219, 118,188 videos across 238 classe:" The B.3DV1OK, dataset consists of 10K scene-level videos with various indoor, outdoor, urban, and suburban environment
Traj-DiT setup. Our Traj-DiT is trained with the Adam optimizer [87], a constant learning rate of 5e-5 and (31. (2) (0.9, 0.95). The batch size is set to 1 128. The number of the DiT Blocks 8 and the hidden size is 512. Training takes about days with 1 NVIDIA Tesla A 100 GPUs for 50K iterations. We use 1000 steps during training and reduce it to 100 steps with DDIM 88 inference.
during
GM-LDM setup. Our GM-LDM is trained with the Adam optimizer, a constant learning rate of le and (31, 32) = (0.9. 0.95). The batch size is also set to 128. Training takes about a week with 16 NVIDIA Tesla A100 GPUs for 150K iterations. We use Stable Diffusion v2. 1 as our initial model. The noise schedule and other hyper-parameters are consistent with the Traj-DiT.
SDS+ loss setup. For refining, the learning rates of the Gaussian parameters fs, q. 8, Ot and e are set be 0.0001, 0.01, 0.001, 0.01, and 0.01, respectively. The learning rate of the learnable text embedding ú is set to be 0.001. The timestep is annealing from 0. to 0.027 with square root schedule, where T is the total denoising steps. We use Stable Diffusion v2.1 as our refining model as well. The rendering resolution is set to be 512 during refinement.
B Discussions
Ways to Model p(X'' C). There are different approaches to modeling the joint distribution of image sequences and camera trajectories for real-world captures. We choose to independently model the distribution of camera trajectories and the conditional distribution of image sequences based on the camera trajectories. The image sequence is directly rendered with a joint 3D representation by the proposed GM-LDM, ensuring 3D consistency. Another alternative is to first directly model the image sequences and then infer the camera trajectories from the image sequences. Concurrent work, GS-LRM 13, adopts this approach. Specifically, it uses pose-free image sequences generated by SORA, then recovers camera information through COLMAP [89, and finally performs sparse-view 3D reconstruction. However, this approach heavily relies the 3D consistency of the pose-free image sequences and the accurate estimation of the camera parameters. One can also choose to directly model the joint distribution of camera trajectories and image sequences. However, this requires further exploration of network architectures and camera-differentiabl 3DGS operator. We will explore the potential of this approach in future work.
Why Director3D Needs a Detailer? The core reason remains the limitations of multi-view datasets in terms of diversity and quantity, leading biased and sparse distributions. Although MVImgNet has a large quantity of scenes, it only consists of 238 categories, with the majority being simple objects staged in indoor scenes. While DL3DV-IOK exhibits good diversity, it only consists of 10,000 scenes in total. Therefore, even under the 2D data and model priors, it is still challenging for GM-LDM to meet the quality requirements for directly generating realistic 3D scenes with details. This limitation can be alleviated by introducing more diverse and larger multi-view datasets.
Linitations. Although the proposed Director3D possesses the capability generate real-world 3D scenes with adaptive camera trajectories, it still has some limitations. First, the supported views of our GM-LDM are currently limited, which restricts the range of perspectives in the generated 3D scenes. Further, because the GM-LDM is currently trained with only two real-world datasets, the reliance on additional refining process for open- world generalization limits the efficiency of our framework, which may be alleviated by introducing a wider variety of datasets [0] 92). Similar to text-to-ine to-image diffusion models, the success rate of our model decreases when generating with complicated and compositional prompts, objects with exact numbers, and articulated objects, as shown in Fig. 10
httpp://ggicubb.com//ppLLLL http:///PP-LL---CCCKSSS///////EA under the MVImg Terms of Use http:://gthhb..co//bl/ooo10v//// under the DI3 Terms Use
base
17

---

Stone statue a knight wearing a diamond crown and holding golden sword.
Six oranges in a wooden basket.
Tyrannosaurus Rex skeleton in an exhibition hall.
Figure 10: Failure cases.
Table 3: Quantitative ablation study of SDS++ loss
Setting
NIQE +
CLIP-Score t
full
4.10
83.7
w/o refining
5.95
79.3
A 4.12
83.1
A = 0
7.18
81.3
Wefg
4.26
79.6
4.27
80.4
Broader Impacts. This paper presents framework whose goal is to advance the field of text-to-3D generation for efficiently generating realistic 3D scene assets. Since is a 3D generative framework, it has the potential to create harmful 3D content if the user provides harmful text prompts as inputs. This harmful behavior can be avoided by integrating harmful content detection models used 2D) generative models.
C Diversity and Fine-grained Control
We showcase the diverse generation results with the same prompts in Fig. II The results show that Director3D is able to generate diverse camera trajectories and 3D scenes even with single prompt.
We showcase the body generation results with fine-grained control (i.e., gender and clothing) in Fig. 12 Note that there are no categories for human body in the multi-view datasets used for training the GM-LDM. The results show that Director3D is able finely control the 3D scenes with text modifications.
D More Generation Results
We further provide 40 cases of multi-view image results in Fig. generated from wide-range text prompts to demonstrate the visual quality and generalization ability our method.
E More Ablation Studies
We provide qualitative ablation study with more cases in Fig. 14 The corresponding quantitative ablation study is shown in the Tab, 3 Although the ablation with A, 0 incurs a slight deterioration in metrics, the qualitative results noisy in terms of visual quality. These results can further demonstrate the effectiveness SDS+ loss.
Additionally. we showcase the generation results with randomly generated camera trajectories for some object-level prompts in Fig The results are generated by the GM-LDM without refining. can be observed that without scene-specific camera trajectories, the generated image sequences are unsatisfying, with weird perspectives or limited camera range. This ablation study further demonstrate the importance of scene-specific camera trajectories for our Director3D.
18

---

A bustling train station in the early morning...
rusty old car
Figure 11: Generation results with diversity.
A man wearing a graphic T-shirt and cargo shorts
A woman wearing a floral blouse and denim shorts
A woman wearing black blazer and a pencil skirt
A man wearing a checkered shirt and tan chinos
A woman wearing leather jacket and black skinny jeans
Figure 12: Generation results with fine -na-ind control.
19

---

A castle castle-shaped sandcastle
A silver wristwatch ticking on glass bedside table
small Christmas Legos
tree made of
luxurious underwater hotel with transparent walls offering...
blue diamond on white pedestal
mug filled with steaming coffee
A serene temple complex the mountains, with monks...
wooden chest with golden
A small, rustic cabin sits alone in peaceful, snow-covered forest
A smoldering camplire under a clear starry night, embers...
A vibrant coral reef leeming with marine life, divers exploring...
A pink pigay bank on shelf
rooftop terrace garden at night, with potted plants, string lights...
garden sunrise, with raked sand patierss...
sports chrome finishes parked by...
expansive field, and blackened, with cracks...
sparkling
quiel monastery hidden within misty forest, ancient stone...
A modern station with curved walls, flat escalator...
rocky hilltop with where walls and columns stand...
A lone, ancient tree stands tall in the middle quiet field
A long abandoned castle perched cliff,
A modern airport terminal at dawn, glass walls reflecting...
A teddy bear with a red bow
donut with pink icing
high-end bakery with rectangular display...
high-tech greenhouse with rectangular planting...
large swimming pool with rectangular lanes, circular..
A delicate china teacup resting loce doily
A dense marshland under full moon, where fog clings to...
A vintage typewriter on desk
frozen lake beneath the northern lights, the lee cracked...
cold research station in Antarctica, surrounded by...
A compact urban coffee shop with square
corporate lobby with a large rectangular recoption...
A sunken shipwreck off a coral coast, with colorful
stome water well with wooden
A brown leather suitcase on train platform
magic dagger, mistery, ancient, photorealistic, HD
steaming mug of hot chocolate with whipped cream
Figure 13: More multi-view image results.
20

---

full
A 0 A 0
Welg=1
refining
vintage clock hanging on brick wall
white seashell on sandy beach
A cozy cabin with smoke rising from the chimney...
A crumbling fortress hill, its walls overtaken by nature...
Figure 14: Additional qualitative ablation study of SDS++ loss for more cases.
Ast randon maj). Wel retining
A blue and white china cup saucer
vet mundomm mnj inforetiing
A blue denim jacket draped wooden chair
Auffy white pillow resting red velvet couch
Figure 15: Generation results with randomly generated camera trajectories.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim clearly in Abstract and Sec. Vec.Taber about contributions and scope made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: See Sec. 5.1 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
All datasets we use are public data, and our code will be open source in the future.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssicnpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Sec. 5.1 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Since most of the experiments are visually qualitative evaluations, they cannot be converted into reports with quantitative error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////.eerr...cco//////////a
Answer: [Yes]
Justification: We have read the NeurlPS Code of Ethics in detail to make sure that our study meets the standards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
See Appendix
B
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing assets used in the paper properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: All urls are anonymized.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "FvMdKDE3x9", "reviewer": "Reviewer_YUdV", "summary": "This paper proposes a framework for simultaneous text-to-3D scene and camera trajectory generation. The authors propose a 3-stage pipeline to (1) generate a dense camera trajectory from input text, (2) use multi-view latent diffusion from a sparse subset of the generated trajectory to generate the 3D scene representation (Gaussian splats), and (3) refine the Gaussian splats with a modified SDS loss.", "questions": "Please see the weakness sections.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles a practical problem of generating a 3D scene representation while also synthesizing a camera trajectory from text. It has implications of potential further applications for video/movie synthesis using explicit 3D representations.\n+ The paper is well-written, the presentation is clear, and the method description is easy to follow and understand.", "weaknesses": "- While this paper tackles a new problem, I have concerns with the problem statement. First, what is a \"real-world\" camera trajectory? Virtually any kind of camera motion could be created in the real world (be it hand-held shakiness or really smooth orbital trajectories that could be achieved via physical equipment). It seems that the camera motions that Traj-DiT could synthesize are mostly orbital (object-centric) -- why would this be considered as a \"real-world\" trajectory?\n- I don't quite get why 3D scene generation and camera trajectory generation should be a coupled problem. NeRFs / Gaussian splats with good quality are not necessarily created via \"camera trajectories\", but rather via a broad range of covered viewpoints.\n- Using a diffusion model to synthesize trajectories, the number of frames would be fixed. How does one determine such trajectory length? How can one vary the length?\n- It is unclear why Cinematographer generates a trajectory with dense frames while only a sparse subset is ever used subsequently.\n- To evaluate the quality of the synthesized camera trajectory, I believe the method should also be evaluated and compared against with a video generation quality metric (e.g. FVD). The 3D scene representation could be a well-trained NeRF / Gaussian splat, and rendered videos under different trajectories could be quantified and compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your appreciation of the **practical problem**, the **potential applications**, the **clear presentation**, the **easy-to-follow method**, and the **well-written paper** of Director3D. We address your questions point-by-point below.\n\n---\n\n***Q1: The meaning of “real-world” camera trajectories.***\n\nThe reason we emphasize \"real-world\" is that: Due to the collection costs, most large-scale real-world video datasets are primarily captured by handheld cameras rather than professional track cameras. Compared with the user-defined and precise camera trajectories of synthetic datasets (e.g., Objaverse and ShapeNet), the “real-world” camera trajectories  (e.g., MVImgNet and DL3DV-10K) are invariably unstable, noisy, scene-specific and hard to manually define. Director3D is proposed to handle such \"real-world\" camera trajectories. Our method can be well-generalized to different scenes by modeling the trajectory distributions in accordance with the datasets. Since we employ 180-degree captured MVImgNet for object-centric scenes, Traj-DiT accordingly generates 180-degree orbital camera trajectories for object-centric prompts.\n\n---\n\n***Q2: Why modeling \"camera trajectories\" rather than \"covered viewpoints\" for 3D scene generation?***\n\nDirector3D is aimed at employing real-world captured datasets for superior 3D scene generation.\nTherefore, the characteristics of real-world captured datasets are the principal considerations in our proposed solution.\nThese datasets typically consist of handheld captured videos with temporal order, leading us to model the camera trajectories rather than unordered viewpoints.\nThe temporal characteristic greatly eases the modeling problem for Traj-DiT.\nIt also assists in filtering appropriately spaced frames and might help further extending Director3D for 4D dynamic scene generation in the future.\n\n---\n\n***Q3: The number of frames for camera trajectory.***\n\nSetting the frame number requires balancing between the multi-view consistency and the scene scope within the limited model capacity of GM-LDM. We empirically set this by observing the view overlaps between frames.\n\nCurrently, our Traj-DiT can only generate camera trajectories of a fixed frame number (i.e., 29).\nThis issue might potentially be resolved through similar techniques for images or texts (e.g., linear interpolation of temporal embedding) to apply Transformers for variable token numbers.\nWe consider this one of our most significant subsequent improvements.\n\n---\n\n***Q4: Why does Traj-DiT generate dense frames but only a sparse subset is used for GM-LDM?***\n\nWe generate dense frames to ensure that the generated trajectories are smooth for later refining and video rendering.\nFor GM-LDM, we use only a sparse subset to reduce GPU memory cost for training a rendering-based multi-view diffusion model. Due to the overlaps between nearby frames, the sparse subset serves as key frames, limiting the number of Gaussians per scene. The other frames can be effectively rendered using 3DGS rendering with specific camera parameters.\n\n---\n\n ***Q5: Can the quality of the synthesized camera trajectory be evaluated with a video generation quality metric (e.g. FVD) by rendering videos under different trajectories for a specific scene?***\n\nYour advice is very insightful. The video generation process of Director3D is designed to generate trajectories first and then generate scenes, rather than generating trajectories based on scenes.\nTherefore, generating different trajectories for a specific scene is temporarily beyond the scope of the method's functionalities.\n\nThe FVD metric you specifically mentioned is a potential evaluation metric to evaluate the video and trajectory quality of Director3D. However, calculating FVD requires both the generated video set and the GT video set. The latter is currently unavailable for the open-world setting of our evaluation. Therefore, we temporarily choose to evaluate the quality with no-reference image quality assessment and CLIP similarity for video frames.\n\nWe do understand your concern and exploring more comprehensive metrics and benchmarks for evaluating trajectory and video generation is listed as one of our top subsequent goals.\n\nWe further offer quantitative and qualitative comparisons with several scene-level baseline methods (e.g., LucidDreamer and ZeroNVS). Please refer to CQ1 in the \"Global Rebuttal\".\n\nWe also provide results for employing user-specific cameras for 3D scene generation. Please refer to CQ2 in the \"Global Rebuttal\".\n\n---\n\nPlease kindly let us know if our responses have addressed your concerns. We are also happy to answer any of your remaining concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PgDKzzw8M1", "reviewer": "Reviewer_Lv1o", "summary": "This paper presents Director3D, a novel text-to-3D generation framework designed to generate both real-world 3D scenes and adaptive camera trajectories. Specifically, the authors propose the Traj-DiT to generate adaptive camera trajectories, which treats camera parameters as temporal tokens and performs conditional denoising using a transformer model. The authors propose the GM-LDM and SDS++ Loss to generate robust 3D scenes by leveraging the 2D diffusion prior. Extensive experiments demonstrate the effectiveness of Director3D.", "questions": "No additional questions.", "limitations": "1. The baseline methods like Fantasia3D and DreamFusion are mainly for object-level generation, not focusing on scene generation.\n2. The result visualization can be improved, e.g., the imges are quite small for the reviewer to check the visual quality.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper is written clearly and easy to read. \n2. The idea of treating camera parameters as temporal tokens for denoising generation is novel and effective. \n3. The GM-LDM and SDS++ loss functions achieve a fairly high level of realistic scene synthesis. The generated scenarios conform to the textual input as well as being reasonably realistic and consistent. \n4. The proposed Director3D achieves impressive results on both quantitative and qualitative results.", "weaknesses": "1.\tIt would be more convincing to include more text-to-3D scene generation methods in the Quantitative Comparison in the Qualitative Comparison.\n2.\tFor ablation experiments of SDS++ Loss, please use more detailed evaluation metrics and experimental results to illustrate the effects.\n3.\tFurther implementation of conditionally controllable camera view generation would be helpful for the application of this technology. \n4.  Some important work in this area such as 3D-SceneDreamer should be discussed if they are not suitable for the experimental comparison.\n5. The visual quality of the results seems acceptable compared to existing methods like 3D-scene-dreamer, the limitation of the method, including not allowing a large range of the camera movement, should be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your appreciation of the **clearly written paper**, the **novel and effective idea**, the **realistic and consistent scene synthesis**, and the **impressive results** of Director3D. We address your questions point-by-point below.\n\n---\n\n***Q1: More scene-level baselines.***\n\nPlease refer to CQ1 in the \"Global Rebuttal\".\n\n---\n\n***Q2: More evaluation metrics and experimental results for SDS++ loss.***\n\nThanks for you helpful advice. We further provide:\n\n1. a quantitative ablation study as shown in the following table.\n\n```\nTable 2. Quantitative ablation study of SDS++ loss\n--------------------------------------------------\n                  NIQE Score↓   CLIP-Similarity↑  \n--------------------------------------------------\n  Ours (Full)     4.10          0.837             \n  λ_x=0           4.12          0.831             \n  λ_z=0           7.18          0.813             \n  ω_cfg=1         4.26          0.796             \n  ε_src=ε         4.27          0.804             \n  w/o refining    5.95          0.793             \n--------------------------------------------------\n```\n\n2. a qualitative ablation study with more cases in Fig. 5 of the \"Global Rebuttal PDF\".\n\nAlthough the ablation with λ_x=0 incurs a slight deterioration in metrics, the qualitative results are noisy in terms of visual quality.\nThese results can further demonstrate the effectiveness of SDS++ loss.\n\nThanks for you kind advice and we will add these results into our revision.\n\n---\n\n***Q3: Conditionally controllable camera view generation.***\n\nPlease refer to CQ2 in the \"Global Rebuttal\".\n\n---\n\n***Q4: Comparison with 3D-SceneDreamer.***\n\nThanks for your kind reminder. 3D-SceneDreamer is an inpainting-based Text-to-3D scene generation method. \nCompared to 3D-SceneDreamer, Director3D:\n\n1. Can adaptively generate camera trajectories with Traj-DiT, while 3D-SceneDreamer employs user-defined trajectories.\n2. Can directly generate initial 3D scene by the predicted pixel-aligned 3DGS with our GM-LDM, while 3D-SceneDreamer needs to continuously optimize a triplane-base NeRF.\n\nWe also observed that inpainting-based Text-to-3D scene generation methods (e.g., LucidDreamer) support a larger scene scope but the multi-view consistency of objects with complicated geometry is worse than that of Director3D.\nThanks for your kind reminder and we will add these discussions in our next revision.\nAlso, scaling up the supported scene scope is listed as one of our top goals for subsequent improvements.\n\n---\n\n***Q5: Some baselines are mainly for object-level generation, not focusing on scene generation.***\n\nThe prompts utilized for quantitative comparison are provided by the “Single-Object-with-Surroundings Set” of T3Bench, which are primarily object-centric. For a more comprehensive evaluation, we further offer quantitative and qualitative comparisons with several scene-level baseline methods (e.g., LucidDreamer and ZeroNVS). Please refer to CQ1 in the \"Global Rebuttal\".\n\n---\n\n***Q6: The result visualization can be improved.***\n\nThanks for your advice. We rescale the images in our original manuscript to provide more examples demonstrating our open-world generation ability. Additionally, we present high-resolution videos in the anonymous video gallery link for better checking the visual quality. We will make subsequent improvements on this.\n\n---\n\nPlease kindly let us know if our responses have addressed your concerns. We are also happy to answer any of your remaining concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cH0FWhbnEx", "reviewer": "Reviewer_sdLT", "summary": "The paper proposes a scene-generation method from text input. The framework utilizes three models that first generate a trajectory, then produce 3D Gaussians, and finally refine through SDS loss.", "questions": "Please refer to the weakness.", "limitations": "Please refer to the weakness. This work shows brilliant visual results. I hope the authors clarify my questions above to help me better understand how the results are obtained.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The design of using a trajectory generator and 3DGS diffusion is novel and impressive. \n- The proposed method outperforms existing object-level methods as shown in experiments.", "weaknesses": "- The paper's evaluation is limited. The methods compared are object-level generation frameworks. The experiments can be improved by comparing them against scene-level generation methods, e.g. LucidDreamer[16] and ZeroNVS[1].\n- I am confused about how SDS++ loss is presented. The paragraph following Eq. (9) contains numerous undefined variables, making it hard to understand how exactly SDS++ loss is formulated. Can I say it is the loss presented in [67] but integrated with learnable text embedding? What's the difference against the loss proposed in VSD[26] and how does it compare?\n- The proposed trajectory generator is novel. How important is generating the trajectory? Can we just assign some trajectories by retrieving the dataset? It seems that most trajectories in MVImgNet and DL3DV are very similar. How diverse are the generations? I see the results in Fig. 12. But aren't the differences coming from different sampling results of a text-based generator? Since the cameras are already normalized, the orientation of the first frame should not be restricted by the camera trajectory. It seems that the model is overfitting to the orientations in MVImgNet (flying around something on the table). Are the \"randomly generated camera trajectories\" coming from other objects? Is it because GM-LDM only works with one trajectory per scene? Does the model support working with multiple trajectories for the same scene?\n- Is the proposed GM-LDM only training the 3DGS scene on observed views? How do the authors avoid overfitting in the rendering-based denoising? Is the SDS++ loss only employed on interpolated camera trajectory?\n- Is the presented video only showing the views presented in the trajectory or they are novel view synthesis results? How do we know the generated 3D scene is not overfitting to these views? Sparse-view 3D reconstruction suffers from over-fitting issues a lot and I'd like to hear the authors' thoughts on this matter.\n\n[1] Sargent K, Li Z, Shah T, et al. Zeronvs: Zero-shot 360-degree view synthesis from a single real image. CVPR 2024, arxiv 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your appreciation of the **novel and impressive design** and the **brilliant visual results** of Director3D. We address your questions point-by-point below.\n\n---\n\n***Q1: More scene-level baselines.***\n\nThanks for your helpful advice. For a more comprehensive evaluation, we further provide quantitative and qualitative comparisons with several scene-level baseline methods (e.g., LucidDreamer and ZeroNVS). Please refer to CQ1 in the \"Global Rebuttal\".\n\n---\n\n***Q2: About SDS++ loss.***\n\n**Meaning of variables.** Thanks for pointing this out. Please kindly refer to Sec. 3 in the original manuscript for the explanations of each variable and we will add a reference in the method to guide the readers for a better understanding of our SDS++ loss.\n\n**Formulation.** The formulation of SDS++ loss intended to combine the advantages observed in existing SDS losses (Please refer to lines 208–214 in the original manuscript): \n\n1. Eq. 9 - Both Latent-space and Image-space Objectives - HIFA.\n2. Eq. 10 & Eq. 11 - Adaptive Estimation of the Current Distribution - VSD & LODS.\n3. Eq. 12 - Appropriate Target Distribution - DreamFusion & VSD.\n\n**Comparison with other SDS losses.**\nPlease kindly refer to Sec. 5.5 in the original manuscript for the connections between the ablated versions of SDS++ loss and other works. Specifically, in contrast to the SDS+ loss proposed in HIFA, SDS++ loss encompasses an adaptive estimation of the current distribution with learnable text embedding. In comparison to the VSD loss proposed in ProlificDreamer, SDS++ loss incorporates an image-space objective and substitutes the complicated LORA by using learnable text embedding for the estimation of the current distribution.\n\n---\n\n***Q3: About camera trajectories.***\n\n**The importance of camera trajectory generation.** \nSince the videos are naturally captured with cameras within the 3D scene, camera trajectory generation is of great significance for modeling the distribution of real-world captured videos with 3D generative models.\nRetrieval-based assignment can be employed in simple cases. \nHowever, when scaling up the trajectory and scene complicities, it will be difficult for retrieval-based assignment to find scene-specific trajectories for different scenes.\n\n**Similar trajectories for object-centric prompts.**\nSince we use 180-degree captured MVImgNet for object-centric scenes, Traj-DiT follows to generate 180-degree orbital camera trajectories for object-centric prompts. \nPlease refer to Fig. 9 in the original manuscript for the diverse trajectory generation of scene-level prompts.\n\n**The orientation of the first frame.**\nWe would like to clarify that this is not an overfitting problem. Since the camera poses are normalized according to the first frame, the orientations of the first frame for different scenes are always fixed after the normalization (lines 140~142). \n\n**Randomly generated camera trajectories.**\nThe randomly generated camera trajectories are generated by performing unconditional denoising inference of Traj-DiT. The text embedding is set to be `NULL` for unconditional denoising.\n\n**Multiple trajectories for the same scene.**\nAlthough the 3D scene generation (GM-LDM & SDS++) is performed in one trajectory, the generated 3D scenes can be rendered with multiple novel trajectories. Please refer to CQ2 in the \"Global Rebuttal\".\n\n---\n\n***Q4: About over-fitting problem.***\n\n**Is GM-LDM only trained on observed views?** No. We use 8 views for denoising but 29 views are used for supervising GM-LDM.\n\n**How to avoid over-fitting in the rendering-based denoising?** For GM-LDM, the camera parameters used for supervision are invisible to the model, and the images for calculating losses are obtained through 3DGS rendering. Therefore, the immediate 3D Gaussians of GM-LDM need to represent the 3D scene rather than over-fit individual images from different viewpoints.\n\n**Is the SDS++ loss only employed on interpolated camera trajectory?** Yes. We also tried slightly disturbing the interpolated cameras but observed minor improvements.\n\n---\n\n***Q5: Novel view synthesis results.***\n\nThe presented video are rendered with uniformly interpolated cameras within the generated trajectories. Please refer to CQ2 in the \"Global Rebuttal\" for more novel view synthesis results, which can further demonstrate that the generated 3D scenes are not overfitting to training views.\n\n---\n\n***Q6: About over-fitting issues of sparse-view 3D reconstruction.***\n\nFrom our perspective, the over-fitting issues of sparse-view 3D reconstruction are primarily caused by the loss of 3D information with too few input views. \nTherefore, possible solution to overcome overfitting issues is to introduce 3D prior knowledge such as **novel view prior** and **geometric prior**. \nFor example, using sparse-view reconstruction models trained with novel view prior knowledge for 3D scene initialization can help overcome overfitting issues.\nUsing depth or normal estimation with geometric prior for alternative supervision can also enhance the 3D information for sparse-view 3D reconstruction.\n\n---\n\nPlease kindly let us know if our responses have addressed your concerns. We are also happy to answer any of your remaining concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BWtoSgYelu", "reviewer": "Reviewer_yFX5", "summary": "This paper presents Director3D, a text-to-3D generation framework that creates realistic 3D scenes with adaptive camera trajectories. It includes a Cinematographer (Traj-DiT) for generating camera trajectories, a Decorator (GM-LDM) for initial scene generation, and a Detailer (SDS++ loss) for refinement. Using 3D Gaussians for scene representation, extensive experiments demonstrate its effectiveness.", "questions": "1. Is the number of Gaussians equal to the number of pixels in each view? Why set it that way? It seems that 256*256 Gaussians might not be sufficient for a scene-level representation. What is the total number of Gaussians used?\n\n2. Can this pipeline achieve user-specified camera trajectories?\n\n3. Is it possible to generate a complete 3D scene and then reconstruct it? If the camera trajectory encompasses the entire scene, all 3D information is captured. How does the novel view inference ability perform once the three stages are completed? If only can generate denoised views, the contributions will be weakened.", "limitations": "1. What about failure cases? In what situations does the method fail to generate a consistent scene?  I find it hard to believe that training on only two not large datasets can achieve such good results. The performance even surpasses object-level methods trained on Objaverse, which doesn't make sense. \n\n2. I want to confirm how the test set is divided. For each class, is a portion of the videos divided as the test set, or are a portion of the classes divided as the test categories? And I want to confirm again that none of the prompts appeared in the training set?", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The results are good with high spatial consistency.\n\nThe task is interesting. It is a good idea to generate scene-level 3D GS directly.\n\nThe writing is clear.", "weaknesses": "1. The quantitative comparison with object-level methods seems unfair. The given prompts include sence information, which efftets clip score. In quantitative comparisons, the authors should at least compare with scene-level methods like LucidDreamer or others.\n\n2. Why didn't the authors compare their method with camera control video generation methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your appreciation of the **high spatial consistency**, the **interesting task**, the **good idea**, and the **clear writing** of Director3D. We address your questions point-by-point below.\n\n---\n\n***Q1: The fairness of the quantitative comparison and more scene-level baselines.***\n\nThanks for your questions. \nThe prompts used for our quantitative comparison are provided by the \"Single-Object-with-Surroundings Set\" of T3Bench. \nAll methods share the same inputs (i.e., only text prompts) and the metrics are calculated without any cherry-picking.\nWe would like to clarify that SDS-based methods can also generate with scene information. Please see the examples in the project pages of DreamFusion and ProlificDreamer.\nThe camera trajectory generation and initial 3D scene generation trained with real-world datasets help our SDS++ outperform other baselines with pre-defined trajectory and random initialization.\n\nFor a more comprehensive evaluation, we further provide quantitative and qualitative comparisons with several scene-level baseline methods (e.g., LucidDreamer and ZeroNVS). Please refer to CQ1 in the \"Global Rebuttal\".\n\n--- \n\n***Q2: Comparison with camera controllable video generation methods.***\n\nWe mainly focus on Text-to-3D generation in our original manuscript so that we choose baselines with 3D generation ability. However, your advice is very insightful since both our method and the camera controllable video generation methods can generate videos given trajectories. Compared to these methods (e.g., CameraCtrl), Director3D: \n\n1. Can adaptively generate camera trajectory with Traj-DiT, while CameraCtrl uses only user-defined trajectories.\n2. Can generate 3D-consistent videos with 3DGS rendering, while CameraCtrl cannot ensure the 3D consistency.\n3. Can denoise only key frames for 3D scene and render other frames effectively, while CameraCtrl needs to denoise each frame with video diffusion models.\n\nFrom this perspective, our GM-LDM can be regarded as a novel and effective formulation of 3D-consistent camera controllable video diffusion models with 3DGS as the immediate 3D representation. Thanks for your kind reminder and we will add these discussions to our revision.\n\n---\n\n***Q3: Number of Gaussians.***\n\nSince the Gaussian decoder of GM-LDM is converted from the VAE decoder of the original Stable Diffusion, we set the size of Gaussians for each view equal to the size of image pixels, which is 256\\*256. The Gaussians of the scene are merged from the Gaussians of all input views, which is 8\\*256\\*256 ≈ 520K. During subsequent refining, this number will be changed by the Adaptive Density Control as in the original 3DGS. \n\n---\n\n***Q4: User-specific Camera Trajectories.***\n\nYes. Director3D supports using user-specified camera trajectories for 3D scene generation. Please refer to CQ2 in the \"Global Rebuttal\".\n\n---\n\n***Q5: Novel view inference ability.***\n\nEfficient novel view inference ability is one of the key advantages of Director3D over traditional video generation models since there is an immediate 3D representation.\nThe cases presented in our video gallery are rendered with the generated 3DGS and interpolated novel cameras.\nFor more novel view synthesis results after 3D scene generation, Please also refer to CQ2 in the \"Global Rebuttal\".\n\n---\n\n***Q6: Failure cases.***\n\nSimilar to 2D diffusion models, the success rate of Director3D decreases when generating with complicated and compositional prompts, objects with exact numbers, and articulated objects.\n\nWe provide several examples as failure cases in Fig. 4 of the \"Global Rebuttal PDF\". Thanks for your advice and we will add these to our revision.\n\n---\n\n***Q7: The reasons for the good results of Director3D with two limited 3D datasets.***\n\nCompared with object-level methods trained on Objaverse (e.g., MVDream and GRM), several key points contribute to our superior visual results:\n\n1. Compared to synthesized datasets (e.g., Objaverse), the **style** of the real-world captured datasets used in Director3D is much closer to the real images in 2D datasets (e.g., LAION). From this perspective, Director3D can take better advantage of **prior knowledge** from the pre-trained Stable Diffusion networks and the **collaborative training** with the LAION dataset.\n2. The generated scenes of Director3D include **reasonable real-world backgrounds**, and the proposed SDS++ loss provide **superior refining ability** with prior of 2D diffusion model. From this perspective, the SDS++ loss can enhance the entire scenes with realistic shadows, lighting, and reflections, making the results much better than the object-level methods.\n\n---\n\n***Q8:  How to divide the test set and how to confirm that test prompts are not seen during training.***\n\nDirector3D is aimed at generating 3D scenes given open-world prompts. To fully leverage the real-world captured video datasets of limited sizes for this challenging task, we do not divide a test set but employ novel and unseen prompts for evaluation.\n\nWe present evidence from two perspectives which can confirm that test prompts are novel:\n\n1. For object-centric cases, we provide the Top 1 category in the MVImgNet training set with the highest CLIP similarities to the main subjects in our test prompts, as shown in Tab. 1 of the \"Global Rebuttal PDF\".\n2. For scene-level prompts, we provide the Top 1 prompt in the DL3DV-10K training set with the highest CLIP similarities to our test prompts, as shown in Tab. 2 of the \"Global Rebuttal PDF\".\n\nThese pieces of evidence demonstrate that the test prompts used for evaluating Director3D are not included in the training set.\n\n---\n\n**References:**\n\n- ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image. \n- CameraCtrl : CameraCtrl: Enabling Camera Control for Video Diffusion Models. \n\n--- \n\nPlease kindly let us know if our responses have addressed your concerns. We are also happy to answer any of your remaining concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
---

## ***Global Rebuttal***

We express our gratitude to all reviewers for their recognition of the **potential applications** (Reviewer YUdV), the **novel and interesting idea** (Reviewers Lv1o & sdLT & yFX5), the **brilliant results** (Reviewers Lv1o & sdLT & yFX5), and the **well-written paper** (Reviewers YUdV & sdLT & yFX5) of Director3D. 
Their suggestions are beneficial for enhancing our submission. Some common questions are addressed as follows.

---

***CQ1: More evaluation and baselines.***

We thank the reviews for this advice, we further showcase:

1. a quantitative comparison with three scene-level baselines (e.g., GaussianDreamer with initial ground, LucidDreamer and ZeroNVS) for 32 object-centric prompts and 32 scene-level prompts, as shown in the following table.

```
Table 1. Quantitative comparison between Director3D and several scene-level baselines
------------------------------------------------------------------------------------
                            NIQE Score↓   CLIP-Similarity↑   Inference Time (min)↓  
------------------------------------------------------------------------------------
  GaussianDreamer            6.96          71.8               15                     
  ZeroNVS*                   9.84          67.2               90                     
  LucidDreamer-LLFF*         3.53          83.3               40                     
  LucidDreamer-HeadBang*     3.61          82.9               40                     
  LucidDreamer-BackForth*    3.40          74.2               40                     
  Ours                       4.09          83.9               5                      
------------------------------------------------------------------------------------
*: Image-to-3D scene generation methods
```

Our Director3D achieves the highest CLIP-Similarity and the second-best NIQE Score with the shortest inference time. LucidDreamer attains the best NIQE Score; However, we observe that it is plagued by multi-view inconsistency, visible artifacts at edges, and excessive objects as shown in Fig. 7 in the original manuscript.

2. a qualitative comparison with ZeroNVS in Fig. 1 of the "Global Rebuttal PDF".

Without adaptively generated camera trajectories (Traj-DiT) and a high-performance multi-view diffusion model with immediate 3D representation (GM-LDM), ZeroNVS exhibits deteriorated visual quality from multiple viewpoints and requires a time-consuming SDS optimization process from scratch for each scene.

---

***CQ2: User-specific camera trajectories.***

Director3D supports the utilization of both pre-generation and post-generation user-specific camera trajectories.

1. Pre-generation user-specific camera trajectories. Users are capable of employing user-specific camera trajectories instead of the generated camera trajectories from Traj-DiT for 3D scene generation. We present generation results with three user-specific camera trajectories for a specific prompt in Fig. 2 of the "Global Rebuttal PDF".

2. Post-generation user-specific camera trajectories. After generating the 3D scene, users can render novel views by providing novel cameras. We develop an interactive demo for visualizing the generated camera trajectories and 3D Gaussians, which is also capable of rendering the 3D Gaussians with novel cameras, as shown in Fig. 3 of the "Global Rebuttal PDF".

---

Other questions of each reviewer are addressed point-by-point in the separate rebuttals and we will revise the manuscript according to the rebuttals.

***The code of our submission will be released to the community for reproduction and further modification.***

----

***Please note that the figures for rebuttals are in the "Global Rebuttal PDF".***

Meta-review: This paper presents a framework for simultaneous text-to-3D scene and camera trajectory generation.  The proposed framework is a 3-stage pipeline: generating a dense camera trajectory from input text, using multi-view latent diffusion from a sparse subset of the generated trajectory to generate the 3D scene representation (Gaussian splats), and refining the GSs with a modified score distillation sampling (SDS) loss.  The presented experiments demonstrate the outperformance of the proposed method over SOTAs. The main concerns raised by the reviewers were insufficient validation of the proposed method, justification of coupling 3D scene generation and camera trajectory generation, over-fitting problem, and unclear explanation about the modified SDS loss.  A questionable word of “real-world” camera trajectory was also pointed out. The authors’ rebuttal resolved most of the raised concerns.  During the post-rebuttal discussion, the relation between the current scene reconstruction and the frame number used for diffusion was further discussed for clarification. The reviewer YUdV is still concerned with the word of “real-world” camera trajectory although he thinks that the strengths outweigh the weaknesses. AC also thinks that “large-scale real-world video datasets are primarily captured by handheld cameras” is a little bit overstated. Directly connecting the word “real-world” to camera trajectories that are invariably unstable, noisy, and scene-specific does not seem a good idea because “real-world” reminds us of a more general concept.  AC suggests to use another word such as wild.  Since the reviewers are all supportive, the paper should be accepted.  All the discussion should be incorporated into the final version.
Decision: Accept (poster)