Title: Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text
Venue: NeurIPS 2024
Accepted: yes
Keywords: Text-to-3D Scene Generation

Latent Diffusion Models
Abstract: Recent advancements in 3D generation have leveraged synthetic datasets with ground truth 3D assets and predefined camera trajectories. However, the potential of adopting real-world datasets, which can produce significantly more realistic 3D scenes, remains largely unexplored. In this work, we delve into the key challenge of the complex and scene-specific camera trajectories found in real-world captures. We introduce Director3D, a robust open-world text-to-3D generation framework, designed to generate both real-world 3D scenes and adaptive camera trajectories. To achieve this, (1) we first utilize a Trajectory Diffusion Transformer, acting as the \emph{Cinematographer}, to model the distribution of camera trajectories based on textual descriptions. Next, a Gaussian-driven Multi-view Latent Diffusion Model serves as the \emph{Decorator}, modeling the image sequence distribution given the camera trajectories and texts. This model, fine-tuned from a 2D diffusion model, directly generates pixel-aligned 3D Gaussians as an immediate 3D scene representation for consistent denoising. Lastly, the 3D Gaussians are further refined by a novel SDS++ loss as the \emph{Detailer}, which incorporates the prior of the 2D diffusion model. Extensive experiments demonstrate that Director3D outperforms existing methods, offering superior performance in real-world 3D generation.

Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text

Abstract

Recent advancements in 3D generation have leveraged synthetic datasets with ground truth 3D assets and predefined camera trajectories. However, the potential of adopting real-world datasets, which can produce significantly more realistic 3D scenes, remains largely unexplored. In this work, we delve into the key challenge of the complex and scene-specific camera trajectories found in real-world captures. We introduce Director3D , a robust open-world text-to-3D generation framework, designed to generate both real-world 3D scenes and adaptive camera trajectories. To achieve this, (1) we first utilize a Trajectory Diffusion Transformer, acting as the Cinematographer , to model the distribution of camera trajectories based on textual descriptions. (2) Next, a Gaussian-driven Multi-view Latent Diffusion Model serves as the Decorator , modeling the image sequence distribution given the camera trajectories and texts. This model, fine-tuned from a 2D diffusion model, directly generates pixel-aligned 3D Gaussians as an immediate 3D scene representation for consistent denoising. (3) Lastly, the 3D Gaussians are further refined by a novel SDS++ loss as the Detailer , which incorporates the prior of the 2D diffusion model. Extensive experiments demonstrate that Director3D outperforms existing methods, offering superior performance in real-world 3D generation.

1

A stainless steel toaster sitting on

a marble countertop

A paint-splattered easel

A vibrant marketplace in the early

morning...

A faux-fur leopard print hat

A derelict space station orbiting a

desolate planet...

A steep gorge carved by ancient

rivers, with rocky walls and...

A collection of fresh vegetables

arranged in a wicker basket

A bald head of a man

A white swan on a tranquil lake

A snowy woodland path lined

with frost-covered trees and

...

An expansive badlands terrain,

where jagged rock formations...

A cluster of tents pitched near a

forest

1 Introduction

Generating 3D scenes from texts holds great promise for industries such as gaming, robotics, and VR/AR. Previous methods [1, 2, 3], which use score distillation sampling (SDS) to optimize 3D representations such as Neural Radiance Fields (NeRFs) [4, 5], involve lengthy and unstable optimization processes. In contrast, newer approaches employ feed-forward networks [6, 7], e.g. , diffusion and reconstruction models, to directly generate 3D representations from text or text-guided multiview images, significantly enhancing generation speed. Moreover, advancements in 3D Gaussian Splatting [8] further accelerate training and rendering speeds, driving the next wave of progress in text-to-3D generation [9, 10, 11, 12, 13]. However, most existing methods focus solely on object-level 3D generation. Recently, preliminary works [14, 15, 16, 17, 18] have begun addressing scene-level 3D generation. Despite these efforts, visual quality, generation speed, and generalization remain suboptimal due to reliance on only 2D priors or limited few-classes 3D datasets.

In this work, we leverage real-world datasets ( e.g. , MVImgNet [19] and DL3DV-10K [20]) to achieve realistic text-to-3D generation. However, real-world captures from in-the-wild scenes differ significantly from traditional object-level synthetic datasets, introducing new requirements for the text-to-3D generation framework. Firstly, real-world captures feature complex, unpredictable, and scene-specific camera trajectories, unlike the controlled and predefined settings in object-level synthetic datasets like Objaverse [21] shown in Fig. 3 (Left). Secondly, real-world scenes include unbounded backgrounds, complicating the use of common bounded 3D representations such as Tri-planes [22]. Lastly, the diversity and quantity of real-world captures are limited, potentially decreasing the generalization ability for open-world texts.

We address these challenges with a novel framework, Director3D, illustrated in Fig. 1 and Fig. 3 (Right). Fig. 2 shows that our framework supports generating 3D scenes across various domains. In summary, our approach includes the following three key components:

Traj-DiT (Trajectory Diffusion Transformer) as Cinematographer: Generates dense-view camera trajectories from text. Camera parameters (intrinsics and extrinsics) are treated as temporal tokens, and a Transformer model performs conditional denoising of the camera trajectory.

GM-LDM (Gaussian-driven Multi-view Latent Diffusion Model) as Decorator: Uses a sparse-view subset of the camera trajectory for image sequence diffusion, generating pixel-aligned and unbounded 3D Gaussians as intermediate 3D representations. This model, fine-tuned from a 2D latent diffusion model, leverages strong priors and collaborative training with multi-view and single-view data to mitigate the limited diversity and quantity of real-world captures, enhancing generalization.

SDS++ Loss as Detailer: Enhances visual quality of the 3D Gaussians by back-propagates a novel SDS++ loss from images rendered at randomly interpolated cameras within the trajectory.

Text

(Traj-DiT)

Decorator

(GM-LDM)

Detailer

(SDS++ loss)

Pixel-aligned 3D Gaussians

Refined 3D Gaussians

Dense-view Trajectory

Sparse-view

Continues-view

Real-world Dataset

Synthetic Dataset

2 Related Works

Text-to-3D Generation. Since the introduction of DreamField [23], which combines the visionlanguage model CLIP [24] with NeRFs [4], there has been substantial progress in the area of text-to-3D generation. DreamFusion [1] and SJC [25] employ 2D image diffusion models to refine the 3D representation through Score Distillation Sampling (SDS). Subsequent methods [26, 27, 28, 29, 30] enhance the formulations to more effectively utilize 2D diffusion models for text-to-3D generation, achieving greater stability and improved visual quality. Some methods [3, 31, 32, 33] suggest incorporating an additional 3D prior into 2D diffusion models to enhance 3D consistency. Additionally, various methods [34, 35, 36, 37, 7, 38] employ 3D diffusion models to directly produce 3D assets. However, most of these works primarily focus on object-level generation. For scenelevel 3D generation, preliminary works [15, 14, 39, 40, 41, 16, 42] combine image inpainting models [43, 44] and monocular depth estimation models [45, 46] to progressively lift an image into a 3D scene with a user-defined camera trajectory. Nonetheless, the multi-view inconsistency of image inpainting and monocular depth estimation can lead to undesirable geometry and texture artifacts. Also, some works [47, 48] propose using panorama image diffusion models for 3D scene generation, which is limited to some specific types of scenes such as indoor rooms. Therefore, developing a text-to-3D scene generation method with open-world generalization capabilities remains unsolved.

3D Gaussian Splatting (3DGS) [8] introduces parameterized 3D Gaussians as 3D representation and splatting-based rasterization technique for novel view synthesis based on dense-view images, significantly reducing the rendering time compared to NeRF-based methods [4, 49, 5, 50, 51]. Current methods concentrate on improving the geometry quality [52, 53, 54], stabilizing the training [55], adapting it to dynamic scene modeling [56, 57, 58, 59] and applying it to different applications [60, 61, 62, 63, 64]. Further, to train a generalizable sparse-view reconstruction model with 3D Gaussians as the intermediate representation, some methods [65, 66, 67, 38, 12, 13] propose to convert image features into pixel-aligned 3D Gaussians and optimize reconstruction models through back-propagating losses from rendered images.

Meanwhile, DreamGaussian employs 3DGS for text-to-3D generation, with SDS loss as the optimization objective. GaussianDreamer [11] and GSGen [10] further enhance the generation quality and 3D consistency by initializing the 3D Gaussians based on point cloud diffusion models, Point-E [34]. The generalizable sparse-view reconstruction model ( e.g. , GRM [38], LGM [12], and GS-LRM [13]) for 3D Gaussians can also facilitate text-to-3D generation, using multi-view diffusion models [3, 68] to acquire sparse-view images as inputs. Specifically, GS-LRM takes a step forward by employing a large video generation model (i.e., SORA [69]) for text-to-3D scene generation. However, these works heavily rely on the 3D consistency of the multi-view images, which the 2D-based diffusion models can not guarantee. Distinctively, our GM-LDM employs pixel-aligned 3D Gaussians as the intermediate 3D representation for rendering-based multi-view diffusion, directly enforcing 3D consistency during the diffusion process and producing 3D representations.

3 Preliminary

Latent Diffusion Models (LDMs) [44, 70] consist of two key components: an auto-encoder [71] and a latent denoising network. The autoencoder establishes a bi-directional mapping from the space of the original data to a low-resolution latent space: z = E ( x ) , x = D ( z ) , where E and D are the encoder and decoder, respectively. The latent denoising network ϵ θ is trained to denoise noisy latent

given a specific timestep t and condition y . Its training objective for ϵ -prediction is defined as:

Multi-view Diffusion Models aim to model the distribution of multi-view images X with 3D consistency, where each image is captured by a distinct camera within the same static 3D scene. Its objective for x 0 -prediction can be written as:

where the noisy latent is obtained by z t = √ ¯ α t E ( x ) + √ 1 -¯ α t ϵ , ¯ α t is a monotonically decreasing noise schedule and ϵ ∼ N (0 , 1) is a random noise. During inference, a random noise is sampled as z T ∼ N (0 , 1) . By continuously denoising the random noise z T with condition y ( e.g. , text embedding), we can derive a fully denoised latent ˆ z . Then, the denoised latent ˆ z is fed into the latent decoder D to generate the high-resolution image ˆ x = D (ˆ z ) .

where C represents the camera parameters for the different views. Early works [3, 31] in this field are based on 2D LDMs. They fine-tune the 2D LDMs by integrating cross-view connections between the multi-view images into the original single-view 2D LDMs, using multi-view data rendered from 3D datasets. These methods lack strict 3D consistency since there is no actual 3D representation during multi-view denoising. A more advanced approach, DMV3D [7], employs a 3D reconstruction model to generate noise-free 3D representations and predict multi-view images from noisy multi-view inputs by a rendering-based denoising process. This enables 3D generation tasks to be accomplished without per-asset optimization during inference.

Score Distillation Sampling (SDS) [1, 2] uses a pretrained 2D diffusion model to optimize 3D representation. Considering a differentiable 3D representation parameterized by G and a rendering function denoted as R , the rendered image produced for a given camera pose c can be expressed as x = R ( G , c ) . SDS distills the prior of a 2D LDM to optimize 3D representation G as follows:

where ϵ is the ground truth noise, ˆ ϵ is the noise predicted by the 2D LDM with z t as input for timestep t , and w ( t ) represents a weighting function that varies according to the timestep t . The SDS loss can be also converted into a reconstruction-like objective [72]:

4 Method

4.1 Problem Formulation and Overview of Director3D

We consider the multi-view dataset of real-world captures as a joint distribution of image sequences and camera trajectories conditioned on texts, denoted as p (( X , C ) | y ) . Here, X = { x i } i M =1 represents the image sequence, C = { c i } i M =1 denotes the camera trajectory, and M is the number of views. To model this joint distribution, we separately handle the conditional distributions p ( C| y ) and p ( X| ( C , y )) (see Appendix B for detailed discussions). Furthermore, we model each image in the sequence as a rendered view of a unified 3D scene representation G under the corresponding camera, expressed as x i = R ( G , c i ) , where R is the 3D rendering function.

Director3D addresses this by incorporating three collaborative processes analogous to roles in film production: the Cinematographer, the Decorator, and the Detailer. Firstly, the Trajectory Diffusion Transformer (Traj-DiT), serving as the Cinematographer, models the distribution of dense-view camera trajectories, as detailed in Sec. 4.2. For image sequences, directly modeling the dense-view distribution is complex and resource-intensive. To address this, we use a Gaussians-driven Multi-view Latent Diffusion Model (GM-LDM), acting as the Decorator, to model the image distribution through a sparse subset of dense views. This model utilizes pixel-aligned 3D Gaussians as the intermediate representation, described in Sec. 4.3. Finally, to improve the visual quality of the generated 3D scenes, we employ a novel SDS++ loss, functioning as the Detailer, to refine the 3D Gaussians through dense-camera interpolation rendering, as presented in Sec. 4.4.

<latexi sh

1_b

64="5SLR

XO

YPDpGB/

d

c80A

>

9

V

gM

F

3j

W

kq

Q

K

U

J

2o

m

Ny

C

f

+T

7u

n

w

r

z

v

I

<latexi sh

1_b

64="Pq8SQU2

nNIE

0Y

J

OVK

/w

>A

B73

c

r

9L

g5

o

F

X

G

dy

D

M

v

W

z

H

T

R

f

+

<latexi sh

1_b

64="

gyH

V

2B

RDn7m

Jq9

WTGO

>A

+

c

LS

N

F

3U

v

du

o

0

5

k

Z

/

Q

8

wz

I

M

f

X

j

P

KY

r

Cp

E

<latexi sh

1_b

64="+ZSB

kC8

f

pX

cDHO3

v

gV

nI

>A

L

M

F

T

7W

q

d

m

Y

rQ

y

9

P

u

E

/jz

K

o

N

J

R

5

G

0U

w

2

DiT

Block

Self-

A

tt

ention

Layer

MLP

Layer

Text

Timestep

Cross-

A

tt

ention

Layer

×

n

C

t

ˆ

C

t

↓

A park

An apple

4.2 Traj-DiT as Cinematographer

To model the trajectory distribution, we represent the camera trajectory C as a set of camera parameters c i = { r i , t i , f i , p i } , where r ∼ SO(3) and t ∼ R 3 are the rotation and translation of the camera poses, f ∼ R 2 + is the focal lengths and p ∼ R 2 is the principle points. To ensure consistency and comparability across scenes, we normalize the trajectory for each scene in two steps: First, we convert all camera poses to be relative to the first one so that the first camera pose is an identity matrix; Then, we re-scale the translation to make the distance from the first to the farthest camera to 1 . We adapt the architecture of the Diffusion Transformer (DiT) [73] to generate camera trajectories, as illustrated in Fig. 4 (Left). The temporal order of real-world captures, akin to video sequences, necessitates a learnable temporal embedding to differentiate between cameras of different frames. This embedding helps the model capture the sequential dependencies inherent in real-world data. Each DiT block includes a cross-attention layer to extract information from text embeddings encoded by the CLIP [24] text encoder. Additionally, the timestep t modulates the pre-layer normalization and post-layer output scalar, similar to the original DiT, allowing the model to learn temporal dynamics effectively. The model is trained to minimize the x 0 -prediction diffusion objective:

where C θ is the parameterized Traj-DiT model and C t is the noisy camera trajectory.

By leveraging the strengths of the DiT architecture, we aim to enhance the fidelity and coherence of the generated trajectories, instead of relying on pre-defined ones. We showcase two examples of the predicted camera trajectories for different denoising steps t in Fig. 4 (Right), demonstrating the effectiveness of our model in generating smooth and accurate camera paths.

4.3 GM-LDM as Decorator

We propose GM-LDM to model the image sequence distribution p ( X| ( C , y )) and generate immediate 3D Gaussians as the joint 3D scene representation. The GM-LDM, fine-tuned from the Stable Diffusion model with a slightly modified architecture, leverages its image generation prior to enhance 3D scene generation. For efficiency, diffusion is applied to a sparse-view subset of the camera trajectory, significantly reducing computational overhead. During training, sparse-view images are processed through the frozen latent encoder E to obtain multi-view latents Z ∼ R N × c × h × w , where N ≤ M . Noise is then added to these multi-view latents Z to produce noisy latents Z t .

2D-based Denoising. The noisy multi-view latents Z t are fed into the latent denoising network Z θ in parallel. For convenience, we modify the ϵ -prediction of the original Stable Diffusion model to x 0 -prediction. The denoised multi-view latents are obtained as { ˆ Z , F} = Z θ ( {Z t } , C ′ , y, t ) , where F represents additional multi-view features for enhanced 3D information. Sparse-view cameras C ′ are integrated into the network by combining the ray-maps ( o × d , d ) [7] with the noisy latents, where o and d denote the origin and direction of pixel-aligned rays, respectively. We replace self-attention blocks in the original 2D latent denoising network with cross-view self-attention blocks [3] to better capture multi-view correlations. The denoised multi-view latents ˆ Z are supervised using a simple

<latexi sh

1_b

64="k

VynMZO80

ISLR

A

7D2Cf9

>

B

H

c

N

F

pr dug

w

W

Q5

J+3

T

K

G

z/

X

j

vm

q

o

Y

E

U

P

<latexi sh

1_b

64="

H

U

cZT

E9

vCI

SQ

>A

B/X

DL

N

J3

rP

z

k

y

2

m

o

0

d

Y

jn

u

p

W

7

f

O

G

8

+

q

5

R

M

<latexi sh

1_b

64="uk

I

m

jB3YWPoRgEF

>A

+n

c

VDLS

N

J

r

q

M

CG5

7

O

T

0

z

Q

82

w

Z

y

v

X

d

K

p

9

H

/

<latexi sh

1_b

64="SrF

DTINZ

zpGcn

Yw

2

R

oX

>A

C

H

L

P

q3W

9

J

m

g

Ku

M

O

fBU

8

E

0

j/+

5

vV

7

Q

y

k

d

<latexi sh

1_b

64="qDwPNd8X

z

R

pHvV

g

>A

B+3

c

S

FJ

U

2

W

o

k

0

Y

/

KZT

m

y

G

En

j

M

C

5

r

9

7

O

f

<latexi sh

1_b

64="5C2V3NcW

7

ToRSy

z

Q

IY

>A

B8n

DL

gM

F

X

+q

d

v

p

Z

KH

0

k

w

m

GJ

OU

/

j

u

P

r

f

9

E

<latexi sh

1_b

64="j3O05w7

DfZyg

K

LN

cvp

IGA

>

B

n

V

S8

EJ

W

X

k

UQ

9C

Y

H

2

d

M

u

P

o

/

m

zT

r

q

R

+

F

t

= 1

multi-view latent diffusion objective:

Rendering-based Denoising. To generate 3D Gaussians for rendering-based denoising, the denoised multi-view latents ˆ Z and additional features F are input into a Gaussians decoder D G . This decoder outputs Gaussian features { τ i , q i , s i , α i , c i } = D G ( ˆ Z i , F i ) , where τ i , q i , s i , α i , and c i represent the depth, rotation quaternion, scale, opacity, and spherical harmonics coefficients of 256 × 256 3D Gaussians for view i , respectively. The Gaussians decoder D G is initialized with the weights of the original latent decoder D , with re-initialized first and last convolutional layers to handle the additional features and specific Gaussian channels. The predicted depth is then converted into pixel-aligned Gaussian positions µ i = o i + τ i d i . The multi-view 3D Gaussians G = { µ i , q i , s i , α i , c i } N i =1 are concatenated to jointly represent the 3D scene. During training, views are randomly sampled from the dense-view camera trajectory to supervise the predicted 3D Gaussians in image space, ensuring consistent and accurate 3D scene representation:

where R is the rendering function, ℓ ( · , · ) is a reconstruction loss penalizing the difference between images, and ( x, c ) ∈ ( X , C ) is the ground truth of an image and camera pair from the dense views. We use a combination of MSE loss and LPIPS [74] loss for the reconstruction loss ℓ , similar to the original reconstruction loss of the original auto-encoder. The total training loss is simply the sum of the above losses: L = L 2d + L 3d. This approach leverages the strengths of multi-view data and pixel-aligned Gaussian representations to enhance the fidelity and coherence of the generated 3D scenes. By fine-tuning from a robust 2D LDM and using a sparse-view subset, we strike a balance between performance, efficiency, and generalizability, enabling the generation of coherent 3D scenes. As shown in Fig. 5 (Left), during inference, images are rendered with the input cameras C ′ and encoded by the latent encoder E to obtain Gaussian-driven denoised latents:

Inspired by Dual3D [75], GM-LDM inference can toggle between 2D-based and rendering-based denoising. The 2D-based denoising offers better generalization, aligning closely with the original Stable Diffusion, while rendering-based denoising ensures superior 3D consistency due to its immediate joint 3D representation. The 3D Gaussians generated in the final denoising step serve as the initial 3D scene for subsequent refinement.

Collecting and annotating real-world multi-view datasets is laborious, often resulting in limited diversity and quantity, which hinders generalization for open-world texts. To address this, we follow MVDream [3] and collaboratively train the GM-LDM using both multi-view and 2D datasets to enhance generalization. We treat single-view images as a special case of multi-view images with N = M = 1 and apply the same rendering process and training losses, which increases the diversity of training data, thereby improving the model's ability to generalize across diverse scenarios.

<latexi sh

1_b

64="rINLp

f

Y+J

mDvF8kyE2d

U

>A

B

n

c

V

SgM

TX7W

q

Q

RZ

u

z

HK0

9w

3

O/

G

j

C

P

o

5

<latexi sh

1_b

64="OrLU

JX

c2P

C

GgfN

ZYzy vE

>A

H

VD

S

F

3

d

n

B

mqoMu

W

9

Qp

+

T

j

I7/8

w

0

K

R

k

5

<latexi sh

1_b

64="5C2V3NcW

7

ToRSy

z

Q

IY

>A

B8n

DL

gM

F

X

+q

d

v

p

Z

KH

0

k

w

m

GJ

OU

/

j

u

P

r

f

9

E

<latexi sh

1_b

64="

Yg

r

S3

8

J

fdL

/O

KU

>A

B

H

c

VD

N

E

y

9

oP

Q

uF

I

Z

5

Mj

p

v

+

0

C

q

n

X

7

G

W

2

R

T

w

<latexi sh

1_b

64="

Y

GzjdJFqyn gm

r

V

H

OI

>A

B7

c

NS8

E

3U

/

9L

5

o

P

wX

p

u2

Z

KW0

M

v

T

RQ

f

D

+

<latexi sh

1_b

64="

g

A

p

dQ

>

3

V

LS

N

FJ

rP

BGE

km

o

u

y

2

Z

7

/

R

0

8

5

v

z

W

Kj

T

n

O

9

f

w

<latexi sh

1_b

64="c

y

U+FKnDpX0fV

q

I

B

EM

>A

8

LSg

T

7W

d

v

Q

RZ

C

Y

k

w

m

GJ

O

o

/

N

2

j

u

P

H

r

z

5

3

9

<latexi sh

1_b

64="rINLp

f

Y+J

mDvF8kyE2d

U

>A

B

n

c

V

SgM

TX7W

q

Q

RZ

u

z

HK0

9w

3

O/

G

j

C

P

o

5

<latexi sh

1_b

64="NfGR

TA70

vwVU

D

X

B3

nm

Y

>

H

c

LSg

EOy

r

9

oP

Q

8

I

Z

zk

5

Mj

+

/

C

q

d

p

W

2

K

<latexi sh

1_b

64="kREYJ

w7vg

K

f

o

G5nq

>A

B

c

V

NS8

3Ur

/

9L

C

PF

X

pmu2

dy

0

M

O

W

z

H

T

Q

D

I

+

<latexi sh

1_b

64="

NIwU

q

QMr

K

g

BCu

F

Y

>A

+H

c

Z

S8

E

3f

0

L

J

V

GP

9

m7

vp

/

2

D

z

d

W

o

R

n

k

X

O

j

y

5

T

<latexi sh

1_b

64="yW/rn uq

o80

YG

O

d

9

LETA

>

B

c

V

NS

J3

X

k

UQ

F

2g

+

z

D

7

Im

Z

j

w

HC

v

5

M

K

p

P

R

f

<latexi sh

1_b

64="Uu0A+q ywQ

9

C

H

5kEV

g

>

B/X

c

DLS

N

FJ3

rP

z

p

W

Y

KZT

IM

R

O

G

jn

2

8

m

7

d

o

v

f

<latexi sh

1_b

64="U2Yf5SI/

qMun

XgJ3

C

O

>A

B7

cjVDL

N

E

z

G

r

H

8

9

WP

T

wo

k

Q

p

d

Z

K

0

v

F

y

+

m

R

E

Z

θ

ˆ

z

x

z

t

E

Z

t

ˆ

Z

ˆ

Z

G

z

/epsilon1

ˆ

x

D

/epsilon1

θ

D

G

G

Back-propagate

ˆ

X

′

G

L

SDS++

{

ˆ

y, y, φ

}

4.4 SDS++ Loss as Detailer

To enhance the details and visual quality of the 3D Gaussians, we propose the SDS++ loss, leveraging the 2D diffusion prior for refinement. Our research on existing SDS-based methods identifies three key points crucial for success, this includes: (1) Appropriate Target Distribution [26]: This ensures that the rendered images align effectively with the textual conditions, avoiding over-smoothing and over-saturation. (2) Adaptive Estimation of the Current Distribution [26, 76]: This provides a counter optimization objective, pushing the rendered images away from the current distribution to enhance details. (3) Latent-space and Image-space Objectives [72]: Combining these objectives helps prevent noisy or over-smoothing artifacts that may arise from using only one of them. During refining, we first render the 3D Gaussians G with a randomly sampled camera c from the continuous interpolated camera trajectory to produce an image x = R ( G , c ) . This image is encoded into the latent space by E ( x ) = z , then disturbed with randomly sampled noise and timestep t to produce a noisy latent z t . The 2D diffusion model ϵ θ then predicts the denoised latent ˆ z from z t . As illustrated in Fig. 5 (Right), the proposed SDS++ loss can be formulated by:

where λ z and λ x are the weights for latent-space and image-space objectives, respectively, ˆ z is the predicted latent, and x = D (ˆ z ) is the predicted image. The predicted latent can be derived by Eq. 4. We use a compositional predictions ˆ ϵ as follows:

Instead of setting ˆ ϵ src = ϵ as in the standard SDS loss, we introduce a learnable source prediction ˆ ϵ src for adaptive estimation of the current distribution:

where ϵ θ ( z t , ˆ y, t ) uses a learnable text embedding ˆ y to efficiently estimate the current distribution. This approach leverages the original latent denoising network [76] and is trained by minimizing ∥ ϵ θ ( z t , ˆ y, t ) -ϵ ∥ 2 2 along with the refining process. The target prediction employs classifier-free guidance for improved text alignment:

where ω cfg is the classifier-free guidance scale. SDS++ loss integrates the above three key points, ensuring efficient and realistic refinement of 3D Gaussians.

5

Experiments

5.1 Implementation Details

We utilize the MVImgNet [19] for object-level and DL3DV10K [20] for scene-level real-world multiview datasets. Text prompts for each scene are generated using the multi-modal large language model InternLM-XComposer [77]. To enhance generalization, we incorporate the 2D dataset LAION [78]. For GM-LDM, we set the lengths of dense and sparse views to M = 29 and N = 8 , respectively. The classifier-free guidance scale is 7 . 5 for 2D-based denoising and 1 for rendering-based denoising to ensure 3D consistency. Following Dual3D [75], we balance 3D consistency and generalization by using 1/10 rendering-based denoising steps. Image and latent resolutions are set to 256 and 32 , respectively. For SDS++ loss, the weights for latent-space and image-space losses are λ z = 1 and λ x = 0 . 01 . ω cfg is set to 7 . 5 , with refining iterations set to 1000 . Generating a scene takes approximately 5 minutes. Further implementation details are provided in Appendix A.

5.2 Generation Results.

We show the generation results of camera trajectories and image sequences for various text prompts in Fig. 6. For object-level prompts, the generated camera trajectories typically circle and face the objects, aligning well with the distribution in MVImgNet. In contrast, scene-level prompts yield more diverse and complex camera trajectories, showcasing the effectiveness of our Traj-DiT model. Our method also generates realistic images across different types of prompts, demonstrating the effectiveness and generalization ability of GM-LDM and SDS++ loss. Additional generation results are available in Appendix D. These results highlight the robustness of our approach in handling both object-level and scene-level prompts for 3D scene generation.

a castle made of stone

55050505

An ornate stone fountain

GRM

Ours

GRM

Ours

GaussianDreamer

Ours

GaussianDreamer

Ours

DreamScene

Ours

DreamScene

Ours

LucidDreamer

Ours

LucidDreamer

Ours

NTANA

A blue and white vase with a lid

A lighthouse on a rocky shore

A wooden bench in an autumn park

A red and white lighthouse on a cliff

5.3 Qualitative Comparison.

Wequalitatively compare our method with several baseline methods, as shown in Fig. 7. (1) GRM [38] is a feed-forward text-to-3D generation method for 3D Gaussians using multi-view images generated by 2D-based diffusion models. It supports only object-level 3D generation. Our comparisons using object descriptions show that GRM produces unrealistic 3D Gaussians limited to objects, due to its training on synthetic datasets. In contrast, our method generates high-quality 3D scenes with both objects and backgrounds. (2) GaussianDreamer [11] is a state-of-the-art SDS-based method for 3D Gaussians, integrating priors from both 2D and 3D diffusion models. While it can generate objects with ground layers, it tends to produce over-saturated textures. Our method, however, generates more realistic scenes with better handling of shadows, lighting, and material reflections. (3) DreamScene [17] uses Formation Pattern Sampling and strategic camera sampling for 3D Gaussians. Since it is not open-sourced, we use examples from its project page. Although it generates scenewide consistent 3D scenes, the results are overly saturated and cartoonish. (4) LucidDreamer [16], based on Text2NeRF [14], uses 2D foundation models for 3D Gaussians. While it can generate photo-realistic textures, the multi-view consistency is poor, with visible artifacts at object edges due to inaccurate monocular depth estimation. It also struggles with excessive object generation from descriptive prompts due to reliance on single-view inpainting. These comparisons highlight the superior performance of the proposed Director3D for realistic 3D generation.

5.4 Quantitative Comparison.

We present a quantitative comparison between our framework and several baseline models in Tab. 1. For this experiment, we use the Single-Object-with-Surroundings 3 set of T3Bench [83], which contains 100 prompts closely matching the descriptions in MVImgNet. The quantitative results are evaluated using CLIP-Score [84], NIQE [85], and BRISQUE [86] metrics. For each 3D scene generated by different methods, we render a video and uniformly sample 36 frames to calculate the average score for each metric. For baselines without adaptive camera trajectories, videos are rendered by circling around the 3D representations at a fixed elevation. The BRISQUE and NIQE results indicate that our method significantly outperforms existing baseline models in terms of image quality. Additionally, the CLIP-Score shows our method's superior ability to align generated images with their textual descriptions, even without refining. These results underscore the robustness and effectiveness of our framework in generating high-quality, semantically aligned 3D scenes.

We also conduct a quantitative comparison with three scene-level baselines for 32 object-centric prompts and 32 scene-level prompts in the Tab. 2. Our method achieves the highest CLIP-Score and the second-best NIQE Score with the shortest inference time. Without adaptively generated camera trajectories (Traj-DiT) and a high-performance multi-view diffusion model with immediate 3D representation (GM-LDM), ZeroNVS exhibits deteriorated visual quality from multiple viewpoints and requires a time-consuming SDS optimization process from scratch for each scene. LucidDreamer attains the best NIQE Score; However, we observe that it is plagued by multi-view inconsistency, visible artifacts at edges, and excessive objects as shown in Fig. 7.

5.5 User-specific Camera Trajectories

Director3D supports the utilization of both pre-generation and post-generation user-specific camera trajectories. For pre-generation user-specific camera trajectories, users are capable of employing user-specific camera trajectories instead of the generated camera trajectories from Traj-DiT for 3D scene generation. For post-generation user-specific camera trajectories, users can render novel views by providing novel cameras after generating the 3D scene. We develop an interactive demo for visualizing the generated camera trajectories and 3D Gaussians, which is also capable of rendering the 3D Gaussians with novel cameras, as shown in Fig. 8.

novel views

distance view

A mountain pass, where the wind howls through craggy ...

novel views

distance view

AANBαM

5.6 Ablation Study

Director3D is formulated to a sequential process to model the joint distribution of camera trajectories and image sequences. The Traj-DiT component generates the base camera trajectory for each scene, while the GM-LDM provides the initial 3D scene. These two models are crucial for producing meaningful 3D scenes. Our ablation study primarily focuses on the SDS++ loss to highlight its significance. As shown in Fig. 9, we conduct comprehensive experiments to analyze the impact of the SDS++ loss. First, we remove the refining process entirely. Although the initial 3D Gaussians generated by GM-LDM match the input text, the visual quality is unsatisfactory, with missing details. This is expected due to the limited diversity and quantity of the multi-view dataset used for training GM-LDM. Setting ϵ src = ϵ degrades the SDS++ loss into SDS+ loss [72]. The results show a significant decrease in visual quality and an

full

w/o

refining

/epsilon1

src

=

/epsilon1

ω

cfg=1

λ

x

= 0

λ

z

= 0

A delicious hamburger on a wooden table

over-smoothing issue, highlighting the importance of adaptive estimation of the current distribution. Setting ω cfg = 1 turns the SDS++ loss into the LODS loss [76] with an additional image-space objective. This results in noisy details, as the conditional noise prediction alone does not provide a clear optimization direction. By setting λ x = 0 and λ z = 0 respectively, we observe that using only the latent-space objective leads to noisy details and artifacts, while using only the image-space objective results in missing details.

The full model, incorporating the proposed SDS++ loss, achieves the best visual quality with clear and realistic details. These findings underscore the importance of each component in the SDS++ loss and its role in refining 3D Gaussians. Additional ablation studies, including qualitative and quantitative of more cases and those with randomly generated trajectories, are provided in Appendix E.

6 Conclusion

In this paper, we propose an open-world text-to-3D generation framework capable of generating real-world 3D scenes with adaptive camera trajectory, named Director3D. We first introduce a Cinematographer ( i.e. , Traj-DiT) that can generate dense-view camera trajectories given texts. Then, a Decorator ( i.e. , GM-LDM) and a Detailer ( i.e. , SDS++ loss) are proposed for initial generation and further refining, respectively, with 3D Gaussians as the 3D scene representation. We demonstrate the effectiveness of our method with extensive experiments. We believe our work makes essential contributions to the text-to-3D generation community, especially in discovering the potential of leveraging real-world multi-view datasets for realistic 3D generation. Our future works include improving the generation scope, boosting model efficiency and quality, and leveraging more datasets.

<latexi sh

1_b

64="/VI

GSZE

q

F

m2

B3

k

pw

>A

n

c

N

8

W

X

D

5CR

U0

o

M

j

O

7Y

g

P

r

f

Q

z

v

d

K

L

J

+

y

u

9

H

T

<latexi sh

1_b

64="0A+XS5

uj

3r

G

Bkg

T

7WFNI

>

H

c

VDL

UZd ov

q

R

Q

O

y

/

C

mY

9E

z

2

p

n

f

8

J

K

w

M

P

<latexi sh

1_b

64="0gv

X

yk7

nUj

9

/

r

V

>A

C

c

Z

S

NBFI

3G2

q

zW

Q

MJu

L

R

E

KD

5

O

m

+

o

Y

T

H

8f

w

P

p

d

<latexi sh

1_b

64="

I

+CQof

m

pq83

L2/

kS

Xd

Y

>A

B

H

c

VD

N

F

W

vg

R

0

O

y

T

J

G

z

n

M

w

5

7

9

K

r

E

P

References

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022.

Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774 , 2022.

Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.16512 , 2023.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV) , 2020.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) , 41(4):1-15, 2022.

Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400 , 2023.

Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:2311.09217 , 2023.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (SIGGRAPH) , 42(4), July 2023.

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 , 2023.

Zilong Chen, Feng Wang, Yikai Wang, and Huaping Liu. Text-to-3d using gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054 , 2024.

Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-lrm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:2404.19702 , 2024.

Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven 3d scene generation with neural radiance fields. arXiv preprint arXiv:2305.11588 , 2023.

Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 7909-7920, October 2023.

Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384 , 2023.

Haoran Li, Haolin Shi, Wenli Zhang, Wenjun Wu, Yong Liao, Lin Wang, Lik hang Lee, and Pengyuan Zhou. Dreamscene: 3d gaussian-based text-to-3d scene generation via formation pattern sampling. arXiv preprint arXiv:2404.03575 , 2024.

Shijie Zhou, Zhiwen Fan, Dejia Xu, Haoran Chang, Pradyumna Chari, Suya Bharadwaj, Tejas You, Zhangyang Wang, and Achuta Kadambi. Dreamscene360: Unconstrained text-to-3d scene generation with panoramic gaussian splatting. arXiv preprint arXiv:2404.06903 , 2024.

Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9150-9161, 2023.

Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 22160-22169, 2024.

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 13142-13153, 2023.

Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16123-16133, 2022.

Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 867-876, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning (ICML) , pages 8748-8763. PMLR, 2021.

Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12619-12629, 2023.

Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.16213 , 2023.

Oren Katzir, Or Patashnik, Daniel Cohen-Or, and Dani Lischinski. Noise-free score distillation. arXiv preprint arXiv:2310.17590 , 2023.

Zi-Xin Zou, Weihao Cheng, Yan-Pei Cao, Shi-Sheng Huang, Ying Shan, and Song-Hai Zhang. Sparse3d: Distilling multiview-consistent diffusion for object reconstruction from sparse views. arXiv preprint arXiv:2308.14078 , 2023.

Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. arXiv preprint arXiv:2311.17984 , 2023.

Jinbo Wu, Xiaobo Gao, Xing Liu, Zhengyang Shen, Chen Zhao, Haocheng Feng, Jingtuo Liu, and Errui Ding. Hd-fusion: Detailed text-to-3d generation leveraging multiple noise estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3202-3211, 2024.

Zexiang Liu, Yangguang Li, Youtian Lin, Xin Yu, Sida Peng, Yan-Pei Cao, Xiaojuan Qi, Xiaoshui Huang, Ding Liang, and Wanli Ouyang. Unidream: Unifying diffusion priors for relightable text-to-3d generation. arXiv preprint arXiv:2312.08754 , 2023.

Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008 , 2023.

Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453 , 2023.

Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751 , 2022.

Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463 , 2023.

Zhicong Tang, Shuyang Gu, Chunyu Wang, Ting Zhang, Jianmin Bao, Dong Chen, and Baining Guo. Volumediffusion: Flexible text-to-3d generation with efficient volumetric encoder. arXiv preprint arXiv:2312.11459 , 2023.

J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 20875-20886, 2023.

Yinghao Xu, Zifan Shi, Wang Yifan, Sida Peng, Ceyuan Yang, Yujun Shen, and Wetzstein Gordon. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint arXiv: 2403.14621 , 2024.

Jiabao Lei, Jiapeng Tang, and Kui Jia. Rgbd2: Generative scene synthesis via incremental view inpainting using rgbd diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2023.

Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene generation. arXiv preprint arXiv:2302.01133 , 2023.

Dana Cohen-Bar, Elad Richardson, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Set-the-scene: Globallocal training for generating controllable nerf scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops , pages 2920-2929, October 2023.

Jaidev Shriram, Alex Trevithick, Lingjie Liu, and Ravi Ramamoorthi. Realmdreamer: Text-driven 3d scene generation with inpainting and depth diffusion. arXiv preprint arXiv:2404.07199 , 2024.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances In Neural Information Processing Systems (NeurIPS) , 33:6840-6851, 2020.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, 2022.

Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Rene Ranftl, Katrin Lasinger, David Hafner, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence , PP:1-1, 08 2020.

Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. arXiv preprint arxiv:2307.01097 , 2023.

Yikun Ma, Dandan Zhan, and Zhi Jin. Fastscene: Text-driven fast 3d indoor scene generation via panoramic gaussian splatting. arXiv preprint arXiv:2405.05768 , 2024.

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In 35th Conference on Neural Information Processing Systems , pages 27171-27183. Curran Assoicates, Inc., 2021.

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Sara Fridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.12775 , 2023.

Mulin Yu, Tao Lu, Linning Xu, Lihan Jiang, Yuanbo Xiangli, and Bo Dai. Gsdf: 3dgs meets sdf for improved rendering and reconstruction. arXiv preprint arXiv:2403.16964 , 2024.

Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. ACM Transactions on Graphics (SIGGRAPH) , 2024.

Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 , 2023.

Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. arXiv preprint arXiv:2312.16812 , 2023.

Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. In International Conference on Learning Representations (ICLR) , 2024.

Yuheng Jiang, Zhehao Shen, Penghao Wang, Zhuo Su, Yu Hong, Yingliang Zhang, Jingyi Yu, and Lan Xu. Hifi4g: High-fidelity human performance rendering via compact gaussian splatting. arXiv preprint arXiv:2312.03461 , 2023.

Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaussians for open-vocabulary scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5333-5343, 2024.

Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20051-20060, 2024.

Yansong Qu, Shaohui Dai, Xinyang Li, Jianghang Lin, Liujuan Cao, Shengchuan Zhang, and Rongrong Ji. Goi: Find 3d gaussians of interest with an optimizable open-vocabulary semantic-space hyperplane. In Proceedings of the 32nd ACM International Conference on Multimedia , pages 5328-5337, 2024.

Jonas Kulhanek, Songyou Peng, Zuzana Kukelova, Marc Pollefeys, and Torsten Sattler. Wildgaussians: 3d gaussian splatting in the wild. arXiv preprint arXiv:2407.08447 , 2024.

Yuze Wang, Junyi Wang, and Yue Qi. We-gs: An in-the-wild efficient 3d gaussian representation for unconstrained photo collections. arXiv preprint arXiv:2406.02407 , 2024.

Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. arXiv preprint arXiv:2403.14627 , 2024.

David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In The Twelfth International Conference on Learning Representations (ICLR) , 2024.

OpenAI. Sora: Creating video from text. https://openai.com/sora , 2024.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. Advances In Neural Information Processing Systems (NeurIPS) , 35:36479-36494, 2022.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations (ICLR) 2014, Banff, AB, Canada , 2014.

Junzhe Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d generation with advanced diffusion guidance. arXiv preprint arXiv:2305.18766 , 2023.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 4195-4205, October 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) , pages 586-595, 2018.

Xinyang Li, Zhangyu Lai, Linning Xu, Jianfei Guo, Liujuan Cao, Shengchuan Zhang, Bo Dai, and Rongrong Ji. Dual3d: Efficient and consistent text-to-3d generation with dual-mode multi-view latent diffusion. arXiv preprint arXiv:2405.09874 , 2024.

Xiaofeng Yang, Yiwen Chen, Cheng Chen, Chi Zhang, Yi Xu, Xulei Yang, Fayao Liu, and Guosheng Lin. Learn to optimize denoising scores for 3d generation. arXiv preprint arXiv:2312.04820 , 2023.

Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112 , 2023.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances In Neural Information Processing Systems (NeurIPS) , 35:25278-25294, 2022.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 300-309, 2023.

Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shapeguided generation of 3d shapes and textures. arXiv preprint arXiv:2211.07600 , 2022.

Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2023.

Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, and Jiajun Wu. ZeroNVS: Zero-shot 360-degree view synthesis from a single real image. CVPR, 2024 , 2023.

Yuze He, Yushi Bai, Matthieu Lin, Wang Zhao, Yubin Hu, Jenny Sheng, Ran Yi, Juanzi Li, and Yong-Jin Liu. T 3 bench: Benchmarking current progress in text-to-3d generation. arXiv preprint arXiv:2310.02977 , 2023.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a referencefree evaluation metric for image captioning. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making a 'completely blind' image quality analyzer. IEEE Signal Processing Letters , 20(3):209-212, 2013.

Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing , 21(12):4695-4708, 2012.

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , San Diega, CA, USA, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations (ICLR) 2021, Virtual Event, Austria , 2021.

Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In International Conference on Computer Vision (ICCV) , 2021.

Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In ACM Transactions on Graphics (SIGGRAPH) , 2018.

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

A Implemental Details

Datasets. For real-world multi-view datasets, we leverage MVImgNet [19] and DL3DV10K [20] datasets for object-level and scene-level captures, respectively. The MVImgNet dataset is a large-scale object-level dataset, comprising 219,188 videos across 238 classes 4 . The DL3DV10K dataset consists of 10 K scene-level videos with various indoor, outdoor, urban, and suburban environments 5 .

Traj-DiT setup. Our Traj-DiT is trained with the Adam optimizer [87], a constant learning rate of 5 e -5 and ( β 1 , β 2 ) = (0 . 9 , 0 . 95) . The batch size is set to 128 . The number of the DiT Blocks is 8 and the hidden size is 512 . Training takes about 2 days with 1 NVIDIA Tesla A100 GPUs for 50 K iterations. We use 1000 steps during training and reduce it to 100 steps with DDIM [88] during inference.

GM-LDMsetup. Our GM-LDM is trained with the Adam optimizer, a constant learning rate of 1 e -4 and ( β 1 , β 2 ) = (0 . 9 , 0 . 95) . The batch size is also set to 128 . Training takes about a week with 16 NVIDIA Tesla A100 GPUs for 150 K iterations. We use Stable Diffusion v2.1 6 as our initial model. The noise schedule and other hyper-parameters are consistent with the Traj-DiT.

SDS++ loss setup. For refining, the learning rates of the Gaussian parameters µ , q , s , α , and c are set to be 0 . 0001 , 0 . 01 , 0 . 001 , 0 . 01 , and 0 . 01 , respectively. The learning rate of the learnable text embedding ˆ y is set to be 0 . 001 . The timestep t is annealing from 0 . 75 T to 0 . 02 T with a square root schedule, where T is the total denoising steps. We use Stable Diffusion v2.1 as our refining model as well. The rendering resolution is set to be 512 during refinement.

B Discussions

Ways to Model p ( X , C ) . There are different approaches to modeling the joint distribution of image sequences and camera trajectories for real-world captures. We choose to independently model the distribution of camera trajectories and the conditional distribution of image sequences based on the camera trajectories. The image sequence is directly rendered with a joint 3D representation by the proposed GM-LDM, ensuring 3D consistency. Another alternative is to first directly model the image sequences and then infer the camera trajectories from the image sequences. Concurrent work, GS-LRM [13], adopts this approach. Specifically, it uses pose-free image sequences generated by SORA, then recovers camera information through COLMAP [89], and finally performs sparse-view 3D reconstruction. However, this approach heavily relies on the 3D consistency of the pose-free image sequences and the accurate estimation of the camera parameters. One can also choose to directly model the joint distribution of camera trajectories and image sequences. However, this requires further exploration of network architectures and camera-differentiable 3DGS operator. We will explore the potential of this approach in future work.

Why Director3D Needs a Detailer? The core reason remains the limitations of multi-view datasets in terms of diversity and quantity, leading to biased and sparse distributions. Although MVImgNet has a large quantity of scenes, it only consists of 238 categories, with the majority being simple objects staged in indoor scenes. While DL3DV-10K exhibits good diversity, it only consists of 10,000 scenes in total. Therefore, even under the 2D data and model priors, it is still challenging for GM-LDM to meet the quality requirements for directly generating realistic 3D scenes with details. This limitation can be alleviated by introducing more diverse and larger multi-view datasets.

Limitations. Although the proposed Director3D possesses the capability to generate real-world 3D scenes with adaptive camera trajectories, it still has some limitations. First, the supported views of our GM-LDM are currently limited, which restricts the range of perspectives in the generated 3D scenes. Further, because the GM-LDM is currently trained with only two real-world datasets, the reliance on an additional refining process for open-world generalization limits the efficiency of our framework, which may be alleviated by introducing a wider variety of datasets [90, 91, 92]. Similar to text-to-image diffusion models, the success rate of our model decreases when generating with complicated and compositional prompts, objects with exact numbers, and articulated objects, as shown in Fig. 10.

Six oranges in a wooden basket.

Broader Impacts. This paper presents a framework whose goal is to advance the field of text-to-3D generation for efficiently generating realistic 3D scene assets. Since it is a 3D generative framework, it has the potential to create harmful 3D content if the user provides harmful text prompts as inputs. This harmful behavior can be avoided by integrating harmful content detection models as used in 2D generative models.

C Diversity and Fine-grained Control

We showcase the diverse generation results with the same prompts in Fig. 11. The results show that Director3D is able to generate diverse camera trajectories and 3D scenes even with a single prompt.

We showcase the body generation results with fine-grained control ( i.e. , gender and clothing) in Fig. 12. Note that there are no categories for human body in the multi-view datasets used for training the GM-LDM. The results show that Director3D is able to finely control the 3D scenes with text modifications.

D More Generation Results

We further provide 40 cases of multi-view image results in Fig. 13 generated from wide-range text prompts to demonstrate the visual quality and generalization ability of our method.

E More Ablation Studies

We provide a qualitative ablation study with more cases in Fig. 14. The corresponding quantitative ablation study is shown in the Tab. 3. Although the ablation with λ x = 0 incurs a slight deterioration in metrics, the qualitative results are noisy in terms of visual quality. These results can further demonstrate the effectiveness of SDS++ loss.

Additionally, we showcase the generation results with randomly generated camera trajectories for some object-level prompts in Fig 15. The results are generated by the GM-LDM without refining. It can be observed that without scene-specific camera trajectories, the generated image sequences are unsatisfying, with weird perspectives or limited camera range. This ablation study further demonstrate the importance of scene-specific camera trajectories for our Director3D.

a rusty old car

a donut with pink icing

photorealistic, 8K, HD

A brown leather suitcase on a

train platform

<latexi sh

1_b

64="0A+XS5

uj

3r

G

Bkg

T

7WFNI

>

H

c

VDL

UZd ov

q

R

Q

O

y

/

C

mY

9E

z

2

p

n

f

8

J

K

w

M

P

<latexi sh

1_b

64="

I

+CQof

m

pq83

L2/

kS

Xd

Y

>A

B

H

c

VD

N

F

W

vg

R

0

O

y

T

J

G

z

n

M

w

5

7

9

K

r

E

P

<latexi sh

1_b

64="/VI

GSZE

q

F

m2

B3

k

pw

>A

n

c

N

8

W

X

D

5CR

U0

o

M

j

O

7Y

g

P

r

f

Q

z

v

d

K

L

J

+

y

u

9

H

T

<latexi sh

1_b

64="0gv

X

yk7

nUj

9

/

r

V

>A

C

c

Z

S

NBFI

3G2

q

zW

Q

MJu

L

R

E

KD

5

O

m

+

o

Y

T

H

8f

w

P

p

d

A vintage clock hanging on a brick wall

A white seashell on a sandy beach

A cozy cabin with smoke rising from the chimney...

A crumbling fortress on a hill, its walls overtaken by nature...

w/o

refining

/epsilon1

src

=

/epsilon1

ω

cfg=1

λ

x

= 0

λ

z

= 0

full

A blue and white china cup on a saucer

A blue denim jacket draped over a wooden chair

+ w/

random traj.

w/o

refining

+ w/

random traj.

w/o

refining

+ w/

random traj.

w/o

refining

A fluffy white pillow resting on a red velvet couch

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim clearly in Abstract and Sec. 1 about contributions and scope made in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Appendix B

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Sec. 5.1 and Appendix A

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All datasets we use are public data, and our code will be open source in the future.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Sec. 5.1 and Appendix A

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Since most of the experiments are visually qualitative evaluations, they cannot be converted into reports with quantitative error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Appendix A

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the NeurIPS Code of Ethics in detail to make sure that our study meets the standards.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All existing assets used in the paper are properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: All urls are anonymized.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Director3D, a text-to-3D generation framework that creates realistic 3D scenes with adaptive camera trajectories. It includes a Cinematographer (Traj-DiT) for generating camera trajectories, a Decorator (GM-LDM) for initial scene generation, and a Detailer (SDS++ loss) for refinement. Using 3D Gaussians for scene representation, extensive experiments demonstrate its effectiveness.\n\nSTRENGTHS:\nThe results are good with high spatial consistency.\n\nThe task is interesting. It is a good idea to generate scene-level 3D GS directly.\n\nThe writing is clear.\n\nWEAKNESSES:\n1. The quantitative comparison with object-level methods seems unfair. The given prompts include sence information, which efftets clip score. In quantitative comparisons, the authors should at least compare with scene-level methods like LucidDreamer or others.\n\n2. Why didn't the authors compare their method with camera control video generation methods?\n\nQUESTIONS:\n1. Is the number of Gaussians equal to the number of pixels in each view? Why set it that way? It seems that 256*256 Gaussians might not be sufficient for a scene-level representation. What is the total number of Gaussians used?\n\n2. Can this pipeline achieve user-specified camera trajectories?\n\n3. Is it possible to generate a complete 3D scene and then reconstruct it? If the camera trajectory encompasses the entire scene, all 3D information is captured. How does the novel view inference ability perform once the three stages are completed? If only can generate denoised views, the contributions will be weakened.\n\nLIMITATIONS:\n1. What about failure cases? In what situations does the method fail to generate a consistent scene?  I find it hard to believe that training on only two not large datasets can achieve such good results. The performance even surpasses object-level methods trained on Objaverse, which doesn't make sense. \n\n2. I want to confirm how the test set is divided. For each class, is a portion of the videos divided as the test set, or are a portion of the classes divided as the test categories? And I want to confirm again that none of the prompts appeared in the training set?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a scene-generation method from text input. The framework utilizes three models that first generate a trajectory, then produce 3D Gaussians, and finally refine through SDS loss.\n\nSTRENGTHS:\n- The design of using a trajectory generator and 3DGS diffusion is novel and impressive. \n- The proposed method outperforms existing object-level methods as shown in experiments.\n\nWEAKNESSES:\n- The paper's evaluation is limited. The methods compared are object-level generation frameworks. The experiments can be improved by comparing them against scene-level generation methods, e.g. LucidDreamer[16] and ZeroNVS[1].\n- I am confused about how SDS++ loss is presented. The paragraph following Eq. (9) contains numerous undefined variables, making it hard to understand how exactly SDS++ loss is formulated. Can I say it is the loss presented in [67] but integrated with learnable text embedding? What's the difference against the loss proposed in VSD[26] and how does it compare?\n- The proposed trajectory generator is novel. How important is generating the trajectory? Can we just assign some trajectories by retrieving the dataset? It seems that most trajectories in MVImgNet and DL3DV are very similar. How diverse are the generations? I see the results in Fig. 12. But aren't the differences coming from different sampling results of a text-based generator? Since the cameras are already normalized, the orientation of the first frame should not be restricted by the camera trajectory. It seems that the model is overfitting to the orientations in MVImgNet (flying around something on the table). Are the \"randomly generated camera trajectories\" coming from other objects? Is it because GM-LDM only works with one trajectory per scene? Does the model support working with multiple trajectories for the same scene?\n- Is the proposed GM-LDM only training the 3DGS scene on observed views? How do the authors avoid overfitting in the rendering-based denoising? Is the SDS++ loss only employed on interpolated camera trajectory?\n- Is the presented video only showing the views presented in the trajectory or they are novel view synthesis results? How do we know the generated 3D scene is not overfitting to these views? Sparse-view 3D reconstruction suffers from over-fitting issues a lot and I'd like to hear the authors' thoughts on this matter.\n\n[1] Sargent K, Li Z, Shah T, et al. Zeronvs: Zero-shot 360-degree view synthesis from a single real image. CVPR 2024, arxiv 2023.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nPlease refer to the weakness. This work shows brilliant visual results. I hope the authors clarify my questions above to help me better understand how the results are obtained.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Director3D, a novel text-to-3D generation framework designed to generate both real-world 3D scenes and adaptive camera trajectories. Specifically, the authors propose the Traj-DiT to generate adaptive camera trajectories, which treats camera parameters as temporal tokens and performs conditional denoising using a transformer model. The authors propose the GM-LDM and SDS++ Loss to generate robust 3D scenes by leveraging the 2D diffusion prior. Extensive experiments demonstrate the effectiveness of Director3D.\n\nSTRENGTHS:\n1. This paper is written clearly and easy to read. \n2. The idea of treating camera parameters as temporal tokens for denoising generation is novel and effective. \n3. The GM-LDM and SDS++ loss functions achieve a fairly high level of realistic scene synthesis. The generated scenarios conform to the textual input as well as being reasonably realistic and consistent. \n4. The proposed Director3D achieves impressive results on both quantitative and qualitative results.\n\nWEAKNESSES:\n1.\tIt would be more convincing to include more text-to-3D scene generation methods in the Quantitative Comparison in the Qualitative Comparison.\n2.\tFor ablation experiments of SDS++ Loss, please use more detailed evaluation metrics and experimental results to illustrate the effects.\n3.\tFurther implementation of conditionally controllable camera view generation would be helpful for the application of this technology. \n4.  Some important work in this area such as 3D-SceneDreamer should be discussed if they are not suitable for the experimental comparison.\n5. The visual quality of the results seems acceptable compared to existing methods like 3D-scene-dreamer, the limitation of the method, including not allowing a large range of the camera movement, should be discussed.\n\nQUESTIONS:\nNo additional questions.\n\nLIMITATIONS:\n1. The baseline methods like Fantasia3D and DreamFusion are mainly for object-level generation, not focusing on scene generation.\n2. The result visualization can be improved, e.g., the imges are quite small for the reviewer to check the visual quality.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework for simultaneous text-to-3D scene and camera trajectory generation. The authors propose a 3-stage pipeline to (1) generate a dense camera trajectory from input text, (2) use multi-view latent diffusion from a sparse subset of the generated trajectory to generate the 3D scene representation (Gaussian splats), and (3) refine the Gaussian splats with a modified SDS loss.\n\nSTRENGTHS:\n- The paper tackles a practical problem of generating a 3D scene representation while also synthesizing a camera trajectory from text. It has implications of potential further applications for video/movie synthesis using explicit 3D representations.\n+ The paper is well-written, the presentation is clear, and the method description is easy to follow and understand.\n\nWEAKNESSES:\n- While this paper tackles a new problem, I have concerns with the problem statement. First, what is a \"real-world\" camera trajectory? Virtually any kind of camera motion could be created in the real world (be it hand-held shakiness or really smooth orbital trajectories that could be achieved via physical equipment). It seems that the camera motions that Traj-DiT could synthesize are mostly orbital (object-centric) -- why would this be considered as a \"real-world\" trajectory?\n- I don't quite get why 3D scene generation and camera trajectory generation should be a coupled problem. NeRFs / Gaussian splats with good quality are not necessarily created via \"camera trajectories\", but rather via a broad range of covered viewpoints.\n- Using a diffusion model to synthesize trajectories, the number of frames would be fixed. How does one determine such trajectory length? How can one vary the length?\n- It is unclear why Cinematographer generates a trajectory with dense frames while only a sparse subset is ever used subsequently.\n- To evaluate the quality of the synthesized camera trajectory, I believe the method should also be evaluated and compared against with a video generation quality metric (e.g. FVD). The 3D scene representation could be a well-trained NeRF / Gaussian splat, and rendered videos under different trajectories could be quantified and compared.\n\nQUESTIONS:\nPlease see the weakness sections.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}