GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs
Gege Gao1,2,3,4 Weiyang Liu1,5,* Anpei Chen2,3,4 Andreas Geiger3,4,† Bernhard Schölkopf1,2,4,†
1
Max Planck Institute for Intelligent Systems – Tübingen 2 ETH Zürich 3 University of Tübingen4Tübingen AI Center 5 University of Cambridge * Directional lead † Shared last author
graphdreamer.github.io

User Text Input

WeatheredStanding in front ofWise-looking

Stack of Spell Books

WizardNext to

Robed
Gazing into

BeardedBall

MysticalUser can also directly input / edit the scene graphGlowing

WizardSitting on

Placed on
Rune-etched

Scene

ObjectDesk

Leather-boundAncient

Attributes:- Wizard: bearded, robed, wise-looking- Crystal Ball: small, glowing, transparent, mystical- Stack of Ancient Spell Books: leather-bound, weathered, rune-etched- Wooden Desk: sturdy, carved, aged

Attribute

CarvedSturdyRelationship

ChatGPT

Edges (Relationships):- Wizard -> standing in front of -> Wooden Desk
- Crystal Ball -> placed on -> Wooden Desk
- Wizard -> gazing into -> Crystal Ball
- Stack of Ancient Spell Books -> sitting on -> Wooden Desk
- Stack of Ancient Spell Books -> next to -> Crystal Ball

Aged

Small

Scene Graph

Desk

Nodes (Objects):- Wizard- Wooden Desk
- Crystal Ball
- Stack of Ancient Spell Books

WoodenBall

User“A Wizard standing in front of a Wooden Desk, gazing into a Crystal
Ball placed on the Wooden Desk, with a Stack of Ancient Spell Books
sitting on the Wooden Desk and next to the crystal ball.”

CrystalTransparent

Books

Please follow the examples in the Visual Genome dataset and generate ascene graph that best describes the following text:

Optimization via score distillation sampling

Compositional 3D Scene

Figure 1. GraphDreamer takes a scene graph as input and generates a compositional 3D scene where each object is fully disentangled. To save the effort ofbuilding a scene graph from scratch, the scene graph can be generated by a language model (e.g., ChatGPT) from a user text input (left box).

Abstract
As pretrained text-to-image diffusion models become increasingly powerful, recent efforts have been made to distillknowledge from these text-to-image pretrained models for
optimizing a text-guided 3D model. Most of the existing
methods generate a holistic 3D model from a plain text
input. This can be problematic when the text describes acomplex scene with multiple objects, because the vectorized
text embeddings are inherently unable to capture a complex
description with multiple entities and relationships. Holistic
3D modeling of the entire scene further prevents accurate
grounding of text entities and concepts. To address this limitation, we propose GraphDreamer, a novel framework to
generate compositional 3D scenes from scene graphs, where
objects are represented as nodes and their interactions asedges. By exploiting node and edge information in scenegraphs, our method makes better use of the pretrained textto-image diffusion model and is able to fully disentangle
different objects without image-level supervision. To facilitate modeling of object-wise relationships, we use signed
distance fields as representation and impose a constraint toavoid inter-penetration of objects. To avoid manual scenegraph creation, we design a text prompt for ChatGPT togenerate scene graphs based on text inputs. We conductboth qualitative and quantitative experiments to validate theeffectiveness of GraphDreamer in generating high-fidelity
compositional 3D scenes with disentangled object entities.

1. Introduction
Recent years have witnessed substantial progresses in text-to3D generation [21, 29, 39], largely due to the rapid development made in text-to-image models [32, 33] and text-image
embeddings [31]. This emerging field has attracted considerable attention due to its significant potential to revolutionize
the way artists and designers work.
The central idea of current text-to-3D pipelines is to leverage knowledge of a large pretrained text-to-image generativemodel to optimize each randomly sampled 2D view of a
3D object such that these views resemble what the input
text describes. The 3D consistency of these 2D views istypically guaranteed by a proper 3D representation (e.g.,neural radiance fields (NeRF) [23] in DreamFusion [29]).
Despite being popular, current text-to-3D pipelines still suffer from attribute confusion and guidance collapse. Attributeconfusion is a fundamental problem caused by text-image
embeddings (e.g., CLIP [31]). For example, models often
fail at distinguishing the difference between “a black cat
on a pink carpet” and “a pink cat on a black carpet”. Thisproblem may prevent current text-to-3D generation methods
from accurately grounding all attributes to corresponding
objects. As the text prompt becomes even more complex, involving multiple objects, attribute confusion becomes moresignificant. Guidance collapse refers to the cases where thetext prompt is (partially) ignored or misinterpreted by themodel. This typically also happens as the text prompt gets
more complex. For example, “a teddy bear pushing a shopping cart and holding baloons”, with “teddy bear” being
ignored. These problems largely limit the practical utility oftext-to-3D generation techniques.
A straightforward solution is to model the multi-object
3D scene in a compositional way. Following this insight,
recent methods [5, 18, 28, 47] condition on additional context information such as 3D layout which provides the sizeand location of each object in the form of non-overlapping
3D bounding boxes. While a non-overlapping 3D layout
can certainly help to produce a compositional 3D scene witheach object present, it injects a strong prior and greatly limits
the diversity of generated scenes. The non-overlapping 3Dbox assumption can easily break when objects are irregular(non-cubic) and obscuring each other. For example, the text
prompt “an astronaut riding a horse” can not be represented
by two non-overlapping bounding boxes. To avoid theselimitations while still achieving object decomposition, we
propose GraphDreamer, which takes a scene graph (e.g.,[13]) as input and generates a compositional 3D scene. Unlike 3D bounding boxes, scene graphs are spatially morerelaxed and can model complex object interaction. While
scene graphs are generally easier to specify than spatial 3D
layouts, we also design a text prompt to query ChatGPT
that enables the automatic generation of a scene graph from
unstructured text. See Figure 1 for an illustrative example.
GraphDreamer is guided by the insight that a scene graphcan be decomposed into a separate and semantically unambiguous text description of every node and edge1 . The
decomposition of a scene graph into multiple textual descriptions makes it possible to distill knowledge from textto-image diffusion models, similar to common text-to-3D
methods. Specifically, to allow each object to be disentangled from the other objects in the scene, we use separate
identity-aware positional encoder networks (i.e., object feature fields) to encode object-level semantic information anda shared Signed Distance Field (SDF) network to decode
the SDF value from identity-aware positional features. The
color value is decoded in a way similar to the SDF value.
Scene-level rendering is performed by integrating objects
based on the smallest SDF value at each sampled point in 3D
space. More importantly, with both SDF and color values ofeach object, we propose an identity-aware object rendering
that, in addition to a global rendering of the entire 3D scene,renders different objects separately. Our local identity-aware
1 Cf. the assumption of independent causal mechanisms [34]rendering allows the gradient from the text-dependent distillation loss (e.g., score distillation sampling [29]) to be
back-propagated selectively to corresponding objects without affecting the other objects. The overall 3D scene will besimultaneously optimized with the global text description tomatch the scene semantics to the global text. In summary,
we make the following contributions:• To the best of our knowledge, GraphDreamer is the first
3D generation method that can synthesize compositional
3D scenes from either scene graphs or unstructured text
descriptions. No 3D bounding boxes are required as input.
• GraphDreamer uses scene graphs to construct a disentangled representation where each object is optimized via its
related text description, avoiding object-level ambiguity.
• GraphDreamer is able to produce high-fidelity complex 3D
scenes with disentangled objects, outperforming both stateof-the-art text-to-3D methods and existing 3D-boundingbox-based compositional text-to-3D methods.
• In Appendix C, we envision a new paradigm of semantic
3D reconstruction – Inverse Semantics, where a visionlanguage model (e.g., GPT4-V) is used to extract a scenegraph from an input image (i.e., scene graph encoder) and
GraphDreamer is used to generate a compositional 3D
scene from the scene graph (i.e., scene graph decoder).

2. Related Work
Text-to-2D generation. Driven by large-scale image-textaligned datasets [35], text-to-image generation models [1, 32,
33] have made great progress in producing highly realistic
images. Among these models, generative diffusion models
learn to gradually transform a noisy latent z with noise ϵtypically from a Gaussian distribution, towards image data xthat reproduce the semantics of a given text prompt y. This
generative process slowly adds structure to the noise, basedon a weighted denoising score matching objective [12, 27].
2D-lifting for 3D generation. In contrast to existing textto-image generation models, text-guided 3D generative models [10, 21, 22, 24, 26, 29, 38, 39] usually optimize a 3Dmodel by guiding its randomly rendered 2D view based on
the pretraining knowledge of some text-to-image generation model, because of the shortage of text-3D paired assets.
DreamFusion [29] and subsequent work [4, 17, 37, 42, 44]
propose to optimize the 3D model by distilling a pretrained
diffusion model [32, 33] via score distillation sampling [39].
Generate objects with SDF. In text-to-3D generation, recent works [10, 17, 19, 21, 29, 37, 39, 40, 42] parameterized
the 3D scene as a NeRF [2, 23] or a hybrid pipeline combining NeRFs with a mesh refiner [11, 17, 36, 38]. In our
approach, we use a signed distance field (SDF) as the geometry representation instead of NeRF densities, as we aim atmodeling multi-object scenes in a compositional way, where
objects may be coupled in various ways. SDF provides cru-

cial inside/outside information, allowing for geometry constraints to prevent unexpected intersections between objects,
and is ideal for complex scenes as it facilitates customizationof initial locations and scales of object SDFs.
Hybrid 3D representation for disentanglement. Anotherline of work uses hybrid representations to learn disentangled 3D objects [15, 20, 43]. The works [6, 7] put forwarda hybrid approach that represents the face/body as meshes
and the hair/clothing as NeRFs, enabling a disentangled reconstruction of avatars. [48] adopts this representation andproposes a text-to-3D method that generates compositional
head avatars. However, the use of a parametric head model
limits this method to human head generation. Their disentanglement only applies to two objects (e.g., face and hair),and in contrast, ours can be used for multiple objects.

3. Preliminaries
Score distillation sampling (SDS). SDS [29] is a techniquethat optimizes a 3D model by distilling a pretrained text-toimage diffusion model. Given a noisy image zt renderedfrom a 3D model parameterized by Θ, and a pretrained textto-image diffusion model with a learned noise prediction
network ϵϕ (·), SDS uses ϵϕ (·) as a score function that predicts the sampled noise ϵ contained in xt at noise level t asϵ̂ϕ (y, t) = ϵϕ (zt ; y, t), where y is a given conditional text
embedding. The score is then used to warp the noisy zt
towards real image distributions, by guiding the direction of
the gradients that update the parameters Θ of the 3D model:
  \label {eq:sds} \nabla _\Theta \mathcal {L}(z; y) = \mathbb {E}_{t,\epsilon }\biggl [w(t)\Bigl (\hat {\epsilon }_\phi (y,t) - \epsilon \Bigr )\frac {\partial z}{\partial \Theta }\biggr ] (1)

where w(t) is a weighting function that depends on t and ϵis the sampled isotropic Gaussian noise, ϵ ∼ N (0, I).
SDF volume rendering. To render a pixel color of a target camera, we cast a ray r from the camera center oalong its viewing direction d, then sample a series of points
p = o+td in between the near and far intervals [tn , tf ]. Following NeRF [23], the ray color C(r) can be approximatedby integrating the point samples,  \label {eq:volume_rendering} C(\br ) = \sum _{i=1}^{N} w_i c_i = \sum _{i=1}^{N} T_i \alpha _i c_i (2)where wi is the color weighting function, Ti represents
the cumulative transmittance, which is calculated as Ti =Q
i−1
j=1 (1 − αj ), and αi denotes the piece-wise constant opacity value for the i-th sub-interval, with αi ∈ [0, 1].
Unlike NeRFs that directly predict the density for a given
position p, methods based on implicit surface representation
learn to map p to a signed distance value with a trainablenetwork and extract the density [46] or opacity [41] fromthe SDF with a deterministic transformation. We extract theopacity following NeuS [41]. NeuS formulates the transformation function based on an unbiased weighting function,
which ensures that the pixel color is dominated by the intersection point of camera ray with the zero-level set of SDF,

  \alpha _i = \text {max} \left ( \frac {\Phi _\beta (u_i) - \Phi _\beta (u_{i+1})}{\Phi _\beta (u_i)} , 0\right ) \label {eq:neus} (3)where Φβ (·) is the Sigmoid function with a trainable steepness β, and ui is the SDF value of the sampled position.

4. Method
Consider generating a scene of M objects, O = {oi }M
i=1
from a global text prompt y g . When the scene is complex
or has many attributes and inter-object relationships to specify, y g will become very long, and the generation will beaccompanied by guidance collapse [3, 16]. We thus propose
to first generate a scene graph G(O) from y g following thesetting of [13], which precisely describes object attributes
and inter-object relationships. We provide an example of afour-object scene in Figure 1 for better illustration.

4.1. Leveraging Scene Graphs for Text Grounding
Given user text input y g , objects {oi }M
i=1 in the text (which
can be detected either manually or automatically, e.g., using
ChatGPT2 ) form the nodes in graph G(O), as shown in
Figure 1. To provide more details to an object oi , the user
can add additional descriptions, such as “Wise-looking” and“Leather-bound”, which become the attributes attached tooi in G(O). Combining oi with all its attributes simply bycommas, we get an object prompt y (i) for oi that can beprocessed by text encoders.
The relationship between each pair of objects oi and oj istransformed into edge ei,j in G(O). For instance, the edge
between node “Wizard” and “Desk” is “Standing in front of”.
For a graph with M nodes, there are possibly C2M edges.
By combining oi , ei,j , and oj , we obtain edge prompt y (i,j)
that exactly defines the pairwise relationship, e.g., “Wizard
standing in front of Wooden Desk”. Note that there might
be no edge between two nodes, e.g., between “Wizard” and“Stack of Ancient Spell Books”. We denote the number ofexisting edges in G(O) as K, with K ≤ C2M .
From this example, we also see that using graph G(O)is a better way to customize a scene compared to a puretext description y g , in terms of both flexibility in attaching
attributes to objects and accuracy in defining relationships.
By processing the input scene graph,
we now obtain a set of(1 + M + K) prompts Y G(O) as:  \label {eq:graph_prompts} \mathcal {Y}\bigl (\mathcal {G}(\mathcal {O})\bigr )=\bigl \{y^g, y^{(i)}, y^{(i,j)} \mid o_i \in \mathcal {O}, e_{i,j} \in \mathcal {G}(\mathcal {O}) \bigr \}  (4)which are used to guide scene generation from the perspective of both individual objects and pairwise relationships.
2 ChatGPT4, https://chat.openai.com

f (1)(p)...

f (2)(p)...

f (3)(p)...

A position p
in 3D space...

pf (4)(p)Object fields

SDF decoder

Aged

SturdyAncient

Sitting on

Color decoder

Volumerendering

Local node-wise rendering

Desk

Wise-looking

Standing in front of

WizardNext toGlowing

SDS Loss

Node text description

Standing in front of

Stack of Spell Books

Robed
Gazing into

BeardedBall

Mystical

Robed

CarvedPlaced on
Rune-etched

WizardDesk

Leather-boundWeatheredWise-looking

BeardedSDF-based representation
WoodenIdentity-aware
volume rendering

Small

CrystalTransparent

Global text description

SDS Loss

SDS Loss

WizardRay r

Global rendering

Edge text description

Local edge-wise rendering

Figure 2. The overall pipeline of GraphDreamer. Specifically, GraphDreamer first decomposes the scene graph into global, node-wise and edge-wise text
description, and then optimizes the SDF-based objects in the 3D scene using their corresponding text description.

GraphDreamer consists of three learnable modules: apositional feature encoder Fθ (·), a signed distance network
uϕ1 (·), and a radiance network cϕ2 (·). The entire modelis parameterized by Θ = {θ, ϕ1 , ϕ2 }. There are two goals
in optimizing GraphDreamer: (i) to model the complete
geometry and appearance of each object, and (ii) to ensurethat object attributes and interrelationships are in accordancewith the scene graph G(O). The overall training process is
illustrated in Figure 2.

4.2. Disentangled Modeling of Objects
Positional encodings are useful for networks to identify thelocation it is currently processing. To achieve the first goalof making objects separable, we need to additionally identify
which object a position belongs to. Therefore, instead of
one positional feature embedding, we encode a position p
into multiple feature embeddings by introducing a set of
positional hash feature encoders, each parameterized by θi ,corresponding to the number of objects,
  \mathcal {F}_{\theta }(\cdot )=\{\mathcal {F}_{\theta _i}(\cdot )\}_{i=1}^M ~~~~~ \theta =\{\theta _i\}_{i=1}^M (5)These feature encoders then form different object fields, i.e.,
one field per object across the same scene space Ω ⊂ R3 .
Individualized object fields. Given a position p ∈ Ω, thefeature that forms the field of object oi is obtained as:
  f^{(i)}(\bp ) = \mathcal {F}_{\theta _i}(\bp ) \in \mathbb {R}^F ~~~~ i \in \{1, \cdots , M\} (6)

where F is the number of feature dimensions, the same for allFθi (·). Here, for each Fθi (·) we adopt the multi-resolutionhash grid encoding from Instant NGP [25] following [17, 37,42] to reduce computational cost.
These identity-aware feature embeddings are then passed
to the shared shallow MLPs for SDF and color prediction,e.g., the SDF u(i) (p) ∈ R and color c(i) (p) ∈ R3 values forobject oi ’s field are predicted as:  u^{(i)}(\bp ) = u_{\phi _1}\bigl (f^{(i)}(\bp )\bigr ) ~~~~~~ c^{(i)}(\bp ) = c_{\phi _2}\bigl (f^{(i)}(\bp )\bigr ) (7)

where u(i) (p) indicates the signed distance value from position p to the closest surface of object oi , with negativevalues inside oi and positive values outside, and c(i) (p) thecolor value in oi ’s field where only object oi is considered.
Here, we follow prior work [45] to initialize the SDF approximately as a sphere. We transform u(i) (p) into opacity with
Eq. (3) as γ (i) (p) for the volume rendering of object fields.
In scenes where mutual-obscuring relationships are involved, to generate the complete geometry and appearance
for each object, we need to make hidden object surfaces visible. Therefore, the scene needs to be properly decomposed
before rendering the objects.
Scene space decomposition. Intuitively, a position p ∈ Ω
can be occupied by at most one object. Since the SDF determines the boundary of an object, p can thus be identified
as belonging to object oi if its SDF values {u(i) }M
i=1 areminimized at index i. Based on this, we define an one-hot
identity (column) vector λ(p) for each position p as:
  \label {eq:soft_identity} \boldsymbol {\lambda }(\bp ) = \argmax _{i=1, \cdots , M}\bigl \{ -u^{(i)}(\bp )\bigr \} \in \{0, 1\}^M (8)

Based on λ(p), we can decompose the scene into identityaware sub-spaces and render each object individually withall other objects removed.
Identity-aware object rendering. To render object oi ,given a position p, we multiply its opacity γ (i) in oi ’s fieldwith λ(i) (p) to obtain the opacity for only object oi as:  \label {eq:semantic_opacity} \alpha ^{(i)}(\bp ) = \lambda ^{(i)}(\bp ) \cdot \gamma ^{(i)} \in [0, +\infty ) (9)where λ(i) (p) is the i-th element in vector λ(p). If
λ(i) (p) = 1, which means p is identified as most likely
to be occupied by object oi , we have opacity α(i) (p) ≥ 0 in
object o’s field only, while in all other object fields p will be
empty. Based on this identity-aware opacity α(i) (p), we canobtain the ray color with object oi present only as:  \label {eq:render} C_{\br }^{(i)} = \sum _\bp \alpha ^{(i)}(\bp )T^{(i)}(\bp )\cdot c^{(i)}(\bp ) (10)

where the cumulative transmittance T (i) (pj ) is defined following Eq. (2). By aggregating all rendered pixels, an object
image C (i) is obtained, which contains object oi only. With
C (i) and the object prompt y (i) from the scene graph G(O),we can thus define the object SDS loss following Eq. (1) as:  \label {eq:obj_sds} \nabla _\Theta \mathcal {L}^{(i)}\bigl (C^{(i)}; y^{(i)}\bigr ) ~~~~ o_i \in \mathcal {O} (11)

to supervise GraphDreamer at the object level.

4.3. Pairwise Modeling of Object Relationships
By building up a set of identity-aware object fields, we are
now able to render objects in G(O) individually to matchy (i) . To make two related objects oi and oj respect therelationship defined in edge prompt y (i,j) , we need to renderoi and oj jointly. Therefore, a combination of two object
fields is required.
Edge rendering. Given an edge ei,j connecting nodes oi
and oj , we can accumulate an edge-wise opacity α(i,j) (p)at position p from opacity values of oi and oj based on the
one-hot identity vector λ(p) as:  \alpha ^{(i,j)}(\bp ) = \sum _{k\in \{i,j\}} \lambda ^{(k)}(\bp ) \cdot \gamma ^{(k)}(\bp ) (12)

and similarly, an edge-wise color c(i,j) (p) at position p as:
  c^{(i,j)}(\bp ) = \sum _{k\in \{i,j\}} \lambda ^{(k)}(\bp ) \cdot c^{(k)}(\bp ) (13)

from which we can render a ray color across two object
fields of oi and oj following the same integration process as
Eq. (2), and obtain an edge image C (i,j) in with both objects
involved. We can thus define an edge SDS loss as:
  \label {eq:edge_sds} \nabla _\Theta \mathcal {L}^{(i,j)}\bigl (C^{(i,j)}; y^{(i,j)}\bigr ) ~~~~ e_{i,j} \in \mathcal {G}(\mathcal {O}) (14)

to match the edge prompt y (i,j) .
Scene rendering. To provide global scene graph G(O) constraints, we further render the whole scene globally by combining all object fields together. Similarly as for the edge
rendering, we use λ(p) to accumulated the global opacity
αg (p) and a global color c(p) at position p over all object
fields as:
  \label {eq:global_opacity} \alpha ^g(\bp ) = \boldsymbol {\lambda }^T(\bp ) \cdot \boldsymbol {\gamma }(\bp )~~~~~~c^g(\bp ) = \boldsymbol {\lambda }^T(\bp ) \cdot \boldsymbol {c}(\bp ) (15)

with γ(p) and c(p) denote the column vectors of opacity
and color values at p over all object fields. Through the
integration process, we can render the entire scene into ascene image C g that represents the entire scene graph. Wedefine a scene-level SDS loss, ∇Θ Lg C g ; y g , to globally
optimize GraphDreamer to match the scene prompt y g .
Efficient SDS guidance. So far, a number of (M + K + 1)
SDS losses are introduced, corresponding to the number ofprompts we obtained in Eq. (4). However, optimizing all
constraints jointly is intractable. Instead, in each training
step, we include two SDS losses only: (i) an object SDS
loss Eq. (11) for only one object oi , with each step choosing
a different object oi looping through O; (ii) an edge SDS
loss Eq. (14) of one ei,j connected to oi , looping throughall existing edges connected to oi . The scene SDS loss is
included only in the training step after each traversal of O,
and is used solely in that step without any other SDS loss.
Thus, the total SDS loss for optimizing GraphDreamer is:  \label {eq:total_sds} \mathcal {L}_{SDS} = \begin {cases} \nabla _\Theta \mathcal {L}^{(i)} + \nabla _\Theta \mathcal {L}^e, & i = s\text { \% } (M+1) \\ \nabla _\Theta \mathcal {L}^g, & \text {others} \end {cases} (16)

where s indexes the current training step, and e refers to oneof the edges connected with oi .

4.4. Training Objectives
Apart from the SDS guidance, to further optimize the prediction of unobserved positions, i.e., inside objects and on
hidden surfaces at object intersections, we include two geometry constraints for physically plausible shape completion.
Penetration constraint. Since each point p ∈ Ω can beinside or on the surface of at most one object o in the setof objects O, and for regions outside the actual objects,
u(p) ∈ (0, +∞) with p ∈ Ω \ O, we define a penetration
measurement at point p:

  \mathcal {N}^-(\bp ) = \sum _{i=1}^M \max \Bigl \{ \text {sgn}\bigl (-u^{(i)}(\bp )\bigr ), 0 \Bigr \} (17)where sgn(x) is the sign function. Intuitively, N − (p) measures the number of objects inside which the point p is
located, according to the predicted u(i) (p). Using this measurement, we propose to implement a penetration constraint,
  \label {eq:penet_loss} \mathcal {L}_{penet}(\bp ) = \max \bigl \{0, \mathcal {N}^-(\bp ) - 1\bigr \} (18)

to constrains the penetration number N − (p) not to exceed
1, which is averaged over all sampled points during training.
Eikonal constraint. At each sampled position p, we adopt
the Eikonal loss [8] on SDF values u(i) (p) from all object
fields, formulated as

  \mathcal {L}_{eknl} = \frac {1}{NM} \sum _{i, p}\biggl ( \Big \| \nabla u^{(i)}(\bp ) \Big \|_2 - 1 \biggr ) where N is the size of the sample set Pr on ray r.

(19)

Magic3D

MVDreamGraphDreamer(w/o graph)

GraphDreamer

Two macaw
parrots
SharingWith two
straws

A milkshake"two macaw parrots sharing amilkshake with two straws"

An octopus
Playing
The piano"an octopus playing the piano"“a Wizard standing in front of a Wooden Desk,gazing into a Crystal Ball perched atop the
Wooden Desk, with a Stack of Ancient Spell
Books perched atop the Wooden Desk”(Scene Graph can be found in Figure 1)

Pushing

A teddy bear

PushingWith right hand

Holding
In left hand

A shopping cart

A handful ofballoons

Full ofvegetables

Floating

Colorful“a Teddy Bear pushing a Shopping Cart fullof vegetables with right hand and holding a
Handful Of Balloons in left hand”

A hippoBitingthroughLarge

A watermelon

Juicy"a hippo biting through a watermelon"

A badger

Wearing aparty hat

Eating
Decorated
With birthday
candles

A cakeLayered“a badger wearing a party hat and blowing
out birthday candles on a cake”

CrackedTwo halves

A chick

FluffyStanding next toYellow

An eggshell
Broken in two

Newborn“a DSLR photo of an Eggshell broken in two
with an adorable Chick standing next to it”

A corgi
RidingA rocket“a corgi riding a rocket”

Figure 3. Qualitative comparison with baseline approaches and the ablated configuration (w/o graph). GraphDreamer generates scenes with all composing
objects being separable. Moreover, with accurate guidance from scene graphs, object attributes and inter-object relationships produced by GraphDreamer
match the given prompts better. We recommend to zoom-in for details.

CLIP
GraphDreamer
Magic3D [17] MVDream [37]GraphDreamer
Score(w/o graph)
mean
std.

0.3267
0.03620.31020.00610.30190.0254

CLIP Score

0.33260.0252GraphDreamer

Table 1. Quantitative results. The mean and standard deviation (std.) values
are summarized from CLIP scores of 30 multi-object scenes, with thenumber of objects in the scene ≥ 2. For better comparison, we provide theresult of GraphDreamer (w/o graph), the configuration with the scene graph
G(O) dropped from GraphDreamer, and thus the conditioning text for all
renderings collapse to a single y g .

where {β1 , β2 , β3 } are hyperparameters.

0.201

0.009

w. Self Prompt
w. Other Prompts

0.30
0.25
0.20

0.10

Implementation details. We adopt a two-stage coarse-tofine optimization strategy following previous work [4, 17,29, 37]. In the first stage of 10K denoising steps, we render
images of 64 × 64 resolution only for faster convergence anduse DeepFloyd-IF3 [14, 33] as our guidance model, which isalso trained to generate 64px images. In the second stage of
10K steps, the model is refined by rendering 256px images
and uses Stable Diffusion4 [32] as guidance. Both stages of
optimization use 1 Nvidia Quadro RTX 6000 GPU; GraphDreamer uses 16.88/18.58/20.05 GB for generating 2/3/4
objects, while Magic3D/MVDream uses 11.44/20.33 GB.
Baseline approaches. We report results of two state-of-theart approaches, Magic3D [17] and MVDream [37]. Bothapproaches use a frozen guidance model without additional
learnable module [42] and do not have special initialization
requirements [4] for geometry. We use the same guidancemodels and strategy to train Magic3D, while for MVDream,
since it proposes to use a fine-tuned multi-view diffusion
model5 , we follow its official training protocol and use its
released diffusion model as guidance. The experimental
results of GraphDreamer and the baselines are all obtained
after training for 20K steps in total.
Evaluation criteria. We report the CLIP Score [30] in quantitative comparison with baseline models and evaluation on
object decomposition. The metric is defined as:(21)

which measures the similarity between a prompt text y for
an image C and the actual content of the image, with EC (C)the visual CLIP embedding and EY (y) the textual CLIP
embedding, both encoded by the same CLIP model6 .
3 DeepFloyd-IF, huggingface.co/DeepFloyd/IF-I-XL-v1.0
4 Stable diffusion, huggingface.co/stabilityai/stable-diffusion-2-1-base
5 MVDream-sd-v2.1-base-4view, huggingface.co/MVDream/MVDream
6 CLIP B/32, huggingface.co/openai/clip-vit-base-patch32

std.

0.012

0.15

5. Experiments and Results

  \text {CLIPScore}(C, y) = \cos {\bigl <E_C(C), E_Y(y)\bigr >} 

mean

0.308

0.35

CLIP Score(20)

w. Other Prompts ↓std.

Table 2. The CLIP scores of individual object images C (i) . Metric w. Self
Prompt refers to scores calculated between C (i) and its own prompt y (i) ,and w. Other Prompts between C (i) and prompts of all other objects inthe same scene {y (j) , j ̸= i, oj ∈ O}. Detailed experimental settings andanalysis on these figures as well as the on the chart showing in Figure 4,
can be found in Subsection 5.3.

Total training loss. Our final loss function for training
GraphDreamer thus consists of four terms:
  \label {eq:total_loss} \mathcal {L}_{\Theta } = \beta _1 \mathcal {L}_{SDS} + \beta _2 \mathcal {L}_{penet} + \beta _3 \mathcal {L}_{eknl} w. Self Prompt ↑meanmean ± std.
mean ± 2 std.
mean ± 3 std.

Object(s)Figure 4. Error bands of object CLIP scores.

5.1. Comparison to Other Methods
We report quantitative results in Table 1 and qualitative results in Figure 3. The figures in Table 1 are summarizedfrom CLIP scores of 30 multi-object scenes, with the number of objects ≥ 2. GraphDreamer (full model) achieves thehighest CLIP score. Qualitative results shown in Figure 3
also suggest that GraphDreamer is generally more applicable in producing multiple objects in a scene with various
inter-object relationships. As can be observed from baseline
results, semantic mismatches are more commonly found,such as, in the example of “two macaw parrots sharing amilkshake with two straws”, Magic3D generates two “milkshakes” and MVDream produces one “parrot” only, which
mixed up the number attribute of “milkshake” and “parrots”,
and in the example of ”a hippo biting through a watermelon”,
two objects “hippo” and “watermelon” are blended into one
object; GraphDreamer, on the other hand, models both individual attributes and inter-object relationships correctlybased on the guidance of the input scene graph.

5.2. Ablation Study
To evaluate the effect of introducing scene graph G(O) as
guidance on preventing guidance collapse, we consider todrop G(O) from GraphDreamer, termed w/o graph, andcompare the results in Table 1 and Figure 3. GraphDreamer(w/o graph) collapses the conditioning text from a set ofprompts defined in Eq. (4) to only the global prompt y g andthus involves no object/edge rendering (described in Section 4.2 and 4.3) in training; LSDS from Eq. (16) becomes∇Θ Lg , with all other implementation details unchanged. As
reported in the fourth column of Table 1, the mean CLIP
score of the ablated configuration decreases by more than 3

A baby bunnyAn
astronautSitting on top of
RidingA stack ofpancakes

A horseBrown“an astronaut riding
a brown horse”“a baby bunny sitting on
top of a stack of pancakes”

A blue jay
A babydragonStanding on
Hatching
out ofA large basket of
rainbow macarons
A stone egg“a blue jay standing on a large
basket of rainbow macarons”“a baby dragon hatchingout of a stone egg”

Shimmering tail

Ethereal

Mermaid

Guarding

SittingonTreasure
chest

Coral
throneAncient
Barnacleencrusted

Long
flowing hair

A gummy bear

Intricate

Playing

Colorful

The saxophone

Living coral

Overflowing
with coins

Located next to“a gummy bear playing
the saxophone”“a fantasy scenario of a Mermaid sitting
on a Coral Throne, with a Treasure
Chest located next to the Coral Throne”

Figure 5. More qualitative compositional examples from GraphDreamer.

std. compared to the full model, indicating that the abilityof this configuration to generate 3D scenes that match given
prompts is significantly reduced. The results in Figure 3 also
corroborate such a decline, given that the problems such as
missing objects and attribute confusion arise again. Bothevaluations suggest the need for the scene graph G(O) in
GraphDreamer for combating guidance collapse problems.

5.3. Decomposition Analysis
We consider scenes with the number of objects ≥ 2. To
further quantitatively evaluate whether objects in a scene are
well separated and rendered into object images individually,
we calculate two CLIP metrics for each object image C (i) :(i) w. Self Prompt (abbr., wSP) refers to the CLIP score between C (i) and its own object prompt y (i) , and (ii) w. Other
Prompts (abbr., wOP) refers to the CLIP scores between
C (i) and all other object prompts {y (j) , j ̸= i, oj ∈ O} inthe same scene. Intuitively, if a scene is well decomposed,
each object image C (i) should contain one object oi only
without any part of other objects, and thus the scores wSP
should be much higher than the scores wOP. Table 2 reports a statistical summary on the metrics of 64 objects, for(i)each object, we render images from 4 orthogonal views Cv(v = 1, 2, 3, 4), and thus we get 4 wSP scores and multiple

wOP scores per object. We calculate the mean and standard
deviation (std.) of these wSP and wOP scores separately
over the view images. The figures reported in the table are
averaged over all 64 objects. The mean and std. values arealso presented in an error band graph in Figure 4, where
the x-axis is the index of objects. From this graph it can beobserved more obviously that the wSP CLIP score is significantly higher than the wOP CLIP score, without overlap
between the mean wSP scores and the mean ± 3 std. error
band of the wOP scores, which shows that the scenes areproperly decomposed into individual objects. More compositional examples can be found in Figure 5.

6. Concluding Remarks
This paper proposes GraphDreamer, which generates compositional 3D scenes from scene graphs or text (by leveraging
GPT4-V to produce a scene graph from the text). GraphDreamer first decomposes the scene graph into global, nodewise, and edge-wise text descriptions and then optimizes the
SDF-based objects with the SDS loss from their corresponding text descriptions. We conducted extensive experiments
to show that GraphDreamer is able to prevent attribute confusion and guidance collapse, generating disentangled objects.

Acknowledgment
The authors extend their thanks to Zehao Yu and Stefano Esposito for their invaluable feedback on the initial draft. Our
thanks also go to Yao Feng, Zhen Liu, Zeju Qiu, YandongWen, and Yuliang Xiu for proofreading and for their insightful suggestions on the final draft which enhanced the qualityof this paper. In addition, we appreciate the assistance of
those who participated in our user study. Weiyang Liu andBernhard Schölkopf was supported by the German Federal
Ministry of Education and Research (BMBF): Tübingen AI
Center, FKZ: 01IS18039B, and by the Machine Learning
Cluster of Excellence, the German Research Foundation(DFG): SFB 1233, Robust Vision: Inference Principles andNeural Mechanisms, TP XX, project number: 276693517.
Andreas Geiger and Anpei Chen were supported by the ERC
Starting Grant LEGO-3D (850533) and the DFG EXC number 2064/1 - project number 390727645.

References[1] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat,
Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala,
Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, andMing-Yu Liu. ediff-i: Text-to-image diffusion models withan ensemble of expert denoisers, 2023.
[2] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter
Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.
Mip-nerf: A multiscale representation for anti-aliasing neuralradiance fields. In Proc. of the IEEE International Conf. on
Computer Vision (ICCV), 2021.
[3] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel
Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Trans. on
Graphics, 2023.
[4] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3D: Disentangling geometry and appearance for highquality text-to-3D content creation. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2023.
[5] Helisa Dhamo, Fabian Manhardt, Nassir Navab, and Federico
Tombari. Graph-to-3d: End-to-end generation and manipulation of 3d scenes using scene graphs. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2021.
[6] Yao Feng, Jinlong Yang, Marc Pollefeys, Michael J Black,and Timo Bolkart. Capturing and animation of body andclothing from monocular video. In SIGGRAPH Asia, 2022.
[7] Yao Feng, Weiyang Liu, Timo Bolkart, Jinlong Yang, Marc
Pollefeys, and Michael J Black. Learning disentangled avatars
with hybrid 3D representations. arXiv.org, 2309.06441, 2023.
[8] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, andYaron Lipman. Implicit geometric regularization for learning shapes. In Proc. of the International Conf. on Machinelearning (ICML), 2020.
[9] Jonathan Ho and Tim Salimans. Classifier-free diffusion
guidance. In Advances in Neural Information Processing
Systems (NIPS), 2021.
[10] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel,and Ben Poole. Zero-shot text-guided object generation withdream fields. Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2022.
[11] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel,and Ben Poole. Zero-shot text-guided object generation withdream fields, 2022.
[12] Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan
Ho. Variational diffusion models. arXiv.org, 2107.00630,
2023.
[13] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson,
Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, andFei-Fei Li. Visual genome: Connecting language and visionusing crowdsourced dense image annotations. International
Journal of Computer Vision (IJCV), 2017.
[14] DeepFloyd Lab. DeepFloyd, 2023.
[15] Gengyan Li, Abhimitra Meka, Franziska Mueller, Marcel C
Buehler, Otmar Hilliges, and Thabo Beeler. Eyenerf: a hybrid representation for photorealistic synthesis, animation andrelighting of human eyes. ACM Trans. on Graphics, 2022.
[16] Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva.
Divide & bind your attention for improved generative semantic nursing. arXiv.org, 2307.10864, 2023.
[17] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa,
Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, MingYu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to3d content creation. Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2023.
[18] Yiqi Lin, Haotian Bai, Sijia Li, Haonan Lu, Xiaodong Lin,
Hui Xiong, and Lin Wang. Componerf: Text-guided multiobject compositional nerf with editable 3D scene layout.
arXiv.org, 2303.13843, 2023.
[19] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov,
Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot
one image to 3d object. In ICCV, 2023.
[20] Weiyang Liu, Zhen Liu, Liam Paull, Adrian Weller, and Bernhard Schölkopf. Structural causal 3d reconstruction. In Proc.
of the European Conf. on Computer Vision (ECCV), 2022.
[21] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, andDaniel Cohen-Or. Latent-nerf for shape-guided generation of
3D shapes and textures. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2023.
[22] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, andRana Hanocka. Text2mesh: Text-driven neural stylization formeshes. Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2022.
[23] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik,
Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF:
Representing scenes as neural radiance fields for view synthesis. In Proc. of the European Conf. on Computer Vision(ECCV), 2020.
[24] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky,
and Tiberiu Popa. Clip-mesh: Generating textured meshesfrom text using pretrained image-text models. In SIGGRAPH
Asia, 2022.
[25] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. on Graphics, 2022.

[26] Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto
Tono, Linqi Zhou, and Paul Guerrero. 3d-ldm: Neural implicit
3d shape generation with latent diffusion models, 2022.
[27] Alex Nichol and Prafulla Dhariwal. Improved denoisingdiffusion probabilistic models. arXiv.org, 2021.
[28] Ryan Po and Gordon Wetzstein. Compositional 3D scene
generation using locally conditioned diffusion. arXiv.org,
2303.12218, 2023.
[29] Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv.org,
2209.14988, 2022.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learningtransferable visual models from natural language supervision.
arXiv.org, 2103.00020, 2021.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learningtransferable visual models from natural language supervision. In Proc. of the International Conf. on Machine learning(ICML), 2021.
[32] Robin Rombach, Andreas Blattmann, Dominik Lorenz,
Patrick Esser, and Björn Ommer. High-resolution image
synthesis with latent diffusion models. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2022.
[33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay
Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour,
Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes,
Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad
Norouzi. Photorealistic text-to-image diffusion models withdeep language understanding. In Advances in Neural Information Processing Systems (NIPS), 2022.
[34] B. Schölkopf*, F. Locatello*, S. Bauer, R. Nan Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Towards causal representation learning. Proceedings of the IEEE, 2021.
[35] Christoph Schuhmann, Romain Beaumont, Richard Vencu,
Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes,
Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick
Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion5b: An open large-scale dataset for training next generation
image-text models. In Advances in Neural Information Processing Systems (NIPS), 2022.
[36] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, andSanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In Advances in
Neural Information Processing Systems (NIPS), 2021.
[37] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li,
and Xiao Yang. Mvdream: Multi-view diffusion for 3D
generation. arXiv.org, 2308.16512, 2023.
[38] Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni,
Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3D meshes from text prompts, 2023.
[39] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh,and Greg Shakhnarovich. Score jacobian chaining: Lifting
pretrained 2d diffusion models for 3D generation. In Proc.

IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2023.
[40] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh,and Greg Shakhnarovich. Score jacobian chaining: Lifting
pretrained 2d diffusion models for 3d generation. In CVPR,
2023.
[41] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku
Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Advances in Neural Information Processing Systems(NeurIPS), 2021.
[42] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan
Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity anddiverse text-to-3D generation with variational score distillation. arXiv.org, 2305.16213, 2023.
[43] Jiamin Xu, Zihan Zhu, Hujun Bao, and Weiwei Xu. Hybrid
mesh-neural representation for 3D transparent object reconstruction. arXiv.org, 2203.12613, 2022.
[44] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying
Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shottext-to-3d synthesis using 3d shape prior and text-to-image
diffusion models. In CVPR, 2023.
[45] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, MatanAtzmon, Basri Ronen, and Yaron Lipman. Multiview neuralsurface reconstruction by disentangling geometry and appearance. In Advances in Neural Information Processing Systems(NIPS), 2020.
[46] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Advances in
Neural Information Processing Systems (NeurIPS), 2021.
[47] Guangyao Zhai, Evin Pınar Örnek, Shun-Cheng Wu, Yan Di,
Federico Tombari, Nassir Navab, and Benjamin Busam. Commonscenes: Generating commonsense 3d indoor scenes withscene graphs. In Advances in Neural Information Processing
Systems (NIPS), 2023.
[48] Hao Zhang, Yao Feng, Peter Kulits, Yandong Wen, Justus
Thies, and Michael J Black. Text-guided generation andediting of compositional 3D avatars. arXiv.org, 2309.07125,
2023.

Appendix
Table of Contents
A. Experimental Settings
A.1. Additional Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2. Hyperparameter Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12
12
12

B. Additional Experiments and Results
B.1 . Ablation: Penetration Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 . User Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 . Extended Comparison to Other Text-to-3D Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13
13
13
15

C. Inverse Semantics: A Paradigm of Semantic Reconstruction with GPT4V-guided GraphDreamer

16

D. Failure Cases and Limitations

17

A. Experimental Settings
A.1. Additional Implementation Details
Point-wise identity vector λ(p). As defined in Eq. (8), identity vector λ(p) at a given position p is defined as an one-hot
vector for a better disentanglement in the forward rendering process. To combine λ(p) into differentiate training, we customize
the gradient of λ(p) in back-propagation by combining with a Softmax operation:
  \label {eq:sup_soft_identity} \boldsymbol {\lambda }^{+}(\bp ) = \boldsymbol {\lambda }(\bp ) + \Bigl \{\bs (\bp ) - \bf {sg}\bigl [\bs (\bp )\bigr ] \Bigr \} ~~~~~~~~ \bs (\bp ) = \text {Softmax} \bigl ( -\bu (\bp ) \bigr ) (22)

where second term s(p) − sg[s(p)] is zero in value, with sg(·) standing for the stop-gradient (e.g., .detach() in PyTorch)
operation, and thus contributes only the gradient for updating GraphDreamer.
Penetration Constraint Lpenet . In Eq. (18), we introduced a rather intuitive definition for the penetration constraint, whichis, however, not continuous over object identities i ∈ {1, · · · , M }. To further refine this constraint, we implement Lpenet as:
  \label {eq:sup_loss_penet} \mathcal {L}_{penet} = \frac {1}{(M-1)N} \sum _{j\neq i,\bp } \Bigl \{\texttt {ReLU} \bigl [\bd (\bp ) - \bu (\bp ) \bigr ]\Bigr \}^2 ~~~~~~~~ \bd (\bp ) = \texttt {ReLU}\bigl [ \boldsymbol {\lambda }^{+}(\bp ) \cdot -\bu (\bp )\bigr ] (23)

where N is the size of sampled positions on the current ray, λ+ (p) is the differentiable identity vector defined in Eq. (22),and term d(p) is an one-hot vector. If p is inside or on the surface of object o(i) , and u(i) (p) gets the minimum among otherobject SDFs, the only non-negative element of d(p) equals the absolute value of u(i) (p). Then, Lpenet prevents all other SDF
values (j ̸= i) from being negative. Or else, if p is outside of all objects, Lpenet has not impact on u(p). An ablation study forthis constraint can be found in Section B.

A.2. Hyperparameter Settings
Training loss coefficients. We set coefficients {β1 , β2 , β3 } defined in Eq. (20) based on the magnitude of each loss term, as
β1 = 1 for LSDS , β2 = 100 for Lpenet , and β3 = 10 for Leknl .
Classifier free guidance. Classifier-free guidance [9] weight (CFG-w) is a hyperparameter that trades off image quality anddiversity for guiding the 2D view. We schedule CFG-w for the training of GraphDreamer roughly based on the number ofobjects M as below, considering that the larger the number M , the more difficult it is for mode seeking.
Method(Nmber of Objects)
Coarse Stage (<10K steps)
Fine Stage (10K∼20K steps)

GraphDreamer
M =2
M ≥3

Magic3D

MVDream

50 (IF)
50 (SD)

100 (IF)50 (SD)

50 (MVSD)

100 (IF)50 (SD)

Table 3. CFG-w settings for GraphDreamer and baseline approaches. IF stands for DeepFloyd-IF model, SD for Stable diffusion, and MVSD for MVDream
Stable diffusion, as detailed in Section 5 (para. Baseline approaches).

B. Additional Experiments and Results
B.1. Ablation: Penetration Constraint
As introduced in Section 4.4 and defined practically in Eq. (23), the purpose of using the penetration constraint Lpenet is toprevent unexpected penetrations between the implicit surfaces of objects, represented by SDF u(i) (p), in a multi-object scene.
To verify the necessity of this constraint, we ablate Lpenet in training GraphDreamer and report the quantitative and qualitative
results of this ablated configuration, denoted as GraphDreamer w/o Lpenet , in Table 4 and Figure 6.
GraphDreamer(w/o ℒ𝒑𝒆𝒏𝒆𝒕 )GraphDreamer

cute, cartoon

Two macaw
parrotsa fox

Sharingworking on

A milkshakea jigsaw puzzle

With two
straws"a fox working on a jigsaw
puzzle, cute, cartoon""two macaw parrots sharing amilkshake with two straws"

A baby bunnya silver knight

Sitting on top of

gallopingchestnut

GraphDreamer(w/o ℒ𝒑𝒆𝒏𝒆𝒕 )GraphDreamer

A stack ofpancakesa horse"a silver knight gallopinga chestnut horse"“a baby bunny sitting on
top of a stack of pancakes”

Figure 6. Ablation study: GraphDreamer (w/o Lpenet ) stands for the configuration that trains GraphDreamer without penetration constraint Lpenet (p).

As shown in Figure 6, objects generated without Lpenet are more likely to inter-penetrate; as a result, it is much harder to
identify clean segmentation boundaries between objects. The CLIP scores shown in Table 4 also suggest a degradation in theperformance of this ablated configuration in modeling individual objects, as the mean similarities of both object image C (i)with its own prompt y (i) (w. Self Prompt) and scene image C g with global prompt y g (Global) drop down significantly, while
the mean similarity of C (i) with other object prompts (w. Other Prompts) increases slightly.
CLIP Score
GraphDreamer (w/o Lpenet )GraphDreamer

w. Self Prompt
mean (↑)
std.

w. Other Prompts
mean (↓)
std.

Globalmean (↑)

std.

0.26650.30770.20700.2006

0.30640.33260.02100.02520.00910.01210.00870.0085

Table 4. Ablation study of the penetration constraint (with or without Lpenet ).

B.2. User Study
Methods
Selected (%)

MVDream
23.12

Magic3D
14.62

GraphDreamer
62.26

Table 5. User study: selecting one from three generated results that best aligns with given text prompts. The results are collected from 31 raters andsummarized over 30 multi-object prompts. All raters are asked to evaluate all prompt examples.

To further evaluate the performance of GraphDreamer in generating guidance-compliant multi-object scenes, we conduct asurvey of the results generated by the baseline approaches and GraphDreamer over 30 multi-object text prompts. We invite
31 raters for this study. All raters are asked to evaluate all examples and, for each prompt, to select one of the three resultsDreamFusion

Fantasia3D

GraphDreamer

Two macaw
parrots
SharingWith two
straws

A milkshake"two macaw parrots sharing amilkshake with two straws"

An octopus
Playing
The piano"an octopus playing the piano"“a Wizard standing in front of a Wooden Desk,gazing into a Crystal Ball perched atop the
Wooden Desk, with a Stack of Ancient Spell
Books perched atop the Wooden Desk”(Scene Graph can be found in Figure 1)

Pushing

A teddy bear

PushingWith right hand

Holding
In left hand

A shopping cart

A handful ofballoons

Full ofvegetables

Floating

Case failed in texture training stage.

Colorful“a Teddy Bear pushing a Shopping Cart full of vegetables withright hand and holding a Handful Of Balloons in left hand”

A hippoBitingthroughLarge

A watermelon

Juicy"a hippo biting through a watermelon"

A badger

Wearing aparty hat

Eating
Decorated
With birthday
candles

A cakeLayered“a badger wearing a party hat andblowing out birthday candles on a cake”

CrackedTwo halves

A chick

FluffyStanding next toYellow

An eggshell
Broken in two

Newborn“a DSLR photo of an Eggshell broken in two
with an adorable Chick standing next to it”

A corgi
RidingA rocket“a corgi riding a rocket”

Figure 7. Extended qualitative comparison with more baseline approaches. Note that Fantasia3D relies on the SDF initialization, which may achieve better
performance if fine-grind initial shapes are provided. Here, we report results using default sphere initialization.

that semantically best fits the given prompt (if all/none of the results fit well, select the one that has the highest fidelity). The
answers are summarized in Table 5. In general, more raters (62.26%) selected the results produced by GraphDreamer over theother two baseline approaches, considering our results to be more consistent with the given prompts.

B.3. Extended Comparison to Other Text-to-3D Methods
To provide a more comprehensive comparison on the performance of recent TT3D methods in generating multi-object scenes,
we extend the quantitative and qualitative comparison reported in Table 1 and Figure 3 respectively with two more baseline
approaches, DreamFusion [29] and Fantasia3D [4] and one more evaluation criteria, CLIP R1-Precision as reported in Table 6.
Both baseline results are obtained after 20K training steps. For DreamFusion, we adopt the same two-stage training protocol as
GraphDreamer and Magic3D, with details provided in Section 5. As shown in Figure 7, problems related to guidance collapse
occur in both methods, while Fantasia3D failed in some cases. Note the performance of the Fantasia3D methodology mayvary depending on how the shape of the SDF surface is initialized, whereas here we have only performed the default sphere
initialization (with a radius of 0.5). We have also conducted generation experiments with ProlificDreamer [42], which adopts athree-stage optimization strategy, and yet it still failed to generated any content in these multi-object cases after 20K steps oftraining in the first stage (both setting CFG-w to 100 or 200), and the results are thus not included.
MetricMagic3D

MVDreamDreamFusion

Fantasia3D

GraphDreamer

CLIP R1-Precision
CLIP-B/32 Score

87.5%
0.326789.2%
0.310280.0%
0.274371.7%
0.224394.2%
0.3326

Table 6. CLIP-B/32 Score & R1-Precision across all 30 multi-object scenes. The reported precision is averaged over 4 orthogonal views for each object.

Videos of Figure 3 and Figure 5 can be found in our project page.

C. Inverse Semantics: A Paradigm of Semantic Reconstruction with GPT4V-guided GraphDreamer
In this section, we envision a new paradigm, called inverse semantics, which first reconstructs a scene graph from an input
image and then produces a compositional 3D scene based on this scene graph. We call it inverse semantics, because it
resembles the idea of inverse graphics in a high-level sense. Inverse semantics differs from inverse graphics in the aspect ofreconstruction that is emphasized; it focuses on interpreting semantic meaning rather than reconstructing visual details. The
comparison between inverse graphics and inverse semantics is given as follows.
Pixel-level reconstruction
Inverse Graphics
Physical parameters

Image

Image

(Albedo, Geometry, Lighting, Viewpoint, etc.)

Encoding

3D representation

Rendering(a) The traditional analysis-by-synthesis reconstruction pipeline
Semantics-level reconstruction
Inverse Semantics
Scene Graph

Image
Encoding

Object-centric 3D representation
Decoding

Image
Rendering(b) The semantic reconstruction pipeline

Figure 8. A paradigm comparison between inverse graphics and inverse semantics

Specifically, we can implement the inverse semantics paradigm with a GPT4V-guided GraphDreamer. We first use GPT4V
to obtain the scene graph from an input image, and then apply GraphDreamer to generate a compositional 3D scene based on
this scene graph. Enhanced by GPT4V’s powerful image understanding capabilities, we can obtain a detailed scene graph fromthe input image and generate a 3D scene from the graph that semantically inverses the given image. An qualitative example of
our inverse semantics paradigm is provided in Figure 9.
Moreover, it is also possible to extract rough estimates of the center coordinates of the objects in the graph from the image
with GPT4V. The coordinate estimates are in 2D (the up-right plane) only, since the image is in 2D. For the third coordinate(on the front axis), GPT4V can offer the relative order by layering objects from the viewer’s perspective. Then, with the
SDF representation of 3D objects, we can use these estimates as the sphere centers when initializing object SDFs u(i) (p) to
specify more accurately the spatial relationships of the objects, and thus to make the generated 3D scene better match the given
image. We expect that the inverse semantics paradigm could be of general interest to the community, and our GPT4V-guided
GraphDreamer serves as an important stepping stone towards this direction.

Please generate a scene graph that best
describes the image in the style of the
Visual Genome dataset, restricting to
the most salient 3 nodes.

User

long
grey hair

grey
beardedManelderly
wide-brimmed

Nodes (Objects):
- Man
- Hat
- Shotgun

ChatGPT

pointedtop

wearing
on headholdingwith both hands

Hat

Shotgunautomatic

grey

curvedmagazine

metallic

Edges (Relationships):- Man -> wearing (on head) -> Hat- Man -> holding with both hands -> Shotgun
Attributes:- Man: elderly, grey bearded, long grey hair
- Hat: wide-brimmed, pointed top, grey
- Shotgun: automatic, curved magazine, metallic

Figure 9. A qualitative example of the inverse semantics paradigm. We generate a scene graph directly from an input image with GPT4V and restrict thenodes present to the most salient ones. GPT4V identifies the objects and provides with proper attributes and edges for each object. This makes it possible toinverse modeling the semantics of a given scene image and extends the potential applications of GraphDreamer.

User

sleepingpink

Please generate a scene graph that best describes
the image, in the style of the Visual Genome
dataset, restricting to the most salient 5 nodes.

restingoncomfortable

Nodes (Objects):- Kirby (Character)- Table- Lamp- Bookshelf- Cushion (Soft Surface)

ChatGPT

soft

casting
light onCushionplush

Edges (Relationships):- Kirby -> resting on -> Cushion- Lamp -> place on -> Table- Lamp -> casting light on -> Kirby- Bookshelf -> standing beside -> Table- Cushion -> positioned in front of -> Table

round

Kirby

woodensturdyLamp

positioned
in front ofplacedon

Table

standing
besideloworganized

yellow
light
illuminatingbrightholding plant

Bookshelf

stocked with books

Attributes:- Kirby (Character): pink, sleeping, round- Table: wooden, sturdy, low- Lamp: yellow light, illuminating, bright- Bookshelf: organized, stocked with books, holding plant- Cushion (Soft Surface): soft, plush, comfortableUser

Can you try to estimate the center coordinates of the objects inthe scene graph? Try your best to give me a rough estimate.
…

Figure 10. An example of more complex semantics represented with more nodes. ChatGPT structures the semantics in this complex image into a scene graphusing the most salient node. To inverse such semantics and generated the scene in 3D, one may further ask GPT to provide with center coordinates for eachnode, which can make the semantic inverse more precise in terms of spatial relations.

D. Failure Cases and Limitations
GraphDreamer still has some limitations. First, the generation quality of a single object is still largely limited by the SDS
optimization. The commonly observed multi-head problems (i.e., Janus problem) may also appear in GraphDreamer. See
Figure 11(a) for an example. The “piglet” in the generated results exhibits multiple heads from different viewpoints. Second,the decomposition of different objects may sometimes fail. See Figure 11(b) for an example. We observe that the disentangled“astronaut” still looks like a reasonable astronaut, but the disentangled violin is affected by some parts of the astronaut. Third,
the semantic meaning of some object in the scene may be incomplete. See Figure 11(c) for an example. The hero characterA piglet

An astronaut

Playing

Playing

The piano

The violin"an piglet playing the piano""an astronaut playing the violin"(a)Determined

Hero
Character

confronting
DragonTerrifying

Large

(b)

seeking
protecting

Glowing

Crystal
MysticalPowersource“A Hero Character confronting a Dragon that
protecting a Crystal, fantasy game scenario”(c)Figure 11. Examples of some failure cases. (a) In some cases, results of GraphDreamer may come with the Janus problem. (b) GraphDreamer failed toseparate the violin from the astronaut’s left hand, which is closely held by that hand. (c) One leg of the Hero is missing. Part of these problems may stem fromthe SDS loss which only offers an approximate gradient (the Jacobian of Stable Diffusion is set to identity) and the known challenges when diffusion models
deal with detailed parts. Optimizing multiple SDS losses Eq. (16) at the same time can also affect disentanglement. Enhanced supervision on object parts andimproved prompts could help.

lacks a leg in the generated result.
We believe that part of these problems stem from the SDS loss which only offers an approximate gradient (the Jacobian of
Stable Diffusion is set to identity) and the known challenges when diffusion models deal with detailed parts. We also suspectthat optimizing multiple SDS losses Eq. (16) at the same time may affect disentanglement, and thus enhanced supervision on
object parts and improved prompts could help.