Imagine That! Abstract-to-Intricate Text-to-Image Synthesis with Scene Graph Hallucination Diffusion
Authors: Shengqiong Wu
Hao Fei
Hanwang Zhang
Tat-Seng Chua
Venue: NeurIPS 2023 poster (2023)
Keywords: Image Synthesis
Scene Graph
Diffusion Model
TL;DR: We solve the text-to-image synthesis under the abstract-to-intricate setting with a novel scene-graph hallucination mechanism with diffusion models.

Abstract: In this work, we investigate the task of text-to-image (T2I) synthesis under the abstract-to-intricate setting, i.e., generating intricate visual content from simple abstract text prompts. Inspired by human imagination intuition, we propose a novel scene-graph hallucination (SGH) mechanism for effective abstract-to-intricate T2I synthesis. SGH carries out scene hallucination by expanding the initial scene graph (SG) of the input prompt with more feasible specific scene structures, in which the structured semantic representation of SG ensures high controllability of the intrinsic scene imagination. To approach the T2I synthesis, we deliberately build an SG-based hallucination diffusion system. First, we implement the SGH module based on the discrete diffusion technique, which evolves the SG structure by iteratively adding new scene elements. Then, we utilize another continuous-state diffusion model as the T2I synthesizer, where the overt image-generating process is navigated by the underlying semantic scene structure induced from the SGH module. On the benchmark COCO dataset, our system outperforms the existing best-performing T2I model by a significant margin, especially improving on the abstract-to-intricate T2I generation. Further in-depth analyses reveal how our methods advance.

Imagine That! Abstract-to-Intricate Text-to-lmage Synthesis with Scene Graph Hallucination Diffusion
Shengqiong Wu
Hao Fei Hanwang Zhang Tat-Seng Chua
NEET+++, School of Computing, National University of Singapore
2 School of Computer Science and Engineering, Nanyang Technological University swucu. nus. edu fhaofei37, dcscts) (aactts)nuuu.etti
hanwangzhangāntu. edu...ueedddu..... edu.
Abstract
In this work, we investigate the task of text-to-image (T2I) synthesis under the abstract-to-intricate setting, i.c., generating intricate visual content from simple abstract text prompts. Inspired by human imagination intuition, propose novel scene- scene-graph hallucination (SGH) mechanism for effective cestract-to-intrican T21 synthesis. SGH carries out scene hallucination by expanding the initial scene graph (SG) of the input prompt with more feasible specific scene structures, which the structured semantic representation of SG ensures high controllability of the intrinsic scene imagination. To approach the T21 synthesis, we deliberately build an SG-based hallucination diffusion system. First, implement the SGH module based on the discrete diffusion technique, which evolves the SG structure by iteratively adding new scene elements. Then, we utilize another continuous-state diffusion model as the T2I synthesizer, where the overt image-generating process is navigated by the underlying semantic scene structure induced from the SGH module. On the benchmark COCO dataset, our system outperforms the existing best persfreeffrming T2I model by significant margin, especially improving on the abstract-to to-intricate T2I generation. Further in-depth analyses reveal how our methods advance.
Introduction
The task of generating images from natural language descriptions, known text-to-image (T2I) synthesis, has attracted significant attention [4, 17, 52]. To approach T2I, various generative models have been explored, including generative adversarial networks (GANs) [48, 52, 61]. variational autoencoders (VAEs) [50], flow-based models [3], and auto-regressive models (ARMs) [10, 41], all of which aim to generate realistic images in high quality and high faithfulness. Most recently, diffusion- based models have been proposed, which simulate the physical process gas diffusion for image generation [23]. Diffusion models have shown unprecedented performance in image synthesis over existing methods, becoming the current state-of-the-ar (SoTA) T2I solution [2, 9, 19, 47, 42].
As long-reached viewpoint [28, 53, 37], sound T2I systems should not only achieve high-quality image generation in simple straightforward visual scenery but be more capable synthesizing realistic images with complex scenes. Typically, detailed textual descriptions to prompt the synthesis process with adequate details for high-quality vision generation. However, in realistic world, it could also be ubiquitous to produce intricate visions without relying lengthy elaborate prompts. For example, users may prefer T2I systems to synthesize dell-detailed images while not taking too much time write descriptions in detail. More crucially, due to the natural modality asymmetry between language and vision, even some simple words can intrinsically describe represent abstract visual scenes with rich and complex details. Whenever mentioning the words
necessarily needed
HHao Fei is the corresponding author. http:/twwww.ccc//cooo/eeee/ Salad
"Code available
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Original (abstract) Prompt:
Tou young men give provntation in the offic:
Tiro youngg men give presentation in the office.
Constructing
EWD
in
office
Issue: Abstract to intricate Failune
Initial
give
presenlation
Enriched Prompt-I:
Human
Imagination over 5G
Two middle aped, nice, entrusiatice confident, with polished shoes and hair give professional presentation in the spacious and moderns conference room the corporate office, room with chaits and rables,
Enriched 5G
table
in
chair
microphone
two
tall
office
light
blod
man
give
presentation
Issue: Vision Distrac Tion
Rendering
Enriched Prompt-Il:
Two young man give presentation in the office, old, nice, confident, endhuiiatic, laughing man with polochd hair, seek hair, with chairs and tables, speaking each other.
Attribute Node
Object Node
Relation Node
Issue: Wrong Binding
(a)
(b)
Figure 1: (a) An example of abstract-to-intricats T21 synthesis. All images are generated by the Latent diffusion Model (LDM) [42]. LDM fails to accurately render the abstract contexts, c.g., 'give presentation' and 'office' of the original prompt. Raw prompts can be enriched via descriptive insertion [8], or addition [48]. Enriched contexts are in blue. (b) We illustrate the human intuition on the abstract-to-intricate T2I process: we always first grasp the semantic structure of the original prompt text, i.e., scene graph (SG), and then carry out imagination with more complete scenes based on the SG. Here the glowing nodes and edges are enriched ones.
with specific scenes, such as classroom, kitchen, office, or actional verbs e.g., traveling, shopping. there is always picture with multifaceted scenes and rich-detailed backgrounds. In word, is worth investigating generating intricate images from succinci abstract prompts.
Yet existing prevailing approaches, even the SoTA diffusion models, may largely fail the abstract-to- intricate T21, due to the lack of necessary details of input prompts (cf. Figure (())). One intuitive workaround is directly enrich the texts, i.c., adding more details for the prompts. Specifically, existing works either consider inserting additional adjectives and attributives to modify the original mentions and scenes [8], or concatenating raw sentences with more tangible explanations and contexts that are elicited from external large language models [48], e.g., ChatGPT [38]. Unfortunately. due to the intrinsic grammar and linguistics rules, such text-based prompt enrichment can be subject to the issue of lower controllability. One problem is the visual distraction, where the main focus of the resulting image is dominated by other newly-added trivial contents, when aggressively inserting intermediate descriptive components into the raw texts, as exemplified in Figure l(a). Besides, directly appending new textual descriptions would increase the prompt length and then lead to incorrect
binding of attributes relations, i.c., making the image deviate from the original user intention.
As a reference, intuitively, we human beings would tackle the Aobrrcc-to-intricall T2I as two- step painting process, i.c., from semantic interpretation to scene imagination. In the semantic interpretation, a painter always first translates the succinet textual prompt into structured skeleton that represents the semantic scene of key mention objects and their relations. Then, based on the initial scene, the painter mentally completes the abstract scene with more concrete and valid details. With the enriched scene structure, the final vision can be more accurately and easily rendered. Motivated by such human intuition, in this work, we propose a grene-graph hallucination (SGH) method for achieving effective abstract-to-intrice representing the input prompt with its scene graph (SG) [46]. SG advances in depicting the intrinsic semantics of texts (or vision) with structured representations, c.g., objects, attributes, and relationships, enabling fine-graince control of the semantic scene [53]. Based on the SG of input text we then carry out scene hallucination, expanding the initial SG with more possible specific scene structures. Also with the SG representation, the imagination process can be much more accurate and controllable.
T2I. As illustrated in Figure (b), we first investigate
To implement the overall idea, we develop an SG-based hallucination diffusion system (namely, Salad) for high-quality image synthesis. Salad is fully diffusion-based T2I system, which mainly consists of scene-driven T21 module and an SGH module. As shown in Figure 2, we first take advantage of the SoTA latent diffusion model [9] as our backbone T21 synthesizer, in which the overt moage-generating process is controlled and navigated by the underlying semantic scene structure.

---

Scene-driven Image Synthesis (3..)3.2)
(3
U-Nt
N
(eC GUNI
(vC Sane
G
Gu
Prompt Text
56 Decoder
-0
Redation Nodes
Scene Groph Hallucination (df. $3.1)
*---
Diffusiom proces
Denoiding
SG mode type Deperdance (NTD)
Oxject
Nodes
(R.)
Figure 2: Overall framework of our proposed SG-based hallucination diffusion system (Salad).
On the other hand, we design the SGH module based on the podality-agnostie discrete diffusion model [25], which evolves and completes the initial SG structure of the input prompt by iteratively generating new scene elements (i.e., SG nodes). During the SG imagination process, the resulting structural representation at each step is fused into the T21 synthesizer via hierarchical integration strategy. Further, we devise scene sampling mechanism, via which the SGH module can generate various SG imaginations, and thus help achieve diversified image synthesis during inference.
We conduct experiments on COCO [33], the widely-used T2I dataset. Results show that Salad outperforms all the cxisting strong-preee peroonpprrorming T2I systems with significant margins. Further analysis reveals that modeling the SG structures helps synthesize high-quality images with stronger semantic controllability. Our proposed SGH mechanism is effective in inducing sound SG structures, helping produce more realistic images from short abstract text prompts. And the scene sampling strategy aids diversified T2I generation. In summary, this paper contributes in five aspects:
We are the first to study the novel setup of intricate image synthesis from abstract texts. T2I with novel SG hallucination mechanism, which is
We solve the abstract-to-intriee implemented via discrete diffusion technique, performing scene enrichment with reasonable imagination.
We propose a diffusion-baste model with a hierarchical scene integration strategy for highly controllable and scalable image generation.
We devise a scene sampling mechanism to generate various scene graphs for diversified image syntheses during inference. Our framework achieves new SoTA results in the cestrractttonnnricca T2I generation.
2
Preliminary
2.1 Scene Graph Representation
The SG (denoted as G) [45] represents the semantic relationships among scene objects in structure, where there are three types of nodes, i.c., object, attribute. and relation. cf. Figure 1(b). We formulate the object node as (or.... .NN; where on denotes n-thh object node; the attribute node set as {.1,1.-.. AN J. where an,m means m-th attribute node of the 7-th object node; the relation node as fri, ... , .NN.N,, }, where ri, means object node o connects the object 0j- All nodes come with a category label fo/a/r, and each type of node has its own unique category vocabulary. For example as in Figure 2 right bottom, the object node on with category label I associated with two attribute nodes 11,1 and a1,M, with category label If and 123. respectively. And the object node 0 connects to the object node og through edge 11,2 with the category label 1f.
2.2 Diffusion Models
Diffusion models (DMs) [23] learn convert simple Gaussian distribution into a data distribution. Technically, DMs consist forward (diffusion) process and a reverse (denoising) process. In the forward process, the given data 2o ~ g(Zp) is gradually corrupted into an approximately standard normal distribution ør p(æT) over T steps by increasingly adding noisy, formulated as

---

q mo) = -IT g The learned reverse process p p(xr) I pe(x (z:)
-
gradually reduces the noise towards the data distribution. To improve the fit of generative model to the data distribution, variational upper bound on the negative lileliilloood is optimized:
Luts (y(msr//mo)) (mo)
log ((0r(20))
Σ
log g( 1 (i, 20) Po (at-1 (x+)
log po (zoo
(I)
Po (2PT)
1=2
where po(-) is estimated by denoising network, which can be time-conditional U-Net [43]. Recently, latent diffusion models (LDMs) [42. 14, 19] are introduced to adopt DMs to operate in an efficient, cow-dimensional latent space, where pre-trained encoder & maps the given data Do into a latent code 20 {(x)), and decoder reconstructs the final output image from the denoised latent D(zo) ~ To- Due to the higher computation sufficiency, this work thus takes the LDM as T2I backbone. Besides, DMs also have been extended to operate in discrete state spaces [1], i.e., performing diffusion and denoising processes over discrete variables, which have demonstrated competitive performances for discrete data, such as text [24] and layout [5]. Hence, we also adopt the discrete DMs to realize the SG induction process. Appendix 1 gives more technical details to the discrete diffusion models.
3 Methodology
Formally, T2I aims to generate an image x that faithfully reflects the desired content in the input prompt text y. To approach abstract-to-intricat T2I, we propose an SG-based hallucination diffusion system (Salad), which is shown in Figure 2. The salad consists of two major modules. First, the SGH (cf. $3.1) is responsible for enriching the initial SG of the text prompt via discrete diffusion model. Then, built upon an LDM (cf. $2.2), the drene-driven Image Synthesis module (SIS, cf. $3.2) performs denoising for image synthesis, during which the derived SG features is fused via a hierarchical scene integration mechanism. The underlying SGH closely collaborates with the upper SIS at each step, and thus the semantic scene skeleton takes fine- fine-grained control of the overt vision rendering. We also describe the optimization (ef. 83.3), and the scene sampling strategy (ef. $3.4).
3.1 Scene Graph Hallucination (SGH)
As affrrementoneed, we formulate the SGH as a discrete denoising diffusion process [25] (cf. Fig. 2). Specifically, in the forward process, the SG of the gold image, marked Go. will be corrupted into a sequence of increasingly noisy latent variables G = {G,, ...... GT]. where each SG node $². E GI,* E {o,a, a, r (t is diffusion step, j is the node index) takes a discrete value with K category labels, and o, a. denotes the nodes' type, i.e., object (o), attribute (a), and relation (r). For simplicity, we omit subscripts j and superscripts in the following description. The discrete diffusion process can be parameterized with multinomial categorical transition matrix:
((<((1r1)) = B (s:) Q: B(B(()))),
where B(s;) denotes the column one-hot vector of st. And Q, is the transition matrix, with (Qi)i q(5t j|ss-1 i) representing the probabilities that 8;-1 transitions 8t. Due to the property
of the Markov chain, the cumulative probability of s; at arbitrary timestep from 8p can be derived as ((5)]5))= BT (s.) Q. B(so), where Q Q.Q.-1
Q Inspired by [1, 19], employ mask-and-replace strategy to design the Q. For each node St. we define three probabilities: 1) a probability of 7, transition to [MASK] node, 2) a probability of K Be be resampled uniformly over all the K categories, and 3) a probability of a, = 1 - K 3, Tt to stay the same node. Notedly, the [MASK] node never transits to other states. Hence, the transition matrix Qe can be formulated as?:
Tix + 3,
3
B.
0
82
+
3
B.
Q:
(3)
7
7A 7
The aforementioned discrete state-space models assume that all the standard nodes are switchable by corruption. However, as stated in 12.1, there three different SG nodes under separate categories. Hence, we apply three disjoint corruption matrices Qi E KK* xK" Qi E K > Qi E RK xk"
for object, attribute, and relation nodes, respectively, where K K K" denotes the size of category labels of three node types respectively.
Appenddx 1 provides detailed formulation of the discrete diffusion process.

---

In the denoising process, we introduce an SG decoder as the neural approximator to estimate the distribution (0($,-155,. y). As shown in Figure SG decoder first employs an adaptive normalization (AdaI.N) to inject the timestep information. A text cross-attention (Text-CA) integrates the input prompt y. Then, a graph cross-attention (Graph-CA) is devised to take in the induced SG (G++1) in the previous [+1 timestep:
H Graph-CA(G++1, H.
(4)
where H' the features yielded from Text-CA. Graph-CA consistently consults the overall picture of the last SG for a more coherent generation when making the current decision.
Intuitively, among the object, attribute, and relation nodes, the object nodes always come first to determine the scene subjects, followed by their modifier attributes, and then the relations between objects. Thus, instead of parallel induction of three node types, we follow this SG node-type dependence (NTD) intuition, and design an NTD cross-attention (NTD-CA) for the *-type node induction (* can be object or attribute):
NTD-CA(s H") H
(5)
NTD-CA(s) e of H")
==r
Note that we stack multiple layers of the above calculations as one SG decoder. For each state of node types 87, softmax function is put on to obtain the category label distributions: 87 Softmax HH)).
Following [1], optimize the SG decoder by minimizing the variational lower bound Livs (Eq. 1). Also the parameterization trick [19] is leveraged to encourage the system to predict the noiseless node distribution pe (5g) st, 3) each reverse step, which is taken as an auxiliary learning objective to be incorporated with Lois
LSGH = Luth + a logpo(solse.0)
Cun Lo Li +... Lr- + Lr.
Co -logpo v).
(6)
L;- = DxL (x su (ppo
CT Dx (9( (so) (sT)),
where Aj is hyper-parameter for controlling the learning components.
3.2 Scene-driven Image Synthesis (SIS)
1
Object level
U-Net
With the enriched SG from SGH in each de- noising step hand, the backbone T2I diffusion carries out the image synthesis with the guid- ance of that SG. We design hierarchical scene integration (HSI) strategy to ensure the highly effective integration of SG features. Specifically, we consider the fusion at four different hierarchi- cal levels, i.e., object (I.),, relation (L"), region (L"), and global levels (I?) with each focusing on different context scopes, illustrated in Fig- ure 3. We maintain the representations of these three levels the keys K2 & values Vz. via CLIP encoder [40], which are then integrated together via the Transformer attention of U-Net in LDM:
Relation level
Region level
Global level
Altn
K₁
Alm
Atin
56
Figure 3: Hierarchical scene integration (HST) fuses the features under multiple levels: 1) objects (with attributes), 2) relational triplets (i.c., predicct-reedcate- object), 3) regional neighbors. and 4) the whole SG.
HK
H Σ Att H.H.HKK, K ViA)= 1Efo.r.c.9)
Σ 1E(o.r.c.9)
Softmax
Ves
where H is the visual query vectors from the ResNet block LDM. The above hierarchical scene integration is carried out for both the downsampling and upsampling processes in U-Net. By denoising steps, the system finally produces the desired image. Appendix $A.2 gives more details of this part.
3.3 Warm-start Training
To ensure stable learning of the overall system, we take a warm-start training strategy. Firstly, the SGH is separately updated via LSGH (Eq. 6) based on the cntract-to-intrical SG pair annotations,
5

---

until it has converged. Then, both the SIS and SGH modules are optimized jointly by minimizing:
L= CeCas + LSis, where Lsis -x=
co(z1, G., y, t)
(8)
Here we follow [23] to optimize SIS with simple surrogate objective that calculates the mean- squared error loss, and G2 is the intermediate SG by SGH at timestep i, which can be derived from the 8 (o, r) (cf. Figure 2). is the noise in SIS, and co)-) denotes the U-Net (cf. Figure 3).
3.4 Inference with Scene Sampling
Category Distribution $
During inference, we further aim to en- dow the SGH with diversified SG en- richment, and thus lead to T2I diversi- fication. Intuitively, given an abstract prompt, there is often more than one pos- sibility of the potential scenes to imagine. Also, it can be observed that in the de- noising process, the diffusion model has larger potential of divergence only at its earlier stage, while the generation tends to be more stable and certain when the iteration grows. Correspondingly, expect the scene sampling to start in the early denoising steps, and gradually be more determining. Thus, design scene-sampling mechanism. First, instead of picking the best prediction of the category of any node 87 we take the top-A category candidates with corresponding probability distribution v based on the category distribution 87,, R of node $T.. Then, perform sampling
Top-A
Sampling
Nir 12
lb
Lis
On
Ruo
candidates
Figure 4: Illustration of the scene sampling mechanism.
over these candidates with a dynamic probability:
Pig = V +1 c イ. 1
99)
where (o,a,r), and 17 is temperature. It is an annealing process, i.e., when t=T (starting denoising) more random sampling is preferable, while / approaches 0 (denoising ends), SGH tends to be more decisive. Figure 4 exemplifies the mechanism with the object type of nodes ($9,n).
Experiments
4.1
Settings
Data and Resource We conduct T2I generation experiments mainly on the COCO [33] dataset. We also prepare the -onttrrcttt SG pair annotations for training the SGH module, where employ an external textual SG parser [46] and a visual SG parser [59] on the paired images and texts in COCO, to obtain the initial SG and imagined SG, respectively. To enlarge the bbstract-to-intricall SG pairs, we further extend Visual Genome (VG) [30]. Besides, to simulate the abstract-to-intricaae T21 scenario, we further manually extract subset of text-image pairs from raw COCO data (named COCO-A21) in which the texts are short and abstract,' 4 while the images are comparatively complex and intricate. Appendix $B.1 shows all the dataset details.
Baseline and Evaluation We make comparisons with three types of existing strong-performing T21 models. 1) GAN-based models: AttnGAN [52], ObjGAN [32], DFGAM [48], OPGAN [22]. 2) Auto-aggressive model: DALL-R(41], and Cog View [10]. 3) Diffusion-based models: L.M [42], VQ-diffusion [19], LDM-G and Frido [14] with classifier-free guidance. Note that LDM-G and Frido are the current SoTA T2I synthesizers. In addition, two types of text-based enrichment approaches are included baselines: stable-diffusion prompt generator (SD-PG) and SPY inspired by [6]. The enriched text prompts are then utilized as inputs for Frido to generate the final images. We adopt three standard metrics to measure image synthesis performance: 1) Inception score (IS) [44], 2) Fréchet Inception Distance (FID) [21] and 3) CLIP score. Moreover, we use GLIP [31] to measure the fine- -raiind 'object-attribure grounding in images, and Triplet Recall (TriRec.) measure the cebjjctt-reedicte--oe
triplet recall between two SGs. We also adopt the Learned Perceptual Image Patch Similarity (LPIPS) [60] for diversifying generation evaluation. Detailed definitions of evaluation metrics are shown in Appendix BB.2.
We select the texts that mainly include two types of words, i.e., Place Nouns (e.g.. office, airport, bathroom) and Progressive Verbs (e.g.. traveling, knitting, gardening).

---

Table 1: The T2I results on the overall COCO dataset. :: taken from Fan et al. [14]; : copied from Ding el al. [10], 4 taken from Hinz et al. [22]. The best score is in bold, and the sccond best is underlined.
Table 2: Results on the COCO-A21 subset for the Dostract-to-intrical T2I generation. 'SPYt. denotes enriched texts parsed into SG, and then perform SG-to-image generation via Frido.
Model
1st
CLIP f
Model
FID
IS
CLIP
GAN Model
Baseline
AttnGAN
33.10
26.61
AttiGAN
78.19
11.09
52.78
ObjGAN?
36.52
24.09
ObjGAN
75.33
13.16
55.20
DEGAN
21.42
DEGAN
71.24
15.56
56,91
OPGANA
24.70
27.88
DALLE
66.36
16.03
Anto-aggressive Model
CogView
62.85
16.98
63,97
DALLER
27.34
17.90
LDM
55.27
16.20
67.79
Vie
27.10
18.20
VOdiffusion
69.14
15.78
64,58
Diffusion Model
Frido
40.36
18.36
68.53
L.DPP
17.61
19.34
65.00
Text-based Enrichminnt (+Friddo)
VQ-diffusion"
14.06
21.85
67.70
SD-PG
36.50
17.64
65.23
L.DM-GP
12.27
27.82
69.27
SPY
35,59
21.59
67.86
Fridn
11.24
26.84
70.46
SPYt
39.41
26.
66.93
Salad
10.19
29.96
73.83
Salad
31.25
28.63
71.29
Frido
SPY
Salad
FEDRCEST ttt
front
windows
man sits in
man
sitin
bank
the bank
sit
chair
Tshirt
black
wear
sunglass
man is traelllig
wear
man
carrying on
loggage bag
young
in front of
building old
TRAY
nexl
lable in
ktichen
sits in the kitchen
sit on
chair in
TDan
next to
microwave
next to
fridge
Figure 5: Qualitative results by different methods, where the given prompts randomly selected from the test set short and abstract expressions (marked in red), while describing intricate visual scenes.
Implementation We define the maximum number of SG object nodes as 30, and each object node has maximum of 3 attributes. We set the timesteps (7)) for SGH and SIS as 100. For the SIS module, we load the parameters of Stable Diffusion" (v1.4) as the initialization. We use the CLIP (vit-large-patchl)
as our text encoder. We optimize the framework using AdamW [34] with 31 0.9 and 32 = 0.98. The learning rate is set to 5e-5 after 10,000 iterations warmup. For the attention layer in SG decoder and UNet in SIS, we define shared configuration follows: 4 layers, attention heads, 512 embedding dimensions, 2,048 hidden dimensions, and 0.1 dropout rate. We mainly follow the prior works [54, 18, 46] to acquire the visual scene graph (VSG) from the gold image and textual scene graph (TSG) from the text prompt.
4.2 Main Comparisons
Table shows the main T2I generation results on the overall COCO data. We see that diffusion-based methods are consistently better than the other two types of T2I methods, especially on the FID metric. Most importantly, our proposed Salad model yields the overall best results on all metrics. For example, we outperform Frido by 1.05% FID, 3.37% CLIP score, and surpass LDM-G by 2. % IS score. This directly demonstrates the efficacy of our method.
To directly assess the capability of solving the comparisons on the collected COCO-A2I dataset. Also the two text-based enrichment methods included. As shown in Table 2, with text enrichment, both SD-PG and SPY achieve better
T2I synthesis, we further make
htttp::///giaibbbccm////pppppppp 'https://gi wht..//ggtttu..e/. htttp:///gthhb...oo/geeaaa// cww.oppmaa//LLI
7

---

performance than the non- enriched T2I methods on both FID and IS metrics. However, we find that the two enrichment approaches fail show superior results on the CLIP scores. This is mostly because text-based enrichment methods suffer the aforementioned issues of wrong binding visual distraction, etc., which results in lower matching between the generated images and raw (abstract) texts. In contrast, our model presents the overall optimal performance, significantly. This directly verifies the effectiveness of our method with the scene graph hallucination mechanism, i.c. yielding reasonable imaginations of scenes and leading to high-quality abstract-to-intrican T2I synthesis.
4.3 Ablation Studies
Here we present the model ablations to quantify the contribution of each part of our system. The results are shown in Table 3. First, canceling the whole SGH module (w/o both SG guidance and imagina- tion), the most significant performance drops are witnessed, indicating the pivot influence of the SGH mechanism. Also, the drops from 'SGH->Init.. directly reveal the importance of our scene enrich- ment mechanism. Interestingly, by comparing the ablating drop in COCO and COCO-A2I, we can no-
Table 3: Ablation results (FIDJ), 'SGH-=nnit.SG"
using the initial SG of text without SG enrichment throughout the whole T21 generation process. "w/o SGH': : removing the overall SGH module, i.e. also meaning both without scene enrichment and SG guidance. "HSI+ encoding the SG with GCN [35] instead our HSI mechanism.
Item
COCO
COCO-A21
Salad (Ours)
10,19
31.25
tice that the SGH mechanism is especially more im-
5GT IIi.SS
T2.75(-2226
37,7766355
portant under the abstract-to-intriccte setup, while
w/o SGH
16.98(+6.79)
34.32(+3.07)
the SG features more stand out for the general T2I
w/o Graph-CA (Bq
(0.95(+0.76)
32.74(-1.19)
scenario. Besides, both properly modeling SG fea-
w/o NTD-CA (Fq.5)
11.19(-1.0))
34.35
FIsT SGCN
T0.87440688)
38.966-1711)
tures during denoising and modeling the SG node-
type dependence (NTD) contribute to the overall
performance. Further, when we abandon the hierarchical manner of the scene feature integration (HSI) and instead use GCN to encode the overall SG, there are also clear performance drops.
4.4 Qualitative Results
To gain a more intuitive understanding of our model's capability on the abstract-to-intrical T2I synthesis, here we show some qualitative results. We visualize the generated images by different methods, where the given input prompts are in short and abstract format but describe intricate visual scenes. As shown in Figure 5, we find that Frido fails to generate images that precisely reflect the prompt instructions. In contrast, the text-based enriched method, SPY, yields visual results with much more details, in which the visual semantics, unfortunately, deviates much from the raw inputs largely. Overall, Salad is able to produce high-quality images with rich visual scenes and sophisticated contexts, meanwhile ensuring semantic accuracy, i.e., coinciding with the abstract input texts.
4.5 Analyses and Discussions
e LDM RB Frido 8 SPY all Salad"
Via the above experiments, we have demonstrated the tech- nical efficacy of our model. Following we further explore how our methods advance.
CLIP
(%)
Q1: How does SG guidance aid the generation of high- quality images? We first consider assessing the overall semantic matchness between the input texts and the gener- ated images. We make comparisons with baselines without integrating SGs. By observing the CLIP scores in Figure 6, it is clear that our model can achieve higher semantics faithfulness of image synthesis. Further, we try to probe the semantic structural alignment between the inputs and outputs. We mainly measure the 'object-atrribute" corre- spondence between input texts and generated images via GLIP metrie; and also assess if the structural triplets in initial textual SG can also be retrieved in the SG of the generated image (i.e., TriRec. metric). As shown the figure, our Salad system consistently performs much better than the baselines in terms of these two metrics. This significantly suggests that with the SG guidance, the structural controllability of the T2I process is greatly enhanced, and thus leads to high-quality generation.
GLIP
10
12
14
16(%)
TriRet.
79
81
83
85
(%)
Figure 6: Evaluating the matches between input texts and generated images on COCO. For comparison, our ''alad" is downgraded by removing the SG imagination ability (i.e.. SGH-> Ini..
8

---

Q2: Does SGH indeed induce intricate and reasonable SGs? Firstly, in Figure we explore whether the SGH can produce new SG structures during the T21 process, on the COCO and COCO-A21 datasets separately. As seen, after scene enrichment, the average numbers of all three element types (object, attribute, and relation) substantially increase. Notably, the addition is more evident on the COCO-A2I set, where scene enrichment is more needed.
Initial SG
Inagined SG
COC0
C2CO-A21
Obj.
Altr.
Rela.
Obj
Altr.
Rola.
Figure 7: The average number of three types of SG nodes.
Next, we examine if the newly imagined SG structures provide reasonable scenes to the input prompt. We reach this by measuring pairs
Table 4: Comparing the glod SGs (G') with our induced SG (G), and the SG (G") of gener- ated image with TriRec. metrie.
the recall rate (TriRec.) of the between two SGs, i.e., the predicted and the gold SGs. Assuming the SGs (denoted as G*) of gold images entail reasonable scenes, we consider two types of SGs from our 'prediction': 1) SGs induced by SGH (G), and 2) SGs parsed from our synthesized images (G)). ). We also compare with the text-enriched SPY method. As shown in Table 4, we first observe that the induced SGs highly align with the gold SGs, with 82.01 TriRee. score, indicating the induced SGs are sensible for generating high-quality images. Moreover, by comparing the TriRec scores (G' vs. G¹) between SPY and Salad (i.e., 78.61 vs. 86.04), we learn that the synthesized images coordinated by imagined SGs more correspond to the gold images in terms of semantic scene structure. This also suggests that the SGH can induce valid SGs which favorably guide the image generation process.
G vs. G
G vs. G
SPY Salad
78.61
82.01
86.04
Does sampling strategy helps diversified T21? To measure the effectiveness of the sampling strategy in diversifying image generation given the same prompt, we consider both the qualitative analyses and human evaluation, where the former calculates the LPIPS score [60] to assess the perceptual similarity between two images in deep feature space. As shown in Figure 8, our Salad model significantly outperforms LDM and Frido in terms of diversity score and human evaluation, and things go worse upon removing the scene sampling strategy. This demonstrates that our proposed scene sampling mechanism is effective in helping diversified T2I. In Appendix $C.2 we show more examples of diversified generations via the scene sampling mechanism.
Q3:
1
Frido
Salad
Salad wlo Sumpling
LPPS
0.7
60
0.5
60
(A) Anto Diversity Score
Human Evaluation
Figure 8: Comparison of diversity using diver- sity score (I.PIPS) and human evaluation.
5 Related Work
T2I is long- standinn topic in computer vision and multimodal communities. Arrays of explorations have been devoted to achieving stronger image synthesis performances with various deep generative models, such as GANs [4, 17, 48, 52, 61], VAES [29], flow-based approaches [11, 12] and ARMs [7, 13, 39, 491. More recently, the diffusion denoising probabilistic methods (DDPMs) have revealed the greatest potentials on image synthesis, in which the optimal density estimation is more naturally achieved with fixed diffusion process to transform image into Gaussian noise [9, 19, 23, 42, 47]. This work follows the line of diffusion methods and takes the SoTA latent diffusion model (LDM) [42, 14, 19] as the T21 backbone.
Generating high-quality images with complex scenes is the key criterion of a sound T2I system [14, 15, 26]. Many efforts have been paid for synthesizing more realistic and nature-looking images in sophisticated scenes, yet most of which are conditioned on taking the detailed descriptions as inputs I6, 51]. Thus, how to generate high-quality images in intricate scenes from succinct and abstract prompts becomes a meaningful yet challenging task. In this work, we introduce a scene hallucination mechanism, which, built upon the SG structure, performs more accurate and controllable scene completion and eventually helps generate intricate images of higher quality. We consider the SG representations [27] for the input prompt texts as well as the guidance of image synthesis. SG advances intrinsically describing the semantic structures of scenes for texts or images [28, 53), enabling more fine- fine-grained control of complex scenes [53], and thus aiding the final image generation.

---

Scene graph (SG) is a type of structured data that represents multiple objects and their complex relationships in the vision or language scenes, wherein the nodes denote objects attributes and the edges depict relationships between objects [27]. As intrinsically describing the semantic structures of scene semanties for the given texts or images, SG has been widely utilized as type of external feature being integrated into downstream applications for enhancements, e.g.. image retrieval [27, 56), image generation [28, 53] and image captioning [54, 36, 55J. In this work, we consider the SG representations for the input sentences. Compared to the linear sequential nature of the text, SG offers a more intuitive manner to represent the scene semantics in a structured format, enabling more fine-grate fine-grained control of complex scenes [53], and thus aiding the final image synthesis.
Our SGH mechanism also relates to the research of SG enrichment or imagination. While existing methods mostly approach the task by incrementally parsing SG elements [16, 57, 58], such greedy- increment paradigm may largely suffer from trapping in locally optimal SG generation, thus leading to inferior SG imagination. Instead, in this paper, we implement SGH as discrete denoising&ddffusiin process. Discrete diffusion technique [25] is the latest introduced method that replaces the continuous state in standard diffusion models with discrete one. During each denoising step, the entire SG structure is updated and optimized from a global viewpoint, so as to yield a more reasonable enrichment of scene structure. Besides, both the T2I and the SGH are modeled as the same diffusion process in our framework, where the two processes are well synchronized, such that the underlying SG features can perfectly guide the T2I synthesis at each step. To our knowledge, we are the first to investigate SG induction using discrete diffusion models.
6 Conclusion
In this work, explore the text-to-image synthesis task under the abstract-to absttrcctto--nntriatte setup. Drawing inspiration from human intuition, we propose a scene-graph hallucination mechanism, which carries scene imagination based the initial scene graph of the input prompt, expanding the starting SG with more possible specific scene structures. We then develop an SG-based hallucination diffusion system for the -o-itriiat T2I, which mainly includes an SG-guided T2I module and an SGH module. Specifically, we design the SGH module based on the discrete diffusion technique, which evolves the initial SG structure by iteratively adding new scene elements. Then, utilize another continuous diffusion model as the T2I synthesizer, where the overt image-generating process navigated by the underlying semantic scene structure induced by the SGH module. On the standard COCO dataset, our system shows great superiority in the abstract-to-intrie analyses demonstrate that our SG-based hallucination mechanism is able to generate logically sound SG structures, which in return helps produce high-quality scene-riched images.
T2I generation. Further
Broader Impact
Benefits The current text-conditioned image generation approaches largely fail to the abstract- to-intricate T2I due to a lack of necessary details of input prompts. In this work, we propose novel scene-graph hallucination mechanism inspired by human imagination intuition, which expands upon the initial scene graph from the text prompts to generate more feasible and specific scene structures. Furthermore, the enriched timestep-wised SG is leveraged to navigate the T21 generation process, leading to synthesizing more intricate images. Our study demonstrates that hallucinating images based on scene graph structures offer scalability, and modeling these structures enhances the generation of high-quality images with improved semantic controllability.
Potential weakness There can be two potential weaknesses that warrant consideration in our system. Firstly, the effectiveness of our system relies heavily on the quality of scene graph hallucination (SGH), yet the absence of dedicated dataset for the SGH task poses a challenge in training the SGH module. However, we can leverage the richly annotated Visual Genome (VG) dataset, commonly used for training visual SG parsers, to provide initial training for the SGH module under an unconditional setting. Secondly, the training process of a diffusion model for text-to-image (T2I) generation entails substantial computational resources, resulting in increased energy consumption, CO2 emissions, and potential environmental pollution.
Acknowledgements
This research is supported by NExT++ Lab and CCF-Baidu Open Fund.
10

---

References
[1] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne den Berg. Structured denoising diffusion models in discrete state-spaces. In Proceedings of the NeurIPS, pages 17981-17999, 2021.
[2] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel, Multidiffusion: Fusing diffusion paths for controlled image generation. CoRR, abs/2302.0$ 0110,, 2023.
[3] Mohammad Bashiri, Edgar Y. Walker, Konstantin-Kleee Klemens Lurz, Akshay Jagadish, Taliah Muhammad, Zhiwei Ding, Zhuokun Ding, Andreas S. Tolias, and Fabian H. Sinz. A flow-based latent state generative model of neural population responses to natural images. In Proceedings of the NeurIPS, pages 15801- 15815, 2021.
[4] Andrew Brock. Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Proceedings the ICLR, 2019.
[5] Shang Chai, Liansheng Zhuang, and Fengying Yan. Layoutdm: Transformer-based diffusion model for layout generation. CoRR, 2023.
[6] Tuhin Chakrabarty, Arkady Saakyan, Olivia Winn, Artemis Panagopoulou, Yuc Yang, Marianna Apidianaki, and Smaranda Muresan. spy a metaphor: Large language models and diffusion models co-create visual metaphors. OpenReview. 2023.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proceedings of the ICMI., pages 1691-1703, 2020.
[8] Jun Cheng, Fuxiang Wu, Yanling Tian, Lei Wang, Dapeng Tao. Rifegan: Rich feature generation for sext-to-imae synthesis from prior knowledge. In Proceedings of the CVPR, pages 10908 10908-10917, 2020,
[9] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Proceedings of the NeurlPS, pages 8780-8794, 2021.
[10] Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang. and Jie Tang. Cogview: Mastering text-to-image generation via transformers. In Proceedings of the NeurlPS, pages 19822- 19835, 2021.
[11] Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components estimation. In Proceedings of the ICLR, 2015.
[12] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In Proceedings of the ICIR, 2017.
[13] Patrick Esser, Robin Rombach, and Björ Ommer. Taming transformers for nigh-resolution image synthesis. In Proceedings of the CVPR, pages 12873 12883, 2021.
[14] Wan-Cyuat Fan, Yen-Chun Chen. Dongdong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. CoRR, abs/2208. 2121222088.377.... 2022.
[15] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun R. Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training free structured diffusion guidance for compositional (ext-to-image synthesis. CoRR, abs/2212.05033 2250022. 2022.
[16] Sarthak Garg, Helisa Dhamo, Azade Farshad. Sabrina Musatian, Nassir Navab, and Federico Tombari. Unconditional scene graph generation. In Proceedings of the ICCV, pages 16342-16351, 2021.
[17] lan Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets, In Proceedings of the NeurlPS, pages 2672-2680, 2014.
[18] Jiuxiang Gu, Shafiq R. Joty, Jianfei Cai, Handong Zhao, Xu Yang, and Gang Wang. Unpaired image captioning via scene graph alignments. In Proceedings of the ICCV, pages 10322-10331, 2019.
[19] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vcctor quantized diffusion model for text-to-image synthesis. In Proceedings of the CVPR, pages 10686-10650 1066,, 2022.
[20] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-fiee evaluation metric for image captioning. In Proceedings of the EMNLP, pages 7514-7528, 2021.
11

---

[21] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Proceedings the NeurlPS, pages 6626-6637, 2017.
[22] Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic object accuracy for generative text-to-image synthesis. IEEE Trans. Pattern Anal. Mach. Intell., 44(3):1552-1555, 2022.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the NeurlPS, 2020.
[24] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Towards non-autoregressive language models. CoRR, 2021.
[25] Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. CoRR, abs/2303.081377 2023.
[26] Álvaro Barhero Jiménez. Mixture of diffusers for scene composition and high resolution image generation. CoRR, 22422300.02412. 2023.
[27] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia David A. Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In Proceedings of the CVPR, pages 3668-3678, 2015.
[28] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the CVPR, pages 1219-1228 1228, 2018.
[29] Diederik Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the ICLR, 2014.
[30] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, (23(1):32-77) 2017.
[31] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang. Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Yuan, Lei Zhang, neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language- image pre-training. In Proceedings of the CVPR, pages 10955-10965. 2022.
[32] Wenbo Li. Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He. Siwei Lyu. and Jianfeng Gao. Object-driven text-to-image synthesis via adversarial training. In Proceedings of the CVPR, pages 12174- 12182, 2019.
[33] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Proceedings of the ECCV, volume 8693, pages 740-755, 2014.
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings the ICLR, 2019.
[35] Diego Marcheggiani and Ivan Titov. Encoding sentences with graph convolutional networks for semantic role laheling. Proceedings of the EMNLP, pages 1506-1515, 2017.
[36] Victor Siemen Janusz Milewski, Marie-Francine Moens, and lacer Calixto. Are scene graphs good enough to improve image captioning? In Proceedings of the AACL, pages 504-515, 2020.
[37] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the ICMI,, pages 16784-16804, 2022.
[38] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll I. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.08 05555 2022.
[39] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In Proceedings of the ICML, pages 4052-4061, 2018.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the ICML, pages 8748 8763, 2021.
12

---

[41] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proceedings of the ICML, pages 8821-8831, 2021.
[42] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent difffusion models. In Proceedings of the CVPR, pages 10674-10685, 2022.
[43] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the MICCAI, pages 234-241, 2015.
[44] Tim Salimans, Ian J. Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proceedings of the NeurlPS, pages 2226-2234, 2016.
[45] Brigit Schroeder and Subarna Tripathi. Structured query-based image retrieval using scene graphs. In Proceedings of the CVPR, pages 680-684, 2020.
[46] Sebastian Schuster, Ranjay Krishna, Angel X. Chang, Li Fei-Fei, and Christopher D. Manning. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the EMNLP, pages 70-80, 2015.
[47] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proceedings of the ICLR, 2021.
[48] Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Fei Wu, and Xiao-Yuan Jing. DF-GAN: deep fusion generative adversarial networks for text-to-image synthesis. CoRR, abs/2008.05865, 2020.
[49] Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. Proceedings of the NeurlPS, pages 4790-4798, 2016.
[50] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proceedings of the NeurlPS, pages 6306-63155 2017.
[51] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. P+: extended textual conditioning in text-to-image generation. CoRR, 000022 2023.
[52] Tao Xu, Pengehuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngaa: Fine-grained text to image generation with attentional generative adversarial networks. In P Proceedings of the CVPR, pages 1316-1324, 2018.
[53] Ling Yang, Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming-Hsuan Yang. iiffusion-hased scene graph to image generation with masked contrastive pre-training. CoRR, abs/2211. 11188 2022.
[54] Xu Yang, Kaihua Tang. Hanwang Zhang. and Jianfei Cai. Auto-encoding scene graphs for image captioning. In Proceedings of the CVPR, pages 10685-10694, 2019.
[55] Xu Yang, Chongyang Gao, Hanwang Zhang, and Jianfei Cai. Hierarchical scene graph encoder-decoder for image paragraph captioning. In Proceedings the ACM MM, pages 4181-4189, 2020.
[56] Sangwoong Yoon, Woo- Woo-Young Kang, Sungwook Jeon, SeongEun Lee, Changjin Han, Jonghun Park, and Fun-Sol Kim. Image-to-image retrieval by learning similarity between scene graphs In Proceedings of the AAAI, pages 10718-10726, 2021.
[57] Jiaxuan You, Rex Ying, Xiang Ren, William L. Hamilton, and Jure Leskovec. Graphrnm: Generating realistic graphs with deep auto-regressive models. In Proceedings of the ICML, pages 5694-5703, 2018.
[58] Xiang Yu, Ruoxin Chen, Jie Li, Jiawei Sun, Shijing Yuan, Huxiao Ji, Xinyu Lu, and Chentao Wu. Zero-shot scene graph generation with knowledge graph completion. In Proceedings of the ICME, pages 1-6, 2022.
[59] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the CVPR, pages 5831-5840, 2018
[60] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In Proceedings of the CVPR, pages 586-595, 2018.
[61] Minfeng Zhu, Pingho Pan, Wei Chen, and Yi Yang. DM-GAN: dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the CVPR, pages 5802-5810, 2019.
13

---

A Extension of Technical Details
We in this part extend the specific details of our method techniques.
A.1 Discrete Diffusion Model for Scene Graph Hallucination
Here, we detail the forward and reverse processes in the discrete diffusion model [1] for SGH.
Forward Process In the forward process, we consider node $i E G2, * fo, r) (t is diffusion step. j the node index) takes scalar discrete value with K** categories, si E E1, KK* Without introducing confusion, we omit subscripts j and superscripts * in the following description. We define the probabilities that 8t-1 transits 8t using the matrices [Qilis q(x = j|s i), then we can write the forward Markov diffusion process:
((s((sr1)) B (s:)Q.B(s--1),
(10)
where ((ss) denotes the one-hot column vector which length is K. The categorical distribution over 8, is given by the vector QrB(st-1). Due to the property of the Markov chain, starting from 8g, we derive the cumulative probability of 8, arbitrary (-timestep:
(5 (s0) = 5 (#+) Q: ((0)). where Q, = Q-Q-1 ... Q1
(11)
Besides, by conditioning on 80. the posterior of this diffusion process tractable:
g(st st- so)g (so)
(B" (s. (sx)).B((s. ()) (s (0r=))))(((s) B (s: Q.B((u)))
q(5t-1186, s0)
(12)
g 50)
Note that the transition matrix Q: is capable of controlling the data corruption and denoising process, thus should be carefully designed such that it is not too difficult for the reverse network to recover the signal from noises. We follow [1, 19] that exploits mask- -ad---ndd---ee strategy to design Qi which can be defined as:
To + 8
8t
Be
0
8
+
3c
8
0
Q:
(13)
T
76
where for each node St, there are three probabilities, i.e., a probability of n to transition to [MASK] node, a probability of K 3, be resampled uniformly over all the categories, and a probability of ar = 1- K3, Th to stay the same node. Notedly, the [MASK] node never transitions other states. The aforementioned discrete state-space models assume that all the nodes are switchable by corruption. However, it is unreasonable that an object node, 'bed', transitions to a relation node, 'in'. To avoid such invalid transition, we propose to apply disjoint corruption matrices Q Qi Q for object, attribute, and relation nodes, respectively.
Reverse Process In the reverse process, an SG decoder is introduced as a denoising network to estimate the posterior distribution po(-). which takes the node token 8t, time step / and text prompt y. Specifically, each layer of the SG decoder contains two parts: 1) an adaptive normalization (AdaLN), and 2) a transformer block. AdaLN is applied to inject the timestep information:
H = AdaLN( f) = a, (CCLIer)) + b.,
(14)
where CLIP(-) denotes an encoding layer for projecting the node token, and a, and b, are obtained from linear projection of the timestep embedding. Each transformer block contains 1) a full self-attention, i.e., H Softmax( --T -H. 2) text-cross-attention (Text-CA) integrates the VTz
input prompt y:
H H HJ
H' Text-CA(y. H)) Softmax
(15)
Vdi
where H is a conditional feature sequence yielded by text encoder first takes the text prompts y as input. Then, 3) graph cross- attention (Graph-CA) is devised to take the induced SG (G++1) in the previous f timestep:
H'-H G
H Graph- H') Softmax
E.).HO
(16)
dd
14

---

The whole SG
Global level
Region level
wear
front of
caryiigg on
wear
Relation level
Tshirt
building
man
luggage bag sunglass
Object
black
old
young
SG
Figure 9: Different semantic levels in the SG: objects, relation, region, and global levels.
where H and E is the embedding representation of the nodes and edges in G++1, respectively, Next, 4) node-type dependency cross-attention (NTD-CA) is designed for the *-type node induction (* can be object or attribute):
NTD-CA(s$: H)) Softmax(
AM" ('((CIIP(S))
Vdn
CLIP(s))
a
(( (CLIP(()) @ CLIP(s))). ==r
NTD-CA(s @ s H)) = Softmax
Väj
(17)
Finally, softmax function is put to obtain the category label distributions:
87 Softmax(H") (ff").
(18)
Following [1. 19], the SG decoder is optimized by minimizing the variational lower bound (VLB):
fuu Lo + L1 + ... LT-1 Lr,
(19)
Co = -logpo(8o(s1.
(20)
C DKI((g(st-1) Se, sp)) v)),
(21)
C = D (g(3)))))
(22)
where po (ST) the stationary distribution at timestep To estimate the (S(T)), we extend the
forward process by appending rank-one matrix Q++1 that ignores sT and produces a deterministic 8T+1 E GT+1 where GT+1 is the initial SG parsed from the text prompt y, then learning the reverse step (s [8T+1) (0(5)18T+1)
Po (sTr). ). Furthermore, we follow the reparameterization trick proposed in [19], which lets the network predict the noiseless token distribution po po y) each reverse step so as to gain better quality. Based on the reparameterization trick, an auxiliary objective is introduced:
L logpo (so 8t; y)
(23)
To combine this loss with Letb, the final training objects are defined as follows:
LSGH Lub + Cz
(24)
where X₁ is hyper-parameter to control the effect of the auxiliary loss L.g.
A.2 Scene-driven Image Synthesis
As described in $3.2, we propose hierarchical scene integration strategy effectively integrate the SG features. Here, we give more details on this part. Firstly, as shown in Figure 9, the SG is able to represent four different hierarchical levels of semantics corresponding the image:
Object level. In the process of image synthesis, a crucial aspect lies accurately generating each specified individual object, corresponding to the object node and associated attribute notes in SG. for example, black Tshirt, old building, young man, luggage bag. sunglass.
Relation level. A high-quality image is not only the generation of objects but also their intricate relationships which connect two objects, akin to the ceapjecc-peeicatee--e wear Tshirt, man in front of building,
triplets found in carrying on luggage bad,
the SG, such wear sunglass.
Region level. Regional image generation focuses on multiple objects and entangled relation- ships among them, which aligns with the presence of overlapping relation triplets' in SG. Here adopt two overlapped relation triplets the region representation of the image, such as man wear Tshirt in front of building, man wear Tshirt carrying on luggage bag, man wear
Two overlapped relation triplets means there the same subject object in the relation triplets.
15

---

Thirt wear sungalss, man in front of building carrying on luggage bag, man in front of building wear sunglass, man carrying on luggage bag wear sunglass.
Global level. The whole SG provides global semantics to guide the generation of images.
Then, we technically extract collection of concepts from the four semantics levels in the SG, denoted as L (object level), L" (relation level), L (region level), and L (global level). We encode each concept separately:
Uy = (u, u2 .. : }. " CIJP(())
(25)
where Cj E 1,' ; 1,,,
(o,r,c,g)
We further maintain the representation of these four levels as the keys Kgs and values V2 by linear transformations:
L Linear(U V2= = Linear( (UL),
(26)
where i E fo, r, c.. Next, we integrate these features via the Transformer attention of U-Net in LDM, which is formulated in Eq. (7).
B Detailed Experiment Settings
B.1 Datasets
COCO The training and validation data numbers in COCO are 83K and 41k, respectively, We note that, in the evaluation phase, models are evaluated the full COCO 2014 validation set.
Visual Genome (VG) Visual Genome [30] version 1.4 (VG) comprises 108,077 images annotated with scene graphs. Following previous work [28]. we use object and relationship categories occurring the dataset, leaving 178 objects and 45 relationship
least 2,000 and 500 times respectively types, and we ignore small objects, and use images with between 5 and 30 objects and at least three relationships, this leaves us with 62,565 images with an average of 10 objects and 5 relationships per image.
Construction of COCO-A21 We elaborate on the process of constructing the COCO-A2I dataset in the following three steps:
First, we consider Place Nouns and Progressive Verbs are two types of abstract words that can depict intricate images. Therefore, we pre-define the candidate list of Place Nouns: street, sidewalk, kitchen, restroom, bathroom, living room, bedroom. hostel, house, office, bank. and the candidate list of Progressive verbs: traveling, knitting, gardening, shopping, presenting, drawing, baking, studying, etc.
Second, we select the COCO valid dataset in that captions of instances contain the words in dataset.
the candidate list, obtaining the primary-filterin
Third, we filter out the primary- filteriing dataset in which the number of words in the captions of instances is more than 10 and the number of bounding boxes is less than 6, obtaining the final target COCO-A21 dataset.
After the three-step pipeline, we obtain 2,005 text-image pairs.
In Figure 10, we show some examples, where the images tend to contain intricate content, including multiple objects, attributes, and relationships while the corresponding text prompts are relatively short and abstract.
B.2 Evaluation Metric Implication
We employ Fréchet Inception Distance (FID) [21], Inception score (IS) [44], CLIP score [20], and GLIP [31] used in [14] to quantitatively evaluate the quality the generated images, and Learned Perceptual Image Patch Similarity (LPIPS) [60] utilized in [42] to quantify the diversity of the generated images. Additionally, we introduce Triplet Recall (TriRec.) to measure the percentage of the correct relation triplet among all the relations. Technically, given set of ground truth triplets preeect-predicut--bjjeee denoted GT, and the TriRec. computed as TriRec.=|PTT n GT///GT) where PT are the relation triplets extracted from the generated images by visual SG parser.
16

---

lew bags laying around in a living room.
bus tumning a comer on a city street.
holding baby up in a kitchen.
A room with two desk covered in computer equipment.
A kitchen scene with an oven and stove.
A lamily is sitiing around dinner table.
A man sits in diner photographing his meal.
A bathroom sink sitingg undemnath mirror a bathroom.
Figure 10: The examples from COCO-A21 dataset for evaluating abstract-to-intie -nntrrccate T2I generation.
B.3 Human Evaluation Criterion.
In $4.5, we conduct human evaluation, i.e., user study, for more intuitive assessments of the diversification quality of T2I models. Specifically, we generate 10 images conditioned on a single text prompt and ask evaluators cluster these images based their contextual information, such as objects, attributes, and relationships present in the generated images. After evaluating 100 distinct text prompts, we compute the average clustering ratio:
Cluster
Average Clustering Ratio = Average
(27)
10
where Cluster denotes the number of clusters for 10 images generated by a text prompt. Average an average function. Here, higher ratio score signifies greater diversification in image generation.
C More Experiment Results
C.1 Visualizations of SGH Process
In Figure 11, we visualize the SGH process with two examples, where we sample several time steps and plot the generated SGs and the corresponding images. As can be seen, our SGH is capable of imaging certain reasonable structures SGs to enrich the initial SG.
C.2
More Cases
Here we showcase more examples of the generated images via the scene sampling mechanism in Figure 12 and 13. We synthesize 10 images for each given text prompt.
17

---

Text P Prompt (a): fancy chandelier hanging in a small kitchen.
--
100
Oin
2
70
60
50
40
i 20
10
window
window
Fancy
window
haadinng
handing in
has
big handing in has
hig
handing
chandelier
kitchen
chandelter
kitchen
chandelier
kitchen in
Chandelier
kiliceen
glass
fancy
small
in
in (small (tabla)
in in smalll
black
small
chaici
open
Ihaar babll owen
chair
table
oven
nuxt to
next on flwer
next
Hlwwr
white
Text Prompt (b): small room with painting stuffs.
--100
- 90
-
--
70
6-
50
- 40
[ 20
10
with
walls
walls
painting
walls
painting
painting
r00m
stuffs
with casel
with
Busel
window
with easel wooden
Windoww
small
painting
room
In front of
in front of beside
front of beside
white
in couch
small in
couch
in
small messy
couch
floor
on
00
floor
Dn
an
6n
green
brush
bnish
black
red)
Figure 11: Visualization of T21 generation process at t=100, 90, ., along with the enriched SG t=100, 70, 50, 20.
18

---

Text Prompt: Sparse narrow kitchen.
Text Prompt: person is fishing.
Text Prompt: father and play Wii in the living room.
Text Prompt: A bed in a room.
Figure 12: More samples of -oirracctt- T2I generation via scene sampling mechanism.
19

---

Text P Prompt: person is baking.
SL
000
Text P Prompt: A car and traffic light on city street.
Text Prompt: A living room for painting.
Text P Prompt: A giraffe in the zoo.
Figure 13: More generated samples by our model.
20


Reviews:
[{"review_id": "zXJMUxsMrB", "reviewer": "Reviewer_KZmg", "summary": "This paper aims at the abstractive setting for text-to-image generation (T2I). They propose scene-graph hallucination (SGH) to perform imagination over the scene graph of the input prompt and make up the missing information. Then SGH can perform better T2I with the completed scene graph. Experiments on the COCO dataset demonstrate that T2I can significantly bridge the gap of abstract-to-intricate T2I.", "questions": "I do not have additional questions. Please see the Weakness and refer to other reviewers.", "limitations": "The hallucination issues over scene graphs should be carefully addressed.\n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ This paper is well-written and easy to follow.\n+ The goal of abstract-to-intricate T2I is important since we can not expect users always input their full prompt. This setting has the potential to improve practical applications.\n+ The usage of scene graphs is effective and can lead to better imagination for T2I than the Chain-ot-Though (CoT) language prompt.\n+ They provide detailed ablation studies from different aspects (Table 3/4 and Fig. 6/7) as well as rich qualitative examples (Fig. 5 and Fig. 12/13 in Appendix).", "weaknesses": "- Maybe I missed it, but what is the motivation for using scene graphs instead of LLM-completed prompts? Does the imagination over scene graphs work more robustly than LLM?\n- Since they rely on imagination to deal with abstractive issues, the hallucination situation also raises. It will be better to evaluate this issue, and how is the trade-off between this and the T2I performance. How to avoid or mitigate this deficiency in the proposed framework?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for taking the time to provide valuable feedback! especially for the recognition of our work, such as 'well-written and easy to follow', ‘goal of abstract-to-intricate T2I is important’, and 'effective', as your support means a lot to us. Following, present the point-to-point response as follows to address your concerns. \n\n---\n\n**Q1: Maybe I missed it, but what is the motivation for using scene graphs instead of LLM-completed prompts? Does the imagination over scene graphs work more robustly than LLM?**  \n\n **A:** The adoption of scene graphs over LLM-completed prompts is driven by two primary motivations. Firstly, employing scene graphs facilitates greater content controllability. LLM-completed prompts often introduce additional adjectives, attributives, or concatenate raw sentences to provide tangible explanations and contexts. However, such supplementary content inserted by LLMs lacks controllability, potentially leading to the inclusion of more abstract words, as exemplified by 'enthusiastic' and 'confident' in Figure 1, finally posing more challenges for downstream generative models. In contrast, scene graph-based imagination allows for enhanced visual controlling and a stable process, as the hallucination process entails expanding the initial scene graph with specific scene structures, where imaged objects, attributes, and relationships are drawn from limited pre-defined sets. \n\nSecondly, employing scene graphs reduces the difficulty of downstream image generation. Scene graphs offer a concise and precise means of depicting objects and their interrelationships, empowering fine-grained control over the semantic scene during the image generation process.\n\n---\n**Q2: Since they rely on imagination to deal with abstractive issues, the hallucination situation also raises. It will be better to evaluate this issue, and how is the trade-off between this and the T2I performance. How to avoid or mitigate this deficiency in the proposed framework?**   \n\n **A:** As addressed in **Q1**, our scene graph imagination process aims to enrich the initial scene graphs by incorporating more specific scene structures, where objects, attributes, and relationships are selected from limited pre-defined sets, facilitating controlled imagination and mitigating undesired hallucination occurrences.\n\nTo evaluate the proposed scene graph imagination module whether induce reasonable scene graphs (SGs), we actually have assessed the structure alignment, specifically the recall rate (TriRec.) of the ‘object-object’ pairs, between the induced SGs and the SGs of gold images known to represent reasonable scenes. Table 4 shows that 82.01% of 'object-object' pairs in the induced SGs exhibit high alignment with those in the gold SGs, signifying the capacity of our model to synthesize valid and coherent scene content.\n\nMoreover, the impact of the hallucination issue in T2I generation is relatively less significant compared to LLMs. In our T2I generation, hallucinations are relatively acceptable, particularly when dealing with abstractive scenarios, as demonstrated by experimental results. In contrast, LLMs, having no constraint, tends to produce more uncontrollable and undesirable content hallucinations. That being said, the hallucination issue does need to be addressed in some scenarios in our system, such as in the generation of images that emphasize real scenes, which is also a potential research direction for our further work."}, {"review_id": "9j2cmUz7lk", "reviewer": "Reviewer_DqEY", "summary": "The paper proposes a new setting (or sub-domain) of text-to-image generation (T2I), namely abstract-to-intricate T2I. To tackle the new setting, the authors propose a method (Salad) to enrich the scene graphs parsed from the text prompts based on the discrete diffusion models. The enriched scene graphs are used as guidance to generate complicated scenes that align better with the initial concise prompts. They report quantitative results and analysis experiments to show the effectiveness of the two-stage system in multiple metrics.", "questions": "- What is the SGH module's output and the SIS step's input? Fig. 2 seems like you are inputting the scene graph matrix into the UNet, while Sec. 3.2 and Fig. 3 show that you convert the graphs into prompts and feed them to the attention layers.\n\n- Did you fine-tune the Stable Diffusion model by saying \"For the SIS module, we load the parameters of Stable Diffusion3 (v1.4) as the initialization.\"? \n\n- Why do all models achieve such high CLIP scores? Is the CLIP score the cosine similarity between image-text pairs or the CLIP R-precision metric?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses a practical and important problem. Users of the T2I system, in reality, have to write unreasonably long text prompts to generate images with plausible scenes/styles/semantics. Abstract-to-intricate T2I could benefit the users with a more faithful and efficient generation process. \n\n- The scene graph hallucination stage achieved with the discrete diffusion model is interesting. Enriching the text prompt in the scene graph space seems like a more controllable and stable process that can preserve the faithfulness of relations and attributes. \n\n- The authors conduct an extensive analysis of the components of the system to demonstrate its effectiveness.", "weaknesses": "- Task definition. While I understand the concept of abstract-to-intricate, I think the work lacks a more rigorous definition of the task. It seems unclear why places and progressive verbs are included and other nouns or verbs are discarded in the COCO-A2I. I tried the failure prompt in Fig. 1 on Stable Diffusion, and I got good results most of the time. So what makes these prompts an A2I problem instead of a faithfulness problem of all T2I models?\n\n- My major concern is about the experimental setup. It seems that the Salad system is using the pre-trained Stable Diffusion as the image generator compared to other methods like Frido and LDM-G. The comparison could be unfair as all these baselines are trained with much fewer images and inherently have higher FID scores on MSCOCO. In addition, the authors use Frido for text-based enrichment approaches. \n\n- The writing needs improvement. There are multiple typos throughout the paper and some inaccurate sentences. For instance, line 133 \"attribute (o)\", line 226 \"baselines: stable diffusioninspired by [6].\", line 235-236 \"For the SIS module, we load the parameters of Stable Diffusion3 (v1.4) as the initialization.\", line 238 \"UNet\" (UNet in Stable Diffusion?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for going through our paper carefully, and providing valuable constructive feedback, which will surely benefit our work. Following we will address your concerns. And we sincerely hope you can raise your evaluation when you feel that we relieve your concerns.\n\n---\n**Q1: Task definition. I think the work lacks a more rigorous definition of the task. It seems unclear why places and progressive verbs are included and other nouns or verbs are discarded in the COCO-A2I. I tried the failure prompt in Fig. 1 on Stable Diffusion, and I got good results. So what makes these prompts an A2I problem instead of a faithfulness problem of all T2I models?**\n\n**A:** Thank you for pointing this out. Here we can give a rigorous definition of the setting. This work is dedicated to improving the quality of intricate image generation conditioned on succinct abstract prompts, where the prompts are concise and highly abstract and often encapsulate complex scenes. Actually, abstract words have many categories, such as _place noun_, _progressive verb_, _social events_ (e.g., party, concert, conference), and _experience nouns_ (e.g., adventure, journey, festival). For the COCO data, we conduct the pilot study and found that the two types of abstract words, _place nouns_ and _progressive verbs_, are the most common, with a proportion of up to 95% and other types of abstract words account for a very small portion. Thus, we mainly selected these two types for COCO-A2I dataset.\n\nThe results in Fig1 were obtained by the Stable Diffusion (v1.4) checkpoint at the time we were doing our project. Currently, the SD model has been updated several times as it trained on more datasets, achieving better results. Therefore, if you try it now or recently, you may have a chance to obtain good results. That being said, there is no guarantee of good performance for using SD on those abstract prompts. To gain an empirical result, we randomly select 50 abstract prompts from the COCO-A2I dataset, and input them into the latest Stable Diffusion (v2.1), with only a success rate of generating satisfactory results of 54%. Therefore, enabling T2I generative models with imagination ability is still non-trivial. At the same time, we acknowledge that the A2I problem is inherently a problem of addressing the faithfulness of T2I models.\n\n---\n**Q2:  My major concern is about the experimental setup. Salad system is using the pre-trained Stable Diffusion as the image generator compared to other methods like Frido and LDM-G. The comparison could be unfair as all these baselines are trained with much fewer images. In addition, the authors use Frido for text-based enrichment approaches.**\n\n**A:** It is our oversight for being unfair. We were just following the common practice of latent stable diffusion, i.e., leveraging the pre-trained stable diffusion to ensure the stability of training and the diversity of its application scenarios.\n\nThat being said, to gain a fair comparison, during the rebuttal days, without loading the checkpoint as the initialization, we re-train our model on the same COCO training set as baselines. The results are shown as follows, and we find our conclusions still hold water. The superiority of our proposed Salad model is still significant, especially on the COCO-A2I dataset.\n\nResult on COCO:\n| Model | FID | IS | CLIP|\n| :-----: | :----: | :----: | :----: |\n|Frido | 11.24 | 26.84 |70.46|\n| Ours | 10.57 | 28.16 |71.37|\n\nResult on COCO-A2I:\n| Model | FID | IS | CLIP|\n| :-----: | :----: | :----: | :----: |\n|Frido | 40.36|18.36|68.53|\n| Ours | 33. 61|24.16|70.34|\n\n---\n**Q3:  The writing needs improvement. There are multiple typos throughout the paper and some inaccurate sentences.**\n\n**A:** We appreciate it much that you went through the paper carefully. We will carefully correct them all.\n\n---\n**Q4:  What is the SGH module's output and the SIS step's input? Fig. 2 seems like you are inputting the scene graph matrix into the UNet, while Sec. 3.2 and Fig. 3 show that you convert the graphs into prompts and feed them to the attention layers.**\n\n**A:** Sorry for causing the confusion, and we will provide a clearer description in revision.\n1) The output of the SGH module is three node matrices, i.e., object nodes $s_t^o$, the attributes nodes $s_t^a$, and relation nodes $s_t^r$, as shown in the right bottom of Fig 2. The value of three matrices is the label of each node. We can convert the three matrics into a scene graph without the need for any further calculations.\n2) The SIS step takes as input a sequence of tokens derived from the scene graph at different semantic levels. For more details, please refer to Appendix B.2.\n\n---\n**Q5:  Did you fine-tune the Stable Diffusion model by saying \"For the SIS module, we load the parameters of Stable Diffusion3 (v1.4) as the initialization.\"?**\n\n**A:** Yes, we load the SD (version 1.4) weights, and then fine-tune it with our SGH module on the COCO data.   \nThe merits of employing the off-the-shelf checkpoint as the initialization are multifaceted. Firstly, based on the existing pre-trained parameters, we can achieve more stable training and faster convergence. Second, reusing the existing SD weights has been the default practice in the community, which also helps avoid repeating training the backbone from scratch.\n\n---\n**Q6:  Why do all models achieve such high CLIP scores? Is the CLIP score the cosine similarity between image-text pairs or the CLIP R-precision metric?**\n\n**A:** The CLIP score [1] is defined as the cosine similarity between image-text pairs, as employed in the baselines. Specifically, for a generated image with visual CLIP embedding $v$ and an input prompt with textual CLIP embedding $c$, the CLIP score is computed as: $CLIP(v, c) = w * max(cos(v, c), 0)$, where $w$ is a re-scaling parameter. We used the officially-released code for accurate evaluation.\n\n[1] CLIPScore: A Reference-free Evaluation Metric for Image Captioning. EMNLP 2021"}, {"review_id": "JeM5kPHII4", "reviewer": "Reviewer_mzvT", "summary": "The paper proposes a novel approach to text-to-image (T2I) synthesis, specifically focusing on generating intricate visual content from simple abstract text prompts, aka., abstract-to-intricate (A2I) setting. The proposed mechanism, named scene-graph hallucination (SGH), expands the initial scene graph (SG) of the input prompt with more feasible specific scene structures using discrete diffusion technique. A continuous-state diffusion model then serves as the T2I synthesizer navigated by the semantic scene structure induced from the SGH module. Additionally, this paper further devises a scene sampling mechanism to generate various scene graphs. They also construct a more challenging benchmark data, called COCO-A2I, to effectively evaluate the models under the abstract-to-intricate setting. Experiments on two benchmark datasets shows that leveraging SG imagination helps better image generation, where Salad could hallucinate certain scene clues to facilitate the abstract-to-intricate T2I generation. The study also contributes to a better understanding of the efficacy and rationale behind scene graph features, with potential applications in other tasks such as image editing and text-to-video generation.", "questions": "1. Is there any further explanation about the order of the Text-CA, Graph-CA, NTD-CA?\n2. During the SG hallucination, other visual scenes are also introduced. How does the proposed method deal with the visual distraction issues?\n3. How to ensure that the objects, attributes, and relationships in the initial SGs are always remain in the imaged SGs? \n4. Has the main results in Table 1&2 adopted the scene sampling mechanism? So is this the average of multiple experimental results?\n5. In HIS, which semantic level has a significant impact on task performance?\n6. How to leverage the VG dataset to train the SGH module?", "limitations": "I do not foresee any potential negative impact from this work.", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Overall, I enjoy the work much where it studies the specific conditioned image synthesis from an interesting and realistic perspective, with robust and novel technical methods. The paper seems very solid to me, with thorough evaluations from many angles, and the details are given quite sufficiently (with long and informative appendix content). While this is a very technical paper, there is immense interest in diffusion models under such novel perspective. I believe this paper will have the potential to unlock interesting future research.\n\n**Interesting and meaningful perspective.**\nThe paper studies how to generate intricate images from succinct abstract prompts. This can be a very interesting and realistic perspective in many scenarios. The issue essentially lies in the natural modality asymmetry between language and vision, where strict one-to-one correspondences do not exist. This becomes particularly challenging when T2I systems attempt to capture the nuanced content of the prompts and generate corresponding high-quality images, thereby highlighting the need for deep semantic understanding in these systems.\n\n**Innovative methodology.**\nThe authors propose a novel approach to the task of abstract-to-intricate T2I synthesis, where the image-generating process is controlled and navigated by the underlying semantic scene structure. On the one hand, it satisfies the human intuition for handling this task, which is pretty interesting and makes a lot sense to me. If the system could hallucinate some concrete textual clues which has a more corresponding relationship to the visual scenes, the generation process will ease largely. Technically, the devised method, called scene-graph hallucination (SGH), expands the initial scene graph (SG) of the input prompt by iteratively evolving new scene elements via discrete diffusion model, which are theoretically sound and empirically validated through experimental results. On the other hand, the hierarchical scene integration mechanism is able to ensure the highly effective guidance of the semantic scene features.\n\n**Solid evaluations and convincing analyses.**\nThe results are presented clearly and concisely, showcasing a significant improvement for the abstract-to-intricate T2I task at hand. Some in-depth analyses are provided to offer substantial evidence and valuable insights from all different perspectives to support the claims made, such as, exploring the effectiveness and rationale behind the employed scene graph features. Also, the comprehensive implementation details will enhance reproducibility and facilitates the smooth adoption of the proposed approach. The codes are provided.\n\n**Well-structured presentation and clear writing.**\nThis paper is exceptionally clear, well-written (except some notation problems) and good illustration, making it easy to understand. The details are given quite sufficiently, with long and informative appendix content. I’ve gone through the appendix and find almost everything I wanted to see.", "weaknesses": "While I think the paper is solid, it can be significantly improved further if the following issues (major or minor) are considered properly:\n\n(ordered by appearance in the paper)\n\n1. In Figure 1, the issues of vision distraction and wrong binding appear to represent the same issue where the resulted images deviate from the original user intention. Besides, it would be beneficial to maintain consistent terminology throughout. For example, in Figure 1(a), it is referred to as 'vision distraction', whereas in the article, it is mentioned as \"visual distraction.\" \n\n2. Although the authors devise a scene sampling mechanism to generate various scene graphs for diversified image syntheses during inference, the diversity of scene graph remains inadequate. This could be due to the limited number of categories of objects, attributes or relationships in the SGH module, which restricts the model's imagination and exploration capacity. In real scenarios, the objects, attributes can be quite multifarious, for example, the color of the T-shirts.\n\n3. Potentially lack of comparison of the existing scene graph completion work. The performance of scene graph hallucination has a huge impact on the performance of abstract-to-intricate T2I task. It would be interesting to see the performance comparison on existing work about scene graph hallucination, such as [1,2]. Moreover, another intuitive method is the pipeline method, i.e., first scene graph hallucination and then scene graph generation based the imagined SG. Therefore, some evaluating experiments should be provided to make a comparison with the pipeline method.\n\n4. The authors may possibly overlook the imagination ability of the T2I model itself. For example, existing T2I models can synthesis an image based on the prompt, 'a man'. Is the appearance of the man in the generated image, such as hair color, attributed to the imagination of the T2I model?  In other words, the authors fail to provide a clear definition of when the model needs to image. Furthermore, they do not delve into what scene graph needs to be detailed in order to generate intricate images effectively. For instance, in Figure 1(b), the enrich SG not extremely correspond to the image as the screen on the wall are not demonstrated in the enriched SG, but the model is still able to generate it.\n\n5. Some unclear and confusing annotations:\n\n- In Figure 2 right bottom, it is confusing $a_{n, 1}$ whether one value, i.e., the m-th attribute of the object $o_1$, or many values. Same issues for $r_{n,k}$.\n\n- In line 150, the distribution should be $p_{\\theta}(s_t|s_{t+1}, y)$.\n\n- Is $\\mathcal{B}{s_t}$ a row one-hot vector or column one-hot vector? This should be clarified. Besides, there should be a transition symbol in Eq. 2.\n\n- In Eq. 1, the $\\mathcal{L}_{vlb}$ is not explicitly designed for the optimization of conditional image generation. Thus, some clarification should be added, and the correct loss function is demonstrated in Appendix B.1, specifically Equations 19-22.\n\n- No illustration of the $d$ in Eq. 7.\n\n- In the Implementation section, the version of CLIP mentioned is not consistent with the information provided in Appendix C.5. \n\n- No demonstration of the hyper-parameters? top-A and the temperature $\\eta$ in the inference?\n\n- In Table 3, the NTD-CA should be Eq. 5, not Eq. 17\n\n- In Line 314, not ‘object-object’ pairs but ‘subject-predicate-object’ triplets.\n\n- In Line 738, ‘2.005’ should be written as ‘2,005’.\n\n- In Table 5, the second ‘Max’ should be ‘Avg.’\n\n\n6. The paper lacks in-depth analysis of the circumstances under which the proposed SGH module fails to generate reasonable scene graphs, leaving a gap in understanding its limitations.\n\n7. In the section 4.5, the question Q2 is interesting but strikes me as a rather weird way to answer the question. The Figure 7 precisely answers that the SGH is able to induce intricate SGs. As discussed in Q1, there is a strong semantic alignment between the input prompts and the generated images guided of the SG. Consequently, it becomes relatively straightforward to achieve a high TriRec. Score by comparing the induced SG with the SG of the generated image. Nonetheless, this response lacks directness and comprehensiveness regarding whether SGH is capable of producing reasonable SGs. Besides, the concept of a \"reasonable\" SG remains ambiguous, For example, if the ‘table-in-room’ is a reasonable scene triplet, what about the triplet ‘table-in-ocean’? If a reasonable SG means some SGs that conforms natural conditions, can our proposed model generate some abstract or unconventional images?\n\n\n[1] Garg S, Dhamo H, Farshad A, et al. Unconditional scene graph generation. 2021.\n\n[2] Agarwal R, Chandra T S, Patil V, et al. GEMS: Scene Expansion using Generative Models of Graphs. 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate it so much that you went through our work so deep. And we are very much excited to receive your such strong support, which will definitely push us moving forward. Following we will answer your questions one by one.\n\n---\n**Q1:  In Figure 1, the issues of vision distraction and wrong binding appear to represent …**\n\n**A:** Our aim is to analyze this phenomenon at a detailed level, as while the problems in the generated images are alike, there are nuanced variations in their underlying causes. We appreciate your careful review, and in revision, we will ensure the consistent use of terminology.\n\n---\n**Q2: Although the authors devise a scene sampling mechanism to …, the diversity of scene graph remains inadequate.**\n\n**A:** Based on the current experimental results, we see that the imagination module has good imagination capacity, which can help generate intricate images from abstract prompts. However, we also acknowledge that the model's imagination and exploration capabilities are limited. As part of our future work, we will explore how to alleviate this limitation.\n\n---\n**Q3: lack of comparison… Moreover, experiments should…**\n\n**A:** Following [2], we compute the Maximum Mean Discrepancy (MMD) for node and edge types, as well as sub-graph similarities (referred to as NSPDK*) to compare our method with baselines in scene graph expansion. As shown in the table, our method consistently outperforms the baselines, indicating graphs generated by our model are more meaningful and better align with  the observed SG distribution.\n |Matrix|GraphRNN|SceneGraphGen|GEMS|Ours|\n| :-----: | :----: | :----: | :----: | :----: |\n| Node ($\\times 10^4 \\downarrow$ )|5.44|5.92|5.19|4.85|\n| Edge ($\\times 10^2 \\downarrow$)|22.38|0.83|1.13|0.63|\n| NSPDK* ($\\times 10^2 \\downarrow$)|22.60|0.73|1.21|0.69|\n\n---\n**Q4:  The authors may possibly overlook the imagination ability of the T2I model itself …**\n\n**A:** As discussed in Q2, our work lies in enhancing generative models' capacity to comprehend abstract text by envisioning key objects that concretize the abstract text, rather than exhaustively generating all image details. Additionally, we take the imagined SGs as skeletons, which means the SGs and images are not in one-to-one correspondence. Consequently, it is possible that the enriched scene graph does not precisely match the generated image.\n\n---\n**Q5:  Some unclear and confusing annotations.**\n\n**A:** Thank you so much for the careful reviewing, we will revise them and further carefully double-check all the content.\n\n---\n**Q6:  The paper lacks an in-depth analysis … leaving a gap in understanding its limitations.**\n\n**A:** Because of dataset constraints, our method encounters difficulties with infrequent abstract words that prove challenging to concretize using specific key objects, like 'yearn' and 'freedom' in the prompt 'A person who yearns for freedom'. resulting in the failure of high-quality image generation. Actually, we outline a potential vulnerability assessment in Appendix A.3, please refer to it.\n\n---\n**Q7: In section 4.5, the question Q2 is interesting but strikes me as a rather weird way to answer the question…**\n\n**A:** Firstly, we assume that the SGs of gold images entail reasonable scenes. Therefore, if the SGs induced by SGH exhibit a high alignment with the SGs of the gold images, i.e., achieving a high TriRec. score, it indicates the capability of SGH to induce reasonable SGs, directly addressing the question Q2. \n\nSecondly, our proposed method, due to the scene sampling strategy, has the potential to generate unconventional images. This is because we take the TOP-A category candidates and perform sampling over these candidates instead of always picking the best prediction of the category of any node during inference.\n\n---\n**Q8: Is there any further explanation about the order of the Text-CA, Graph-CA, NTD-CA?**\n\n**A:** In the experiment, we also devise the other orders between three condition injection modules. Based on the experiments, we observe that the order illustrated in Figure 2 shows the best performance for SG hallucination.\n\n---\n**Q9:  During the SG hallucination… How does the proposed method deal with visual distraction issues?**\n\n**A:** In this work, there are two approaches to address visual distraction issues. The one is rational SG imagination, aiming to align the imagined SGs with the gold images while minimizing unwanted information induced. The other one is controllable image generation, where the image generation is fine-grainedly guided by the objects, attributes, and relationships specified in the SGs.\n\n---\n**Q10:  How to ensure that the objects … remain in the imaged SGs?**\n\n**A:** Indeed, it is not essential to ensure a strict correspondence between the initial scene graphs (SGs) and the imaged SGs, since the imaged SGs involves many more concrete objects, while the initial SGs maybe include certain abstract objects.\n\n---\n**Q11:  Have the main results in Table 1&2 adopted the scene sampling mechanism? So is this the average of multiple experimental results?**\n\n**A:** Yes, it is. We present the average of multiple experimental results.\n\n---\n**Q12:  In HIS, which semantic level has a significant impact on task performance?**\n\n**A:** The information at different semantic levels contributes to the task performances to different extent, with the semantic information at the relation level showing a notable influence on the task. This is attributed to the fact that the information at relation level encompasses both object-level details and higher-level information. \n\n---\n**Q13:  How to leverage the VG dataset to train the SGH module?**\n\n**A:** As no captions for images in the VG dataset, we first extract the seed graph as initial graph by the algorithm in [1]. Then, we transform the seed graph into a token sequence as the text prompt. Finally, we use the VG data to train our SGH module.\n\n[1] GEMS: Scene Expansion using Generative Models of Graphs. WACV 2023 \n"}, {"review_id": "n2iTPxP7oT", "reviewer": "Reviewer_QdRJ", "summary": "This paper studies a new setup of generating intricate images from abstract prompts. To overcome the issues of vision distraction and wrong binding of using text as the condition to generate images, the author proposes a two stage pipeline by first generating scene graph from abstract text inputs, and then condition on the sythesized scene graph, another model generates images. The proposed diffusion model with SG guidance showcases its controllability and interoperability and it achieves new SoTA results in the abstract-to-intricate T2I setup. Overall, the paper is well-written and the proposed method is interesting and novel. \n", "questions": "It seems like using SG as condition/guidance will lower the FID compared to text conditioning and also in qualitative results, details like faces will be blurry. Any idea how to improve this and explanation of the cause?\n\nAny idea after enriching the SG, if users want to modify some nodes, how would the user interact with the SG/model?\n", "limitations": "As pointed out by the author, the proposed method depends on the generation quality of the SG, while a large-scale SG dataset is rare. Plus, since the pipeline needs to modify the conditioning of the LDM, how to better leverage pre-trained large T2I model is worth investigating.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The introduction section is easy to follow, they provide examples to show the issues of the current T2I generation and motivate the scene graph representation as a potential solution.\n\nThe proposed SGH model is interesting and well adopted from VQ-diffusion etc for image modelling to the problem of scene graph modelling.\n\nThe experiments are convincing with strong performance on COCO, outperforming competitive baselines LDM, VQ-diffusion and Frido, they also construct an abstract-to-intricate dataset from COCO and demonstrate SoTA performance in this setup.\n", "weaknesses": "One claim of the paper is that SG guidance helps image generation with strong semantic controllability, it is not clear to me which experiments can support this claim.\n\nIn table 3, it seems like replacing the HSI module with GCN encoding only drops a little bit of the performance, it is questionable if the design of HSI is necessary.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you that you acknowledge the strengths of our work. Your support is the source of the power to push us forward and further enhance the paper. Following, we show the response to your questions one by one.\n\n---\n**Q1:  One claim of the paper is that SG guidance helps image generation with strong semantic controllability, it is not clear to me which experiments can support this claim.**\n\n**A:** Actually, we have done the evaluation for this claim, please kindly refer to _Q1_ in section 4.5 (line 288). To evaluate the semantic controllability of SGs in image generation, we make comparisons with baselines (i.e., LDM, Frido) without SG guidance by measuring the semantic similarity (measured by _CLIP_ score) and semantic structural alignment (measured by _TriRec._ metric). As shown in Figure 6, the system with the guidance of SG consistently performs much better than the baselines in terms of those two metrics, indicating that SG guidance enhances the semantic controllability of the T2I process, and leads to high-faithful generation.\n\n---\n**Q2:  In Table 3, it seems like replacing the HSI module with GCN encoding only drops a little bit of the performance, it is questionable if the design of HSI is necessary.**\n\n**A:** It is our mistake in wrongly calculating the numbers in Table 3. We incorrectly note the performance drop as 1.17. But the actual performance drop should be 4.71(=35.96-31.25) on the COCO-A2I dataset when replacing the HSI module with GCN. This notable decrease actually underscores the indispensability of the HSI module.\n\nThanks for pointing this out. We will correct it.\n\n---\n**Q3:  It seems like using SG as condition/guidance will lower the FID compared to text conditioning and also in qualitative results, details like faces will be blurry. Any idea how to improve this and an explanation of the cause?**\n\n**A:** FID measures the Fréchet Distance between the distribution of the synthetic images and real-world images within the feature space. In light of this, the lower FID score corresponds to the generation of more realistic images. Therefore, the model additionally using SG as condition/guidance achieves a lower FID score compared to the model only conditioned on text, demonstrating the former model excels in better image generation.\n\nAs seen from the qualitative results, compared to the baselines, our proposed model has significantly enhanced the quality of face synthesis than the baselines. Besides, we kindly note that the images have been compressed to some extent when compiling into PDF file, resulting in blurriness and a decrease in resolution. Later upon acceptance, we will show the high-resolution resulting images on our project page to the public.\n\n---\n**Q4:  Any idea after enriching the SG, if users want to modify some nodes, how would the user interact with the SG/model?**\n\n**A:** Good idea, which seems very promising in our setting. Actually using SG guidance for diffusion-based image editing by the user is feasible, because SG is a highly structural representation that enables more interactive manipulation for users and more controllable image generation. \n\nIn our designed framework, the idea can be implemented by taking the user-modified SG as the conditions and then employing the hierarchical scene integration module to directly fuse the SG feature representations into the diffusion model without SG hallucination. However, in our current system, we did not cover this point. Thanks for your output, we will mention this part as future work.\n\n---\n**Q5: As pointed out by the author, the proposed method depends on the generation quality of the SG, while a large-scale SG dataset is rare. Plus, since the pipeline needs to modify the conditioning of the LDM, how to better leverage the pre-trained large T2I model is worth investigating.**\n\n**A:** Thank you for your insights. Indeed, the size of the SG dataset plays a pivotal role in the imaging ability of the proposed scene graph hallucination (SGH) module. Actually, we have leveraged the rich number of annotated Visual Genome (VG) dataset (with 62k annotated images) for pre-training the SGH module, and then we fine-tune the SGH on the COCO dataset (83k). We believe the data quantity of the training SGH module can be quite substantial in size. Based on the experiments, the proposed SGH moudel is capable of imaging reasonable SGs and facilitates the generation of high-quality images. Of course, it is well believed that with more data, the performance of SGH can be further boosted, for producing more accurate and visually-enriched imagination. "}, {"review_id": "hzhjyM3VHv", "reviewer": "Reviewer_GuSJ", "summary": "In this paper, text-to-image synthesis under the abstract-to-intricate setting is studied. Firstly, the input prompt is hallucinated and expanded into feasible specific scene structures by the proposed SGH mechanism. Then, text-to-image synthesis is implemented through a diffusion-based synthesizer by gradually incorporating semantic scene structure induced from the SGH. Extensive experiments on COCO, especially on the abstract-to-intricate text-to-image setting, prove the method could contribute to synthesizing images reasonably and accurately under the simple text.", "questions": "1.\tPlease provide results compared to text-based enrichment methods on COCO dataset.\n2.\tPlease discuss results compared to the best Firdo other T2I baselines like make-a-scene.\n3.\tWhy bounding boxes of scene graphs for visual guidance are not included in training? Will there be gain by adding bbox?\n4.\tPlease modify writing issues and answer questions in 4,5,7 of Weakness.\n5.\tCitations are missing as described in 2, 6 of Weakness.\n", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed abstract-to-intricate T2I is a vital research topic in the field of text-to-image synthesis since existing large models require delicate and well-designed prompts for controllable image synthesis.\n2.\tAbstract-to-intricate T2I is further promoted by the designed SGH mechanism, avoiding vision distraction from text and wrong focus enriched prompts.\n3.\tExtensive experiments including thorough metrics are conducted to prove the effectiveness. And corresponding analysis and discussion are reasonably stated.\n", "weaknesses": "1.\tExisting experiments compared to the text-based enrichment methods are only conducted on manually selected dataset COCO-A2I with Place Norm and Progressive Verbs, which is not representative and convincing enough.\n2.\tThere exists unfair comparison in T2I results on COCO, since the best FID and CLIP score of Frido are respectively 8.12 and 0.7915. And the best T2I baseline on COCO is not Frido. As far as I know, make-a-scene[1] achieves a 7.55 FID, which is not discussed.\n3.\tScene graphs consist of two parts, which are node and edges representing semantics, bounding boxes referring to sizes and locations of objects. As proved by previous works, incorporating visual guidance into T2I training is beneficial. Why bounding boxes information is not included for training?\n4.\tOverclaims and inaccurate description in contributions:” We propose a diffusion-based model with SG guidances for highly controllable and scalable image generation.” Scene graph hallucinations might include unexpected concepts, which is not controlled by users.\n5.\tUnclear captions and inconsistent description. The caption of Fig.2 is unclear and lacks descriptions about each subfigure. \n6.\tCitations about scene graph-to-image synthesis and scene graph generation are not thoroughly included[2-4].\n7.\tTypos and inconsistent descriptions: In line 226, “diffusioninspired”. In line 245, Frido-G is inconsistent with the description on Tab.1. \n\n[1] Gafni O, Polyak A, Ashual O, et al. Make-a-scene: Scene-based text-to-image generation with human priors[C]//Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV. Cham: Springer Nature Switzerland, 2022: 89-106.\n[2] Sitong Su, Lianli Gao, Junchen Zhu, Jie Shao, and Jingkuan Song. 2021. Fully Functional Image Manipulation Using Scene Graphs in A Bounding-Box Free Way. Proceedings of the 29th ACM International Conference on Multimedia. Association for Computing Machinery, New York, NY, USA, 1784–1792. https://doi.org/10.1145/3474085.3475326\n[3] Lyu X, Gao L, Guo Y, et al. Fine-grained predicates learning for scene graph generation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 19467-19475.\n[4] Herzig R, Bar A, Xu H, et al. Learning canonical representations for scene graph to image generation[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16. Springer International Publishing, 2020: 210-227.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and valuable comments. Your suggestions will surely help consolidate our paper. Following we present the point-to-point response to address your concerns. And if you feel our responses effectively relieve your concerns, please kindly reconsider your evaluation.\n\n ---\n**Q1:  Existing experiments compared to the text-based enrichment methods are only conducted on manually selected dataset COCO-A2I with Place Norm and Progressive Verbs, which is not representative and convincing enough.  Please provide results compared to text-based enrichment methods on COCO dataset.**\n\n**A:** Here, we also provide the results by comparing them with the text-based enrichment methods on the overall COCO dataset, where pre-trained Frido is employed to generate images. Compared with the best baselines, the two enrichment approaches fail to exhibit superior results on the three evaluation metrics, indicating that text-based enrichment not only fails to ease the difficulty of existing models in image synthesis, but also causes the performance decrease of existing models. Moreover, these results further demonstrate the effectiveness of our proposed method.\n| Model | FID | IS | CLIP|\n| :-----: | :----: | :----: | :----: |\n|Frido|11.24|26.84|70.46|\n|SD-PG|15.78|24.76|65.74|\n|SPY|12.63|25.34|66.95|\n\n ---\n**Q2:  There exists an unfair comparison in T2I results on COCO, since the best FID and CLIP score of Frido are respectively 8.12 and 0.7915. And the best T2I baseline on COCO is not Frido. As far as I know, make-a-scene[1] achieves a 7.55 FID, which is not discussed.**\n\n**A:**  Sorry for the mistake of not providing the full comparison with Frido. We conduct a re-evaluation using the open-source checkpoint provided by the authors and the same evaluation code. Following are the experimental results. Comparatively, our model outperforms Frido-CLIP and exhibits superior performance, particularly on the COCO-A2I dataset, showcasing significant improvements. This again validates the effectiveness of the proposed method.\n\nResult on COCO:\n|Model|FID|IS|CLIP|\n| :-----: | :----: | :----: | :----: |\n|Ours|10.19|29.96|74.83|\n|Frido-CLIP|10.87|27.30|73.46|\n\nResult on COCO-A2I:\n|Model|FID|IS|CLIP|\n| :-----: | :----: | :----: | :----: |\n|Ours|31.25|28.63|71.29|\n|Frido-CLIP|38.45|19.22|69.20|\n\nAs for the Make-A-Scene model you indicated here, it is actually trained on the total collection of **CC12m**, **CC**, **YFCC100m**, **Redcaps**, and **COCO**, in amount to 35m text-image pairs, a definitely much larger volume data than ours, which enables to achieve a much lower FID score. Therefore, the direct comparisons with Make-A-Scene can be unfair. Also, the evaluation on Make-A-Scene could be unfeasible for us, as it does not offer an open-source checkpoint.\n\n---\n**Q3:  Scene graphs consist of two parts, which are nodes and edges representing semantics, bounding boxes referring to sizes and locations of objects. As proved by previous works, incorporating visual guidance into T2I training is beneficial. Why bounding boxes information is not included for training?**\n\n**A:** In this work, we placed the major emphasis on modeling the object nodes and semantic relationships between objects and attributes. The experimental results demonstrate that just employing such information has notably improved the quality of image generation. \n\nThat being said, we totally agree that the incorporation of layout information (e.g., the bounding box) will be promising to gain more controllable and high-quality image generation [1], which we consider as future work.\n\n[1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models\n\n ---\n**Q4:  Overclaims and inaccurate description in contributions:” We propose a diffusion-based model with SG guidances for highly controllable and scalable image generation.” Scene graph hallucinations might include unexpected concepts, which is not controlled by users.**\n\n**A:** Actually, here we emphasize the **controllable generation** more about the highly structured SG representations (i.e., the 'object-predicate-subject' triplets) that exhibit the ability of precise control over the image generation process. \n\nYour point could also be reasonable to a certain extent that, hallucinations may come with unexpected outcomes. But in our work, by training with sufficient volume of supervised data, our SGH module learns to (more tends to) have valid imagination (as we evaluated and answered this question in our _Q2_ of section 4.4, line 304). So this is quite different from the hallucination by the existing LLMs; they are two separate concepts. But, we will improve our expressions and make it clearer and more accurate on this.\n\n---\n**Q5:  Unclear captions and inconsistent description. The caption of Fig.2 is unclear and lacks descriptions about each subfigure.  Typos and inconsistent descriptions: In line 226, “diffusioninspired”. In line 245, Frido-G is inconsistent with the description on Tab.1**\n\n**A:** Thanks for going through the paper this carefully; we will correct them all in the revision.\n\n---\n**Q6: Citations about scene graph-to-image synthesis and scene graph generation are not thoroughly included.**\n\n**A:** Thanks for indicating the missing literature, we will carefully check them and consider including them into the related work."}]

Author rebuttal:
# General Response to All Reviewers

Dear reviewers,

Thanks for all of your time to write valuable and constructive comments. Your feedback will definitely assist us in enhancing the quality of our paper, and thus we are committed to incorporating your suggestion in our revision process. Meanwhile, we feel encouraged so much that the reviewers find the new task setting is **'practical and important', 'vital' and 'interesting and meaningful'** (by reviewer #DqEY, #mzvT and #GuSJ), **the proposed method novel, interesting and effective** (by reviewer #QdRJ, #mzvT and #KZmg), and **our experiments solid and comprehensive** (by reviewer #QdRJ, #mzvT, #GuSJ and #DqEY). Your support means a lot to us!  
At this juncture, we would like to re-emphasize the significance of this work.

While existing state-of-the-art (SoTA ) text-to-image (T2I) generation systems (e.g., Stable Diffusion) have shown incredible capability in creating high-quality images, the research on abstract-to-intricate (A2I) T2I has been largely overlooked, which can be also a very important setting in realist world. With such background, this work contributes to the following key aspects:

- We are the first to study the novel T2I setup of **intricate image synthesis from succinct abstract texts**, i.e., A2I T2I, for which we collect a dataset, COCO-A2I.
- We solve the A2I T2I with a novel scene graph (SG) hallucination framework implemented based on the discrete diffusion technique.
- We construct a diffusion-based T2I system, which employs the SG representations for highly controllable and scalable image generation.
- Our system empirically shows great advantages over existing SoTA baselines in the A2I T2I generation.

As recognized by reviewer#mzvT, the paper _'studies the specific conditioned image synthesis from an interesting and realistic perspective, with robust and novel technical methods'_, and _'will have the potential to unlock interesting future research'_. We much believe this work will show a broad impact on the future research of the community. Thus, we will release all the codes and resources upon acceptance.

In response to the reviewers' comments, we have thoroughly reviewed our paper, performed additional experiments, and prepared a comprehensive response. We will fix all the typos and improve the manuscript according to your comments. We hope that our paper adequately addresses your concerns. We also kindly hope reviewer #DqEY and reviewer  #GuSJ (both with borderline rejection) can raise the evaluation if our responses can effectively address the concerns, and look forward to your recognition.

Best regards.

Meta-review: The AC has carefully read the paper, reviews, author response, and the discussions. This paper aims at the abstractive setting for text-to-image generation. They propose scene-graph hallucination (SGH) to perform imagination over the scene graph of the input prompt and make up the missing information. All reviewers are positive towards acceptance. The AC agrees with the reviewers that this is a solid submission for NeurIPS and thus recommends acceptance with a certain confidence.
Decision: Accept (poster)