Title: Imagine That! Abstract-to-Intricate Text-to-Image Synthesis with Scene Graph Hallucination Diffusion
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image Synthesis

Scene Graph

Diffusion Model
Abstract: In this work, we investigate the task of text-to-image (T2I) synthesis under the abstract-to-intricate setting, i.e., generating intricate visual content from simple abstract text prompts. Inspired by human imagination intuition, we propose a novel scene-graph hallucination (SGH) mechanism for effective abstract-to-intricate T2I synthesis. SGH carries out scene hallucination by expanding the initial scene graph (SG) of the input prompt with more feasible specific scene structures, in which the structured semantic representation of SG ensures high controllability of the intrinsic scene imagination. To approach the T2I synthesis, we deliberately build an SG-based hallucination diffusion system. First, we implement the SGH module based on the discrete diffusion technique, which evolves the SG structure by iteratively adding new scene elements. Then, we utilize another continuous-state diffusion model as the T2I synthesizer, where the overt image-generating process is navigated by the underlying semantic scene structure induced from the SGH module. On the benchmark COCO dataset, our system outperforms the existing best-performing T2I model by a significant margin, especially improving on the abstract-to-intricate T2I generation. Further in-depth analyses reveal how our methods advance.

Imagine That! Abstract-to-Intricate Text-to-Image Synthesis with Scene Graph Hallucination Diffusion

Abstract

In this work, we investigate the task of text-to-image (T2I) synthesis under the abstract-to-intricate setting, i.e., generating intricate visual content from simple abstract text prompts . Inspired by human imagination intuition, we propose a novel scene-graph hallucination (SGH) mechanism for effective abstract-to-intricate T2I synthesis. SGH carries out scene hallucination by expanding the initial scene graph (SG) of the input prompt with more feasible specific scene structures, in which the structured semantic representation of SG ensures high controllability of the intrinsic scene imagination. To approach the T2I synthesis, we deliberately build an SG-based hallucination diffusion system. First, we implement the SGH module based on the discrete diffusion technique, which evolves the SG structure by iteratively adding new scene elements. Then, we utilize another continuous-state diffusion model as the T2I synthesizer, where the overt image-generating process is navigated by the underlying semantic scene structure induced from the SGH module. On the benchmark COCO dataset, our system outperforms the existing best-performing T2I model by a significant margin, especially improving on the abstract-to-intricate T2I generation. Further in-depth analyses reveal how our methods advance. 2

1 Introduction

The task of generating images from natural language descriptions, known as text-to-image (T2I) synthesis, has attracted significant attention [4, 17, 52]. To approach T2I, various generative models have been explored, including generative adversarial networks (GANs) [48, 52, 61], variational autoencoders (VAEs) [50], flow-based models [3], and auto-regressive models (ARMs) [10, 41], all of which aim to generate realistic images in high quality and high faithfulness. Most recently, diffusion-based models have been proposed, which simulate the physical process of gas diffusion for image generation [23]. Diffusion models have shown unprecedented performance in image synthesis over existing methods, becoming the current state-of-the-art (SoTA) T2I solution [2, 9, 19, 47, 42].

As a long-reached viewpoint [28, 53, 37], sound T2I systems should not only achieve high-quality image generation in simple straightforward visual scenery but be more capable of synthesizing realistic images with complex scenes. Typically, detailed textual descriptions are necessarily needed to prompt the synthesis process with adequate details for high-quality vision generation. However, in a realistic world, it could also be ubiquitous to produce intricate visions without relying on lengthy elaborate prompts. For example, users may prefer T2I systems to synthesize well-detailed images while not taking too much time to write descriptions in detail. More crucially, due to the natural modality asymmetry between language and vision, even some simple words can intrinsically describe or represent abstract visual scenes with rich and complex details. Whenever mentioning the words

Issue: Vision Distraction

Two young men give a presentation in the

office.

Two middle-aged, nice, enthusiastic,

confident, man with polished shoes

and sleek hair give a professional

presentation in the spacious and modern

conference room of the corporate blue

office, room with chairs and tables.

Two young man give a presentation in the

office, old, nice, confident, enthusiastic,

laughing man with polished hair, seek hair,

room with chairs and tables, speaking to

each other.

Issue: Wrong Binding

Issue: Abstract-to-intricate Failure

Imagination over SG

Two young men give a presentation in the office.

man

give

two

office

a

in

presentation

Initial SG

Rendering

Constructing

Human

Relation Node

Attribute Node

Object Node

（

）

（

）

Enriched Prompt-II:

Enriched Prompt-I:

Original (

abstract

) Prompt:

man

give

two

office

light

in

presentation

microphone

hlod

table

chair

in

in

tall

a

Enriched SG

个

→

a

b

with specific scenes, such as classroom, kitchen, office , or actional verbs e.g., traveling, shopping , there is always a picture with multifaceted scenes and rich-detailed backgrounds. In a word, it is worth investigating generating intricate images from succinct abstract prompts .

Yet existing prevailing approaches, even the SoTA diffusion models, may largely fail the abstract-tointricate T2I, due to the lack of necessary details of input prompts (cf. Figure 1(a)). One intuitive workaround is to directly enrich the texts, i.e., adding more details for the prompts. Specifically, existing works either consider inserting additional adjectives and attributives to modify the original mentions and scenes [8], or concatenating raw sentences with more tangible explanations and contexts that are elicited from external large language models [48], e.g., ChatGPT [38]. Unfortunately, due to the intrinsic grammar and linguistics rules, such text-based prompt enrichment can be subject to the issue of lower controllability. One problem is the visual distraction , where the main focus of the resulting image is dominated by other newly-added trivial contents, when aggressively inserting intermediate descriptive components into the raw texts, as exemplified in Figure 1(a). Besides, directly appending new textual descriptions would increase the prompt length and then lead to incorrect binding of attributes or relations, i.e., making the image deviate from the original user intention.

As a reference, intuitively, we human beings would tackle the abstract-to-intricate T2I as a twostep painting process, i.e., from semantic interpretation to scene imagination . In the semantic interpretation, a painter always first translates the succinct textual prompt into a structured skeleton that represents the semantic scene of key mention objects and their relations. Then, based on the initial scene, the painter mentally completes the abstract scene with more concrete and valid details. With the enriched scene structure, the final vision can be more accurately and easily rendered. Motivated by such human intuition, in this work, we propose a scene-graph hallucination (SGH) method for achieving effective abstract-to-intricate T2I. As illustrated in Figure 1(b), we first investigate representing the input prompt with its scene graph (SG) [46]. SG advances in depicting the intrinsic semantics of texts (or vision) with structured representations, e.g., objects, attributes, and relationships, enabling fine-grained control of the semantic scene [53]. Based on the SG of input text we then carry out scene hallucination, expanding the initial SG with more possible specific scene structures. Also with the SG representation, the imagination process can be much more accurate and controllable.

To implement the overall idea, we develop an S G-based h al lucin a tion d iffusion system (namely, Salad ) for high-quality image synthesis. Salad is a fully diffusion-based T2I system, which mainly consists of a scene-driven T2I module and an SGH module. As shown in Figure 2, we first take advantage of the SoTA latent diffusion model [9] as our backbone T2I synthesizer, in which the overt image-generating process is controlled and navigated by the underlying semantic scene structure.

...

...

...

...

...

...

...

SG Decoder

...

Diffusion process

Denoising process

U-Net

SG Decoder

Scene-driven Image Synthesis

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

SG (   )

Equal

Prompt Text

SG Parsing

Initial SG

Object

Nodes

Attribute

Nodes

Relation

Nodes

(cf.        )

(cf.        )

AdaLN

Text Cross-attetion

Graph Cross-attention

NTD Cross-attention

Self-attetion

Transformer Block

(Text-CA)

(Graph-CA)

(NTD CA)

On the other hand, we design the SGH module based on the modality-agnostic discrete diffusion model [25], which evolves and completes the initial SG structure of the input prompt by iteratively generating new scene elements (i.e., SG nodes). During the SG imagination process, the resulting structural representation at each step is fused into the T2I synthesizer via a hierarchical integration strategy. Further, we devise a scene sampling mechanism, via which the SGH module can generate various SG imaginations, and thus help achieve diversified image synthesis during inference.

We conduct experiments on COCO [33], the widely-used T2I dataset. Results show that Salad outperforms all the existing strong-performing T2I systems with significant margins. Further analysis reveals that modeling the SG structures helps synthesize high-quality images with stronger semantic controllability. Our proposed SGH mechanism is effective in inducing sound SG structures, helping produce more realistic images from short abstract text prompts. And the scene sampling strategy aids diversified T2I generation. In summary, this paper contributes in five aspects:

We are the first to study the novel setup of intricate image synthesis from abstract texts.

We propose a diffusion-based model with a hierarchical scene integration strategy for highly controllable and scalable image generation.

We solve the abstract-to-intricate T2I with a novel SG hallucination mechanism, which is implemented via discrete diffusion technique, performing scene enrichment with reasonable imagination.

We devise a scene sampling mechanism to generate various scene graphs for diversified image syntheses during inference.

Our framework achieves new SoTA results in the abstract-to-intricate T2I generation.

2 Preliminary

2.1 Scene Graph Representation

The SG (denoted as G ) [45] represents the semantic relationships among scene objects in a structure, where there are three types of nodes, i.e., object , attribute , and relation , cf. Figure 1(b). We formulate the object node set as { o 1 , · · · , o N } , where o n denotes n -th object node; the attribute node set as { a 1 , 1 , · · · , a N,M } , where a n,m means m -th attribute node of the n -th object node; the relation node set as { r 1 , 1 , · · · , r N,N } , where r i,j means object node o i connects to the object o j . All nodes come with a category label l o/a/r , and each type of node has its own unique category vocabulary. For example as in Figure 2 right bottom, the object node o 1 with category label l o 1 associated with two attribute nodes a 1 , 1 and a 1 ,M , with category label l a 1 and l a 23 , respectively. And the object node o 1 connects to the object node o 2 through an edge r 1 , 2 with the category label l r 7 .

2.2 Diffusion Models

Diffusion models (DMs) [23] learn to convert a simple Gaussian distribution into a data distribution. Technically, DMs consist of a forward (diffusion) process and a reverse (denoising) process. In the forward process, the given data x 0 ∼ q ( x 0 ) is gradually corrupted into an approximately standard normal distribution x T ∼ p ( x T ) over T steps by increasingly adding noisy, formulated as

q ( x 1: T | x 0 ) = ∏ T t =1 q ( x t | x t -1 ) . The learned reverse process p θ ( x 0: T ) = p ( x T ) ∏ T t =1 p θ ( x t -1 | x t ) gradually reduces the noise towards the data distribution. To improve the fit of a generative model to the data distribution, a variational upper bound on the negative log-likelihood is optimized:

where p θ ( · ) is estimated by a denoising network, which can be a time-conditional U-Net [43]. Recently, latent diffusion models (LDMs) [42, 14, 19] are introduced to adopt DMs to operate in an efficient, low-dimensional latent space, where a pre-trained encoder E maps the given data x 0 into a latent code z 0 = E ( x 0 ) , and a decoder reconstructs the final output image from the denoised latent D ( z 0 ) ∼ x 0 . Due to the higher computation sufficiency, this work thus takes the LDM as T2I backbone. Besides, DMs also have been extended to operate in discrete state spaces [1], i.e., performing diffusion and denoising processes over discrete variables, which have demonstrated competitive performances for discrete data, such as text [24] and layout [5]. Hence, we also adopt the discrete DMs to realize the SG induction process. Appendix §A.1 gives more technical details to the discrete diffusion models.

3 Methodology

Formally, T2I aims to generate an image x that faithfully reflects the desired content in the input prompt text y . To approach abstract-to-intricate T2I, we propose an SG-based hallucination diffusion system (Salad), which is shown in Figure 2. The salad consists of two major modules. First, the SGH (cf. § 3.1) is responsible for enriching the initial SG of the text prompt via a discrete diffusion model. Then, built upon an LDM (cf. § 2.2), the Scene-driven Image Synthesis module (SIS, cf. § 3.2) performs denoising for image synthesis, during which the derived SG features is fused via a hierarchical scene integration mechanism. The underlying SGH closely collaborates with the upper SIS at each step, and thus the semantic scene skeleton takes fine-grained control of the overt vision rendering. We also describe the optimization (cf. § 3.3), and the scene sampling strategy (cf. § 3.4).

3.1 Scene Graph Hallucination (SGH)

As aforementioned, we formulate the SGH as a discrete denoising diffusion process [25] (cf. Fig. 2). Specifically, in the forward process, the SG of the gold image, marked as G 0 , will be corrupted into a sequence of increasingly noisy latent variables G 1: T = { G 1 , G 2 , · · · , G T } , where each SG node s ∗ t,j ∈ G t , ∗ ∈ { o, a, r } ( t is diffusion step, j is the node index) takes a discrete value with K ∗ category labels, and o, a, r denotes the nodes' type, i.e., object ( o ), attribute ( a ), and relation ( r ). For simplicity, we omit subscripts j and superscripts ∗ in the following description. The discrete diffusion process can be parameterized with a multinomial categorical transition matrix:

where B ( s t ) denotes the column one-hot vector of s t . And Q t is the transition matrix, with [ Q t ] ij = q ( s t = j | s t -1 = i ) representing the probabilities that s t -1 transitions to s t . Due to the property of the Markov chain, the cumulative probability of s t at arbitrary timestep from s 0 can be derived as q ( s t | s 0 ) = B ⊤ ( s t ) · ¯ Q t · B ( s 0 ) , where ¯ Q t = Q t Q t -1 · · · Q 1 . Inspired by [1, 19], we employ a mask-and-replace strategy to design the Q t . For each node s t , we define three probabilities: 1) a probability of γ t to transition to [MASK] node, 2) a probability of Kβ t be resampled uniformly over all the K categories, and 3) a probability of α t = 1 -Kβ t -γ t to stay the same node. Notedly, the [MASK] node never transits to other states. Hence, the transition matrix Q t can be formulated as 3 :

The aforementioned discrete state-space models assume that all the standard nodes are switchable by corruption. However, as stated in § 2.1, there are three different SG nodes under separate categories. Hence, we apply three disjoint corruption matrices Q o t ∈ R K o × K o , Q a t ∈ R K a × K a , Q r t ∈ R K r × K r for object, attribute, and relation nodes, respectively, where K o , K a , K r denotes the size of category labels of three node types respectively.

In the denoising process, we introduce an SG decoder as the neural approximator to estimate the distribution p θ ( s t -1 | s t , y ) . As shown in Figure 2, SG decoder first employs an adaptive normalization (AdaLN) to inject the timestep information. A text cross-attention (Text-CA) integrates the input prompt y . Then, a graph cross-attention (Graph-CA) is devised to take in the induced SG ( G t +1 ) in the previous t +1 timestep:

where H ∗ are the features yielded from Text-CA. Graph-CA consistently consults the overall picture of the last SG for a more coherent generation when making the current decision.

Intuitively, among the object, attribute, and relation nodes, the object nodes always come first to determine the scene subjects, followed by their modifier attributes, and then the relations between objects. Thus, instead of parallel induction of three node types, we follow this SG node-type dependence (NTD) intuition, and design an NTD cross-attention (NTD-CA) for the ∗ -type node induction ( ∗ can be object or attribute):

Note that we stack multiple layers of the above calculations as one SG decoder. For each state of node types ˆ s ∗ t , a softmax function is put on to obtain the category label distributions: ˆ s ∗ t = Softmax ( ˆ H ∗ ) .

Following [1], we optimize the SG decoder by minimizing the variational lower bound L vlb (Eq. 1). Also the parameterization trick [19] is leveraged to encourage the system to predict the noiseless node distribution p θ (˜ s 0 | s t , y ) at each reverse step, which is taken as an auxiliary learning objective to be incorporated with L vlb :

where λ 1 is a hyper-parameter for controlling the learning components.

3.2 Scene-driven Image Synthesis (SIS)

With the enriched SG from SGH in each denoising step at hand, the backbone T2I diffusion carries out the image synthesis with the guidance of that SG. We design a hierarchical scene integration (HSI) strategy to ensure the highly effective integration of SG features. Specifically, we consider the fusion at four different hierarchical levels, i.e., object ( L o ), relation ( L r ), region ( L c ), and global levels ( L g ) with each focusing on different context scopes, as illustrated in Figure 3. We maintain the representations of these three levels as the keys K L i & values V L i via CLIP encoder [40], which are then integrated together via the Transformer attention of U-Net in LDM:

SG

Attn

Attn

Attn

U-Net

Object level

Relation level

Region level

ResNet

Trm

Trm

ResNet

Attn

Global level

where H is the visual query vectors from the ResNet block in LDM. The above hierarchical scene integration is carried out for both the downsampling and upsampling processes in U-Net. By denoising T steps, the system finally produces the desired image. Appendix §A.2 gives more details of this part.

3.3 Warm-start Training

To ensure stable learning of the overall system, we take a warm-start training strategy. Firstly, the SGH is separately updated via L SGH (Eq. 6) based on the abstract-to-intricate SG pair annotations,

until it has converged. Then, both the SIS and SGH modules are optimized jointly by minimizing:

Here we follow [23] to optimize SIS with a simple surrogate objective that calculates the meansquared error loss, and G t is the intermediate SG by SGH at timestep t , which can be derived from the s ∗ t , ∗ ∈ { o, a, r } (cf. Figure 2). ϵ is the noise in SIS, and ϵ θ ( · ) denotes the U-Net (cf. Figure 3).

3.4 Inference with Scene Sampling

During inference, we further aim to endow the SGH with diversified SG enrichment, and thus lead to T2I diversification. Intuitively, given an abstract prompt, there is often more than one possibility of the potential scenes to imagine. Also, it can be observed that in the de-

...

...

Top-A

Category

Distribution

A candidates

Sampling

...

...

...

noising process, the diffusion model has a larger potential of divergence only at its earlier stage, while the generation tends to be more stable and certain when the iteration grows. Correspondingly, we expect the scene sampling to start in the early denoising steps, and gradually be more determining. Thus, we design a scene-sampling mechanism. First, instead of picking the best prediction of the category of any node ˆ s ∗ t,j , we take the topA category candidates with corresponding probability distribution Ψ based on the category distribution ˆ s ∗ t,j ∈ R K ∗ of node s ∗ t,j . Then, we perform sampling over these candidates with a dynamic probability:

where ∗ ∈ { o, a, r } , and η is a temperature. It is an annealing process, i.e., when t = T (starting denoising) more random sampling is preferable, while t approaches 0 (denoising ends), SGH tends to be more decisive. Figure 4 exemplifies the mechanism with the object type of nodes ( ˆ s o t,n ).

4 Experiments

4.1 Settings

Data and Resource We conduct T2I generation experiments mainly on the COCO [33] dataset. We also prepare the abstract-to-intricate SG pair annotations for training the SGH module, where we employ an external textual SG parser [46] and a visual SG parser [59] on the paired images and texts in COCO, to obtain the initial SG and imagined SG, respectively. To enlarge the abstract-to-intricate SG pairs, we further extend Visual Genome (VG) [30]. Besides, to simulate the abstract-to-intricate T2I scenario, we further manually extract a subset of text-image pairs from raw COCO data (named COCO-A2I), in which the texts are short and abstract, 4 while the images are comparatively complex and intricate. Appendix §B.1 shows all the dataset details.

Baseline and Evaluation We make comparisons with three types of existing strong-performing T2I models. 1) GAN-based models : AttnGAN [52], ObjGAN [32], DFGAM [48], OPGAN [22]. 2) Auto-aggressive model : DALL-E[41], and CogView [10]. 3) Diffusion-based models : LDM [42], VQ-diffusion [19], LDM-G and Frido [14] with classifier-free guidance. Note that LDM-G and Frido are the current SoTA T2I synthesizers. In addition, two types of text-based enrichment approaches are included as baselines: stable-diffusion prompt generator (SD-PG) and SPY inspired by [6]. The enriched text prompts are then utilized as inputs for Frido to generate the final images. We adopt three standard metrics to measure image synthesis performance: 1) Inception score (IS) [44], 2) Fréchet Inception Distance (FID) [21] and 3) CLIP score . Moreover, we use GLIP [31] to measure the fine-grained ' object-attribute ' grounding in images, and Triplet Recall (TriRec.) measure the ' subject-predicate-object ' triplet recall between two SGs. We also adopt the Learned Perceptual Image Patch Similarity (LPIPS) [60] for diversifying generation evaluation. Detailed definitions of evaluation metrics are shown in Appendix §B.2.

a man is

travelling

Frido

SPY

a man sits in

the kitchen

a man sits in

the bank

Salad

man

sit in

in

man

in front of

wear

carrying on

Tshirt

sit on

man

luggage bag

next to

chair

table

ktichen

chair

in

microwave

windows

bank

in

fridge

sunglass

sit on

young

black

wear

building

old

next to

next to

in front of

NMAA'N

TOAVDR

Implementation We define the maximum number of SG object nodes as 30, and each object node has a maximum of 3 attributes. We set the timesteps ( T ) for SGH and SIS as 100. For the SIS module, we load the parameters of Stable Diffusion 5 (v1.4) as the initialization. We use the CLIP 6 (vit-large-patch14) as our text encoder. We optimize the framework using AdamW [34] with β 1 = 0 . 9 and β 2 = 0 . 98 . The learning rate is set to 5e-5 after 10,000 iterations of warmup. For the attention layer in SG decoder and UNet in SIS, we define a shared configuration as follows: 4 layers, 8 attention heads, 512 embedding dimensions, 2,048 hidden dimensions, and 0.1 dropout rate. We mainly follow the prior works [54, 18, 46] to acquire the visual scene graph (VSG) from the gold image and textual scene graph (TSG) from the text prompt.

4.2 Main Comparisons

Table 1 shows the main T2I generation results on the overall COCO data. We see that diffusion-based methods are consistently better than the other two types of T2I methods, especially on the FID metric. Most importantly, our proposed Salad model yields the overall best results on all metrics. For example, we outperform Frido by 1.05% FID, 3.37% on CLIP score, and surpass LDM-G by 2.1% IS score. This directly demonstrates the efficacy of our method.

To directly assess the capability of solving the abstract-to-intricate T2I synthesis, we further make comparisons on the collected COCO-A2I dataset. Also the two text-based enrichment methods are included. As shown in Table 2, with text enrichment, both SD-PG and SPY achieve better

performance than the non-enriched T2I methods on both FID and IS metrics. However, we find that the two enrichment approaches fail to show superior results on the CLIP scores. This is mostly because text-based enrichment methods suffer the aforementioned issues of wrong binding or visual distraction , etc., which results in lower matching between the generated images and raw (abstract) texts. In contrast, our model presents the overall optimal performance, significantly. This directly verifies the effectiveness of our method with the scene graph hallucination mechanism, i.e., yielding reasonable imaginations of scenes and leading to high-quality abstract-to-intricate T2I synthesis.

4.3 Ablation Studies

Here we present the model ablations to quantify the contribution of each part of our system. The results are shown in Table 3. First, canceling the whole SGH module (w/o both SG guidance and imagination), the most significant performance drops are witnessed, indicating the pivot influence of the SGH mechanism. Also, the drops from 'SGH → Init.SG' directly reveal the importance of our scene enrichment mechanism. Interestingly, by comparing the ablating drop in COCO and COCO-A2I, we can notice that the SGH mechanism is especially more important under the abstract-to-intricate setup, while the SG features more stand out for the general T2I scenario. Besides, both properly modeling SG features during denoising and modeling the SG nodetype dependence (NTD) contribute to the overall

Table 3: Ablation results (FID ↓ ). 'SGH → Init.SG': using the initial SG of text without SG enrichment throughout the whole T2I generation process. 'w/o SGH': removing the overall SGH module, i.e., also meaning both without scene enrichment and SG guidance. 'HSI → GCN': encoding the SG with a GCN [35] instead of our HSI mechanism.

performance. Further, when we abandon the hierarchical manner of the scene feature integration (HSI) and instead use a GCN to encode the overall SG, there are also clear performance drops.

4.4 Qualitative Results

To gain a more intuitive understanding of our model's capability on the abstract-to-intricate T2I synthesis, here we show some qualitative results. We visualize the generated images by different methods, where the given input prompts are in short and abstract format but describe intricate visual scenes. As shown in Figure 5, we find that Frido fails to generate images that precisely reflect the prompt instructions. In contrast, the text-based enriched method, SPY, yields visual results with much more details, in which the visual semantics, unfortunately, deviates much from the raw inputs largely. Overall, Salad is able to produce high-quality images with rich visual scenes and sophisticated contexts, meanwhile ensuring semantic accuracy, i.e., coinciding with the abstract input texts.

4.5 Analyses and Discussions

Via the above experiments, we have demonstrated the technical efficacy of our model. Following we further explore how our methods advance.

▶ Q1: How does SG guidance aid the generation of highquality images? We first consider assessing the overall semantic matchness between the input texts and the generated images. We make comparisons with baselines without integrating SGs. By observing the CLIP scores in Figure 6, it is clear that our model can achieve higher semantics faithfulness of image synthesis. Further, we try to probe the semantic structural alignment between the inputs and outputs. We mainly measure the ' object-attribute ' correspondence between input texts and generated images via GLIP metric; and also assess if the structural triplets in initial textual SG can also be retrieved in the SG of the generated image (i.e., TriRec. metric). As shown in the

66

70

74

76

10

12

14

16

79

81

83

85

(%)

CLIP

↑

(%)

GLIP

↑

(%)

TriRec.

↑

LDM

Frido

SPY

Salad

∗

figure, our Salad system consistently performs much better than the baselines in terms of these two metrics. This significantly suggests that with the SG guidance, the structural controllability of the T2I process is greatly enhanced, and thus leads to high-quality generation.

▶ Q2: Does SGH indeed induce intricate and reasonable SGs? Firstly, in Figure 7 we explore whether the SGH can produce new SG structures during the T2I process, on the COCO and COCO-A2I datasets separately. As seen, after scene enrichment, the average numbers of all three element types (object, attribute, and relation) substantially increase. Notably, the addition is more evident on the COCO-A2I set, where scene enrichment is more needed.

Obj.

Attr.

Rela.

1

3

5

7

Obj.

Attr.

Rela.

1

4

7

9

COCO

COCO-A2I

Initial SG

Imagined SG

Next, we examine if the newly imagined SG structures provide reasonable scenes to the input prompt. We reach this by measuring the recall rate (TriRec.) of the ' subject-predicate-object ' pairs between two SGs, i.e., the predicted and the gold SGs. Assuming the SGs (denoted as G I ) of gold images entail reasonable scenes, we consider two types of SGs from our 'prediction': 1) SGs induced by SGH ( G ), and 2) SGs parsed from our synthesized images ( G I ). We also compare with the text-enriched SPY method. As shown in Table 4, we first observe that the induced SGs highly align with the gold SGs, with 82.01 TriRec. score, indicating the

Table 4: Comparing the glod SGs ( G I ) with our induced SG ( G ), and the SG ( G I ) of generated image with TriRec. metric.

induced SGs are sensible for generating high-quality images. Moreover, by comparing the TriRec. scores ( G I vs. G I ) between SPY and Salad (i.e., 78.61 vs. 86.04), we learn that the synthesized images coordinated by imagined SGs more correspond to the gold images in terms of semantic scene structure. This also suggests that the SGH can induce valid SGs which favorably guide the image generation process.

▶ Q3: Does sampling strategy helps diversified T2I?

0

.

3

0

.

5

0

.

7

40

60

80

(a) Auto Diversity Score

LPIPS

(b) Human Evaluation

(%)

LDM

Frido

Salad

Salad w/o Sampling

strategy. This demonstrates that our proposed scene sampling mechanism is effective in helping diversified T2I. In Appendix §C.2 we show more examples of diversified generations via the scene sampling mechanism.

To measure the effectiveness of the sampling strategy in diversifying image generation given the same prompt, we consider both the qualitative analyses and human evaluation, where the former calculates the LPIPS score [60] to assess the perceptual similarity between two images in deep feature space. As shown in Figure 8, our Salad model significantly outperforms LDM and Frido in terms of diversity score and human evaluation, and things go worse upon removing the scene sampling

5 Related Work

T2I is a long-standing topic in computer vision and multimodal communities. Arrays of explorations have been devoted to achieving stronger image synthesis performances with various deep generative models, such as GANs [4, 17, 48, 52, 61], VAEs [29], flow-based approaches [11, 12] and ARMs [7, 13, 39, 49]. More recently, the diffusion denoising probabilistic methods (DDPMs) have revealed the greatest potentials on image synthesis, in which the optimal density estimation is more naturally achieved with a fixed diffusion process to transform an image into a Gaussian noise [9, 19, 23, 42, 47]. This work follows the line of diffusion methods and takes the SoTA latent diffusion model (LDM) [42, 14, 19] as the T2I backbone.

Generating high-quality images with complex scenes is the key criterion of a sound T2I system [14, 15, 26]. Many efforts have been paid for synthesizing more realistic and nature-looking images in sophisticated scenes, yet most of which are conditioned on taking the detailed descriptions as inputs [6, 51]. Thus, how to generate high-quality images in intricate scenes from succinct and abstract prompts becomes a meaningful yet challenging task. In this work, we introduce a scene hallucination mechanism, which, built upon the SG structure, performs more accurate and controllable scene completion and eventually helps generate intricate images of higher quality. We consider the SG representations [27] for the input prompt texts as well as the guidance of image synthesis. SG advances in intrinsically describing the semantic structures of scenes for texts or images [28, 53], enabling more fine-grained control of complex scenes [53], and thus aiding the final image generation.

Scene graph (SG) is a type of structured data that represents multiple objects and their complex relationships in the vision or language scenes, wherein the nodes denote objects & attributes and the edges depict relationships between objects [27]. As intrinsically describing the semantic structures of scene semantics for the given texts or images, SG has been widely utilized as a type of external feature being integrated into downstream applications for enhancements, e.g., image retrieval [27, 56], image generation [28, 53] and image captioning [54, 36, 55]. In this work, we consider the SG representations for the input sentences. Compared to the linear sequential nature of the text, SG offers a more intuitive manner to represent the scene semantics in a structured format, enabling more fine-grained control of complex scenes [53], and thus aiding the final image synthesis.

Our SGH mechanism also relates to the research of SG enrichment or imagination. While existing methods mostly approach the task by incrementally parsing SG elements [16, 57, 58], such greedyincrement paradigm may largely suffer from trapping in locally optimal SG generation, thus leading to inferior SG imagination. Instead, in this paper, we implement SGH as a discrete denoising&diffusion process. Discrete diffusion technique [25] is the latest introduced method that replaces the continuous state in standard diffusion models with a discrete one. During each denoising step, the entire SG structure is updated and optimized from a global viewpoint, so as to yield a more reasonable enrichment of scene structure. Besides, both the T2I and the SGH are modeled as the same diffusion process in our framework, where the two processes are well synchronized, such that the underlying SG features can perfectly guide the T2I synthesis at each step. To our knowledge, we are the first to investigate SG induction using discrete diffusion models.

6 Conclusion

In this work, we explore the text-to-image synthesis task under the abstract-to-intricate setup. Drawing inspiration from human intuition, we propose a scene-graph hallucination mechanism, which carries out scene imagination based on the initial scene graph of the input prompt, expanding the starting SG with more possible specific scene structures. We then develop an SG-based hallucination diffusion system for the abstract-to-intricate T2I, which mainly includes an SG-guided T2I module and an SGH module. Specifically, we design the SGH module based on the discrete diffusion technique, which evolves the initial SG structure by iteratively adding new scene elements. Then, we utilize another continuous diffusion model as the T2I synthesizer, where the overt image-generating process is navigated by the underlying semantic scene structure induced by the SGH module. On the standard COCO dataset, our system shows great superiority in the abstract-to-intricate T2I generation. Further analyses demonstrate that our SG-based hallucination mechanism is able to generate logically sound SG structures, which in return helps produce high-quality scene-riched images.

7 Broader Impact

Benefits The current text-conditioned image generation approaches largely fail to the abstractto-intricate T2I due to a lack of necessary details of input prompts. In this work, we propose a novel scene-graph hallucination mechanism inspired by human imagination intuition, which expands upon the initial scene graph from the text prompts to generate more feasible and specific scene structures. Furthermore, the enriched timestep-wised SG is leveraged to navigate the T2I generation process, leading to synthesizing more intricate images. Our study demonstrates that hallucinating images based on scene graph structures offer scalability, and modeling these structures enhances the generation of high-quality images with improved semantic controllability.

Potential weakness There can be two potential weaknesses that warrant consideration in our system. Firstly, the effectiveness of our system relies heavily on the quality of scene graph hallucination (SGH), yet the absence of a dedicated dataset for the SGH task poses a challenge in training the SGH module. However, we can leverage the richly annotated Visual Genome (VG) dataset, commonly used for training visual SG parsers, to provide initial training for the SGH module under an unconditional setting. Secondly, the training process of a diffusion model for text-to-image (T2I) generation entails substantial computational resources, resulting in increased energy consumption, CO 2 emissions, and potential environmental pollution.

References

Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Proceedings of the NeurIPS , pages 17981-17993, 2021.

Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. CoRR , abs/2302.08113, 2023.

Mohammad Bashiri, Edgar Y. Walker, Konstantin-Klemens Lurz, Akshay Jagadish, Taliah Muhammad, Zhiwei Ding, Zhuokun Ding, Andreas S. Tolias, and Fabian H. Sinz. A flow-based latent state generative model of neural population responses to natural images. In Proceedings of the NeurIPS , pages 1580115815, 2021.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Proceedings of the ICLR , 2019.

Shang Chai, Liansheng Zhuang, and Fengying Yan. Layoutdm: Transformer-based diffusion model for layout generation. CoRR , 2023.

Tuhin Chakrabarty, Arkady Saakyan, Olivia Winn, Artemis Panagopoulou, Yue Yang, Marianna Apidianaki, and Smaranda Muresan. I spy a metaphor: Large language models and diffusion models co-create visual metaphors. OpenReview , 2023.

Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proceedings of the ICML , pages 1691-1703, 2020.

Jun Cheng, Fuxiang Wu, Yanling Tian, Lei Wang, and Dapeng Tao. Rifegan: Rich feature generation for text-to-image synthesis from prior knowledge. In Proceedings of the CVPR , pages 10908-10917, 2020.

Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Proceedings of the NeurIPS , pages 8780-8794, 2021.

Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: Mastering text-to-image generation via transformers. In Proceedings of the NeurIPS , pages 19822-19835, 2021.

Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components estimation. In Proceedings of the ICLR , 2015.

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In Proceedings of the ICLR , 2017.

Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the CVPR , pages 12873-12883, 2021.

Wan-Cyuan Fan, Yen-Chun Chen, Dongdong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. CoRR , abs/2208.13753, 2022.

Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun R. Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. CoRR , abs/2212.05032, 2022.

Sarthak Garg, Helisa Dhamo, Azade Farshad, Sabrina Musatian, Nassir Navab, and Federico Tombari. Unconditional scene graph generation. In Proceedings of the ICCV , pages 16342-16351, 2021.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the NeurIPS , pages 2672-2680, 2014.

Jiuxiang Gu, Shafiq R. Joty, Jianfei Cai, Handong Zhao, Xu Yang, and Gang Wang. Unpaired image captioning via scene graph alignments. In Proceedings of the ICCV , pages 10322-10331, 2019.

Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the CVPR , pages 10686-10696, 2022.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the EMNLP , pages 7514-7528, 2021.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the NeurIPS , pages 6626-6637, 2017.

Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic object accuracy for generative text-to-image synthesis. IEEE Trans. Pattern Anal. Mach. Intell. , 44(3):1552-1565, 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the NeurIPS , 2020.

Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Towards non-autoregressive language models. CoRR , 2021.

Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. CoRR , abs/2303.08137, 2023.

Álvaro Barbero Jiménez. Mixture of diffusers for scene composition and high resolution image generation. CoRR , abs/2302.02412, 2023.

Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In Proceedings of the CVPR , pages 3668-3678, 2015.

Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the CVPR , pages 1219-1228, 2018.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the ICLR , 2014.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision , 123(1):32-73, 2017.

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded languageimage pre-training. In Proceedings of the CVPR , pages 10955-10965, 2022.

Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao. Object-driven text-to-image synthesis via adversarial training. In Proceedings of the CVPR , pages 1217412182, 2019.

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Proceedings of the ECCV , volume 8693, pages 740-755, 2014.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the ICLR , 2019.

Diego Marcheggiani and Ivan Titov. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the EMNLP , pages 1506-1515, 2017.

Victor Siemen Janusz Milewski, Marie-Francine Moens, and Iacer Calixto. Are scene graphs good enough to improve image captioning? In Proceedings of the AACL , pages 504-515, 2020.

Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the ICML , pages 16784-16804, 2022.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022.

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In Proceedings of the ICML , pages 4052-4061, 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the ICML , pages 87488763, 2021.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proceedings of the ICML , pages 8821-8831, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the CVPR , pages 10674-10685, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the MICCAI , pages 234-241, 2015.

Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proceedings of the NeurIPS , pages 2226-2234, 2016.

Brigit Schroeder and Subarna Tripathi. Structured query-based image retrieval using scene graphs. In Proceedings of the CVPR , pages 680-684, 2020.

Sebastian Schuster, Ranjay Krishna, Angel X. Chang, Li Fei-Fei, and Christopher D. Manning. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the EMNLP , pages 70-80, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proceedings of the ICLR , 2021.

Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Fei Wu, and Xiao-Yuan Jing. DF-GAN: deep fusion generative adversarial networks for text-to-image synthesis. CoRR , abs/2008.05865, 2020.

Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelcnn decoders. In Proceedings of the NeurIPS , pages 4790-4798, 2016.

Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proceedings of the NeurIPS , pages 6306-6315, 2017.

Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. P+: extended textual conditioning in text-to-image generation. CoRR , abs/2303.09522, 2023.

Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the CVPR , pages 1316-1324, 2018.

Ling Yang, Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming-Hsuan Yang. Diffusion-based scene graph to image generation with masked contrastive pre-training. CoRR , abs/2211.11138, 2022.

Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. In Proceedings of the CVPR , pages 10685-10694, 2019.

Xu Yang, Chongyang Gao, Hanwang Zhang, and Jianfei Cai. Hierarchical scene graph encoder-decoder for image paragraph captioning. In Proceedings of the ACM MM , pages 4181-4189, 2020.

Sangwoong Yoon, Woo-Young Kang, Sungwook Jeon, SeongEun Lee, Changjin Han, Jonghun Park, and Eun-Sol Kim. Image-to-image retrieval by learning similarity between scene graphs. In Proceedings of the AAAI , pages 10718-10726, 2021.

Jiaxuan You, Rex Ying, Xiang Ren, William L. Hamilton, and Jure Leskovec. Graphrnn: Generating realistic graphs with deep auto-regressive models. In Proceedings of the ICML , pages 5694-5703, 2018.

Xiang Yu, Ruoxin Chen, Jie Li, Jiawei Sun, Shijing Yuan, Huxiao Ji, Xinyu Lu, and Chentao Wu. Zero-shot scene graph generation with knowledge graph completion. In Proceedings of the ICME , pages 1-6, 2022.

Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the CVPR , pages 5831-5840, 2018.

Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the CVPR , pages 586-595, 2018.

Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. DM-GAN: dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the CVPR , pages 5802-5810, 2019.

A Extension of Technical Details

We in this part extend the specific details of our method techniques.

A.1 Discrete Diffusion Model for Scene Graph Hallucination

Here, we detail the forward and reverse processes in the discrete diffusion model [1] for SGH.

Forward Process In the forward process, we consider a node s ∗ t,j ∈ G t , ∗ ∈ { o, a, r } ( t is diffusion step, j is the node index) takes a scalar discrete value with K ∗ categories, s ∗ t,j ∈ 1 , · · · , K ∗ . Without introducing confusion, we omit subscripts j and superscripts ∗ in the following description. We define the probabilities that s t -1 transits to s t using the matrices [ Q t ] ij = q ( s t = j | s t -1 = i ) , then we can write the forward Markov diffusion process:

where B ( s t ) denotes the one-hot column vector which length is K . The categorical distribution over s t is given by the vector Q t B ( s t -1 ) . Due to the property of the Markov chain, starting from s 0 , we can derive the cumulative probability of s t at arbitrary t -timestep:

Besides, by conditioning on s 0 , the posterior of this diffusion process is tractable:

Note that the transition matrix Q t is capable of controlling the data corruption and denoising process, thus it should be carefully designed such that it is not too difficult for the reverse network to recover the signal from noises. We follow [1, 19] that exploits a mask-and-replace strategy to design a Q t , which can be defined as:

where for each node s t , there are three probabilities, i.e., a probability of γ t to transition to [MASK] node, a probability of Kβ t be resampled uniformly over all the K categories, and a probability of α t = 1 -Kβ t -γ t to stay the same node. Notedly, the [MASK] node never transitions to other states. The aforementioned discrete state-space models assume that all the nodes are switchable by corruption. However, it is unreasonable that an object node, 'bed', transitions to a relation node, 'in'. To avoid such invalid transition, we propose to apply disjoint corruption matrices Q o t , Q a t , Q r t for object, attribute, and relation nodes, respectively.

Reverse Process In the reverse process, an SG decoder is introduced as a denoising network to estimate the posterior distribution p θ ( · ) , which takes the node token s t , time step t and text prompt y . Specifically, each layer of the SG decoder contains two parts: 1) an adaptive normalization (AdaLN), and 2) a transformer block. AdaLN is applied to inject the timestep information:

where CLIP ( · ) denotes an encoding layer for projecting the node token, and a t and b t are obtained from a linear projection of the timestep embedding. Each transformer block contains 1) a full self-attention, i.e., H ∗ = Softmax ( H ∗ · H ∗⊤ √ d 1 ) · H ∗ . 2) a text-cross-attention (Text-CA) integrates the input prompt y :

where H y is a conditional feature sequence yielded by a text encoder first takes the text prompts y as input. Then, 3) a graph cross-attention (Graph-CA) is devised to take the induced SG ( G t +1 ) in the previous t +1 timestep:

man

in front of

wear

carrying on

Tshirt

luggage bag

sunglass

young

black

wear

building

old

Object level

Relation level

Region level

Global level

The whole SG

SG

where H G and E G is the embedding representation of the nodes and edges in G t +1 , respectively, Next, 4) a node-type dependency cross-attention (NTD-CA) is designed for the ∗ -type node induction (* can be object or attribute):

Finally, a softmax function is put on to obtain the category label distributions:

Following [1, 19], the SG decoder is optimized by minimizing the variational lower bound (VLB):

where p θ ( s T ) is the stationary distribution at timestep T . To estimate the p θ ( s T ) , we extend the forward process by appending a rank-one matrix Q t +1 that ignores s T and produces a deterministic s T +1 ∈ G T +1 where G T +1 is the initial SG parsed from the text prompt y , then learning the reverse step p θ ( s T | s T +1 ) = p θ ( s T ) . Furthermore, we follow the reparameterization trick proposed in [19], which lets the network predict the noiseless token distribution p θ (˜ s 0 | s t , y ) at each reverse step so as to gain better quality. Based on the reparameterization trick, an auxiliary objective is introduced:

To combine this loss with L vlb , the final training objects are defined as follows:

where λ 1 is a hyper-parameter to control the effect of the auxiliary loss L s 0 .

A.2 Scene-driven Image Synthesis

As described in §3.2, we propose a hierarchical scene integration strategy to effectively integrate the SG features. Here, we give more details on this part. Firstly, as shown in Figure 9, the SG is able to represent four different hierarchical levels of semantics corresponding to the image:

Object level . In the process of image synthesis, a crucial aspect lies in accurately generating each specified individual object, corresponding to the object node and associated attribute notes in SG, for example, black Tshirt , old building , young man , luggage bag , sunglass .

Region level . Regional image generation focuses on multiple objects and entangled relationships among them, which aligns with the presence of overlapping relation triplets 7 in SG. Here we adopt two overlapped relation triplets as the region representation of the image, such as man wear Tshirt in front of building , man wear Tshirt carrying on luggage bag , man wear

Relation level . Ahigh-quality image is not only the generation of objects but also their intricate relationships which connect two objects, akin to the subject-predicate-object triplets found in the SG, such as man wear Tshirt , man in front of building , man carrying on luggage bad , man wear sunglass .

Thirt wear sungalss , man in front of building carrying on luggage bag , man in front of building wear sunglass , man carrying on luggage bag wear sunglass .

Global level . The whole SG provides global semantics to guide the generation of images.

Then, we technically extract a collection of concepts from the four semantics levels in the SG, denoted as L o (object level), L r (relation level), L c (region level), and L g (global level). We encode each concept separately:

We further maintain the representation of these four levels as the keys K L i and values V L i by linear transformations:

where i ∈ { o, r, c, g } . Next, we integrate these features via the Transformer attention of U-Net in LDM, which is formulated in Eq. (7).

B Detailed Experiment Settings

B.1 Datasets

COCO The training and validation data numbers in COCO are 83K and 41k, respectively. We note that, in the evaluation phase, models are evaluated on the full COCO 2014 validation set.

Visual Genome (VG) Visual Genome [30] version 1.4 (VG) comprises 108,077 images annotated with scene graphs. Following previous work [28], we use object and relationship categories occurring at least 2,000 and 500 times respectively in the dataset, leaving 178 objects and 45 relationship types, and we ignore small objects, and use images with between 5 and 30 objects and at least three relationships, this leaves us with 62,565 images with an average of 10 objects and 5 relationships per image.

Construction of COCO-A2I We elaborate on the process of constructing the COCO-A2I dataset in the following three steps:

First, we consider Place Nouns and Progressive Verbs are two types of abstract words that can depict intricate images. Therefore, we pre-define the candidate list of Place Nouns: street , sidewalk , kitchen , restroom , bathroom , living room , bedroom , hostel , house , office , bank . and the candidate list of Progressive verbs: traveling , knitting , gardening , shopping , presenting , drawing , baking , studying , etc.

Third, we filter out the primary-filtering dataset in which the number of words in the captions of instances is more than 10 and the number of bounding boxes is less than 6, obtaining the final target COCO-A2I dataset.

Second, we select the COCO valid dataset in that captions of instances contain the words in the candidate list, obtaining the primary-filtering dataset.

After the three-step pipeline, we obtain 2,005 text-image pairs.

In Figure 10, we show some examples, where the images tend to contain intricate content, including multiple objects, attributes, and relationships while the corresponding text prompts are relatively short and abstract.

B.2 Evaluation Metric Implication

We employ Fréchet Inception Distance (FID) [21], Inception score (IS) [44], CLIP score [20], and GLIP [31] used in [14] to quantitatively evaluate the quality of the generated images, and Learned Perceptual Image Patch Similarity (LPIPS) [60] utilized in [42] to quantify the diversity of the generated images. Additionally, we introduce Triplet Recall (TriRec.) to measure the percentage of the correct relation triplet among all the relations. Technically, given a set of ground truth triplets ( subject-predicate-object ), denoted GT , and the TriRec. is computed as TriRec.= | PT ∩ GT | / | GT | , where PT are the relation triplets extracted from the generated images by a visual SG parser.

HAC

B.3 Human Evaluation Criterion.

In §4.5, we conduct a human evaluation, i.e., user study, for more intuitive assessments of the diversification quality of T2I models. Specifically, we generate 10 images conditioned on a single text prompt and ask evaluators to cluster these images based on their contextual information, such as objects, attributes, and relationships present in the generated images. After evaluating 100 distinct text prompts, we compute the average clustering ratio:

where | Cluster | denotes the number of clusters for 10 images generated by a text prompt. Average ( · ) is an average function. Here, a higher ratio score signifies greater diversification in image generation.

C More Experiment Results

C.1 Visualizations of SGH Process

In Figure 11, we visualize the SGH process with two examples, where we sample several time steps and plot the generated SGs and the corresponding images. As can be seen, our SGH is capable of imaging certain reasonable structures of SGs to enrich the initial SG.

C.2 More Cases

Here we showcase more examples of the generated images via the scene sampling mechanism in Figure 12 and 13. We synthesize 10 images for each given text prompt.

Text Prompt (a) : A fancy chandelier hanging in a small kitchen.

kitchen

chandelier

handing in

fancy

small

Initial SG

chair

kitchen

chandelier

handing in

table

flower

big

small

window

in

in

next to

in

on

oven

has

chair

kitchen

chandelier

handing in

table

window

in

in

next to

in

oven

has

small

flower

fancy

small

on

chair

kitchen

chandelier

handing in

table

window

in

in

next to

in

oven

has

black

white

big

white

glass

Text Prompt (b)

: A small room with painting stuffs.

stuffs

room

with

small

painting

a

room

with

couch

in

walls

easel

painting

in front of

on

room

with

couch

in

walls

easel

painting

in front of

on

window

canvas

beside

brush

can

floor

on

small

on

on

in

room

with

couch

in

walls

easel

painting

in front of

on

window

canvas

beside

brush

can

floor

on

small

on

on

in

wooden

oil

white

green

red

black

messy

Initial SG

06=

t=80

t=70

09=

t=50

t=40

t=30

t=10

0=3

06=

t=80

t=70

09=

t=50

t=40

t=30

t=10

0=z

国

SLI

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, text-to-image synthesis under the abstract-to-intricate setting is studied. Firstly, the input prompt is hallucinated and expanded into feasible specific scene structures by the proposed SGH mechanism. Then, text-to-image synthesis is implemented through a diffusion-based synthesizer by gradually incorporating semantic scene structure induced from the SGH. Extensive experiments on COCO, especially on the abstract-to-intricate text-to-image setting, prove the method could contribute to synthesizing images reasonably and accurately under the simple text.\n\nSTRENGTHS:\n1.\tThe proposed abstract-to-intricate T2I is a vital research topic in the field of text-to-image synthesis since existing large models require delicate and well-designed prompts for controllable image synthesis.\n2.\tAbstract-to-intricate T2I is further promoted by the designed SGH mechanism, avoiding vision distraction from text and wrong focus enriched prompts.\n3.\tExtensive experiments including thorough metrics are conducted to prove the effectiveness. And corresponding analysis and discussion are reasonably stated.\n\nWEAKNESSES:\n1.\tExisting experiments compared to the text-based enrichment methods are only conducted on manually selected dataset COCO-A2I with Place Norm and Progressive Verbs, which is not representative and convincing enough.\n2.\tThere exists unfair comparison in T2I results on COCO, since the best FID and CLIP score of Frido are respectively 8.12 and 0.7915. And the best T2I baseline on COCO is not Frido. As far as I know, make-a-scene[1] achieves a 7.55 FID, which is not discussed.\n3.\tScene graphs consist of two parts, which are node and edges representing semantics, bounding boxes referring to sizes and locations of objects. As proved by previous works, incorporating visual guidance into T2I training is beneficial. Why bounding boxes information is not included for training?\n4.\tOverclaims and inaccurate description in contributions:” We propose a diffusion-based model with SG guidances for highly controllable and scalable image generation.” Scene graph hallucinations might include unexpected concepts, which is not controlled by users.\n5.\tUnclear captions and inconsistent description. The caption of Fig.2 is unclear and lacks descriptions about each subfigure. \n6.\tCitations about scene graph-to-image synthesis and scene graph generation are not thoroughly included[2-4].\n7.\tTypos and inconsistent descriptions: In line 226, “diffusioninspired”. In line 245, Frido-G is inconsistent with the description on Tab.1. \n\n[1] Gafni O, Polyak A, Ashual O, et al. Make-a-scene: Scene-based text-to-image generation with human priors[C]//Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV. Cham: Springer Nature Switzerland, 2022: 89-106.\n[2] Sitong Su, Lianli Gao, Junchen Zhu, Jie Shao, and Jingkuan Song. 2021. Fully Functional Image Manipulation Using Scene Graphs in A Bounding-Box Free Way. Proceedings of the 29th ACM International Conference on Multimedia. Association for Computing Machinery, New York, NY, USA, 1784–1792. https://doi.org/10.1145/3474085.3475326\n[3] Lyu X, Gao L, Guo Y, et al. Fine-grained predicates learning for scene graph generation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 19467-19475.\n[4] Herzig R, Bar A, Xu H, et al. Learning canonical representations for scene graph to image generation[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16. Springer International Publishing, 2020: 210-227.\n\nQUESTIONS:\n1.\tPlease provide results compared to text-based enrichment methods on COCO dataset.\n2.\tPlease discuss results compared to the best Firdo other T2I baselines like make-a-scene.\n3.\tWhy bounding boxes of scene graphs for visual guidance are not included in training? Will there be gain by adding bbox?\n4.\tPlease modify writing issues and answer questions in 4,5,7 of Weakness.\n5.\tCitations are missing as described in 2, 6 of Weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a new setup of generating intricate images from abstract prompts. To overcome the issues of vision distraction and wrong binding of using text as the condition to generate images, the author proposes a two stage pipeline by first generating scene graph from abstract text inputs, and then condition on the sythesized scene graph, another model generates images. The proposed diffusion model with SG guidance showcases its controllability and interoperability and it achieves new SoTA results in the abstract-to-intricate T2I setup. Overall, the paper is well-written and the proposed method is interesting and novel.\n\nSTRENGTHS:\nThe introduction section is easy to follow, they provide examples to show the issues of the current T2I generation and motivate the scene graph representation as a potential solution.\n\nThe proposed SGH model is interesting and well adopted from VQ-diffusion etc for image modelling to the problem of scene graph modelling.\n\nThe experiments are convincing with strong performance on COCO, outperforming competitive baselines LDM, VQ-diffusion and Frido, they also construct an abstract-to-intricate dataset from COCO and demonstrate SoTA performance in this setup.\n\nWEAKNESSES:\nOne claim of the paper is that SG guidance helps image generation with strong semantic controllability, it is not clear to me which experiments can support this claim.\n\nIn table 3, it seems like replacing the HSI module with GCN encoding only drops a little bit of the performance, it is questionable if the design of HSI is necessary.\n\nQUESTIONS:\nIt seems like using SG as condition/guidance will lower the FID compared to text conditioning and also in qualitative results, details like faces will be blurry. Any idea how to improve this and explanation of the cause?\n\nAny idea after enriching the SG, if users want to modify some nodes, how would the user interact with the SG/model?\n\nLIMITATIONS:\nAs pointed out by the author, the proposed method depends on the generation quality of the SG, while a large-scale SG dataset is rare. Plus, since the pipeline needs to modify the conditioning of the LDM, how to better leverage pre-trained large T2I model is worth investigating.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach to text-to-image (T2I) synthesis, specifically focusing on generating intricate visual content from simple abstract text prompts, aka., abstract-to-intricate (A2I) setting. The proposed mechanism, named scene-graph hallucination (SGH), expands the initial scene graph (SG) of the input prompt with more feasible specific scene structures using discrete diffusion technique. A continuous-state diffusion model then serves as the T2I synthesizer navigated by the semantic scene structure induced from the SGH module. Additionally, this paper further devises a scene sampling mechanism to generate various scene graphs. They also construct a more challenging benchmark data, called COCO-A2I, to effectively evaluate the models under the abstract-to-intricate setting. Experiments on two benchmark datasets shows that leveraging SG imagination helps better image generation, where Salad could hallucinate certain scene clues to facilitate the abstract-to-intricate T2I generation. The study also contributes to a better understanding of the efficacy and rationale behind scene graph features, with potential applications in other tasks such as image editing and text-to-video generation.\n\nSTRENGTHS:\nOverall, I enjoy the work much where it studies the specific conditioned image synthesis from an interesting and realistic perspective, with robust and novel technical methods. The paper seems very solid to me, with thorough evaluations from many angles, and the details are given quite sufficiently (with long and informative appendix content). While this is a very technical paper, there is immense interest in diffusion models under such novel perspective. I believe this paper will have the potential to unlock interesting future research.\n\n**Interesting and meaningful perspective.**\nThe paper studies how to generate intricate images from succinct abstract prompts. This can be a very interesting and realistic perspective in many scenarios. The issue essentially lies in the natural modality asymmetry between language and vision, where strict one-to-one correspondences do not exist. This becomes particularly challenging when T2I systems attempt to capture the nuanced content of the prompts and generate corresponding high-quality images, thereby highlighting the need for deep semantic understanding in these systems.\n\n**Innovative methodology.**\nThe authors propose a novel approach to the task of abstract-to-intricate T2I synthesis, where the image-generating process is controlled and navigated by the underlying semantic scene structure. On the one hand, it satisfies the human intuition for handling this task, which is pretty interesting and makes a lot sense to me. If the system could hallucinate some concrete textual clues which has a more corresponding relationship to the visual scenes, the generation process will ease largely. Technically, the devised method, called scene-graph hallucination (SGH), expands the initial scene graph (SG) of the input prompt by iteratively evolving new scene elements via discrete diffusion model, which are theoretically sound and empirically validated through experimental results. On the other hand, the hierarchical scene integration mechanism is able to ensure the highly effective guidance of the semantic scene features.\n\n**Solid evaluations and convincing analyses.**\nThe results are presented clearly and concisely, showcasing a significant improvement for the abstract-to-intricate T2I task at hand. Some in-depth analyses are provided to offer substantial evidence and valuable insights from all different perspectives to support the claims made, such as, exploring the effectiveness and rationale behind the employed scene graph features. Also, the comprehensive implementation details will enhance reproducibility and facilitates the smooth adoption of the proposed approach. The codes are provided.\n\n**Well-structured presentation and clear writing.**\nThis paper is exceptionally clear, well-written (except some notation problems) and good illustration, making it easy to understand. The details are given quite sufficiently, with long and informative appendix content. I’ve gone through the appendix and find almost everything I wanted to see.\n\nWEAKNESSES:\nWhile I think the paper is solid, it can be significantly improved further if the following issues (major or minor) are considered properly:\n\n(ordered by appearance in the paper)\n\n1. In Figure 1, the issues of vision distraction and wrong binding appear to represent the same issue where the resulted images deviate from the original user intention. Besides, it would be beneficial to maintain consistent terminology throughout. For example, in Figure 1(a), it is referred to as 'vision distraction', whereas in the article, it is mentioned as \"visual distraction.\" \n\n2. Although the authors devise a scene sampling mechanism to generate various scene graphs for diversified image syntheses during inference, the diversity of scene graph remains inadequate. This could be due to the limited number of categories of objects, attributes or relationships in the SGH module, which restricts the model's imagination and exploration capacity. In real scenarios, the objects, attributes can be quite multifarious, for example, the color of the T-shirts.\n\n3. Potentially lack of comparison of the existing scene graph completion work. The performance of scene graph hallucination has a huge impact on the performance of abstract-to-intricate T2I task. It would be interesting to see the performance comparison on existing work about scene graph hallucination, such as [1,2]. Moreover, another intuitive method is the pipeline method, i.e., first scene graph hallucination and then scene graph generation based the imagined SG. Therefore, some evaluating experiments should be provided to make a comparison with the pipeline method.\n\n4. The authors may possibly overlook the imagination ability of the T2I model itself. For example, existing T2I models can synthesis an image based on the prompt, 'a man'. Is the appearance of the man in the generated image, such as hair color, attributed to the imagination of the T2I model?  In other words, the authors fail to provide a clear definition of when the model needs to image. Furthermore, they do not delve into what scene graph needs to be detailed in order to generate intricate images effectively. For instance, in Figure 1(b), the enrich SG not extremely correspond to the image as the screen on the wall are not demonstrated in the enriched SG, but the model is still able to generate it.\n\n5. Some unclear and confusing annotations:\n\n- In Figure 2 right bottom, it is confusing $a_{n, 1}$ whether one value, i.e., the m-th attribute of the object $o_1$, or many values. Same issues for $r_{n,k}$.\n\n- In line 150, the distribution should be $p_{\\theta}(s_t|s_{t+1}, y)$.\n\n- Is $\\mathcal{B}{s_t}$ a row one-hot vector or column one-hot vector? This should be clarified. Besides, there should be a transition symbol in Eq. 2.\n\n- In Eq. 1, the $\\mathcal{L}_{vlb}$ is not explicitly designed for the optimization of conditional image generation. Thus, some clarification should be added, and the correct loss function is demonstrated in Appendix B.1, specifically Equations 19-22.\n\n- No illustration of the $d$ in Eq. 7.\n\n- In the Implementation section, the version of CLIP mentioned is not consistent with the information provided in Appendix C.5. \n\n- No demonstration of the hyper-parameters? top-A and the temperature $\\eta$ in the inference?\n\n- In Table 3, the NTD-CA should be Eq. 5, not Eq. 17\n\n- In Line 314, not ‘object-object’ pairs but ‘subject-predicate-object’ triplets.\n\n- In Line 738, ‘2.005’ should be written as ‘2,005’.\n\n- In Table 5, the second ‘Max’ should be ‘Avg.’\n\n\n6. The paper lacks in-depth analysis of the circumstances under which the proposed SGH module fails to generate reasonable scene graphs, leaving a gap in understanding its limitations.\n\n7. In the section 4.5, the question Q2 is interesting but strikes me as a rather weird way to answer the question. The Figure 7 precisely answers that the SGH is able to induce intricate SGs. As discussed in Q1, there is a strong semantic alignment between the input prompts and the generated images guided of the SG. Consequently, it becomes relatively straightforward to achieve a high TriRec. Score by comparing the induced SG with the SG of the generated image. Nonetheless, this response lacks directness and comprehensiveness regarding whether SGH is capable of producing reasonable SGs. Besides, the concept of a \"reasonable\" SG remains ambiguous, For example, if the ‘table-in-room’ is a reasonable scene triplet, what about the triplet ‘table-in-ocean’? If a reasonable SG means some SGs that conforms natural conditions, can our proposed model generate some abstract or unconventional images?\n\n\n[1] Garg S, Dhamo H, Farshad A, et al. Unconditional scene graph generation. 2021.\n\n[2] Agarwal R, Chandra T S, Patil V, et al. GEMS: Scene Expansion using Generative Models of Graphs. 2023\n\nQUESTIONS:\n1. Is there any further explanation about the order of the Text-CA, Graph-CA, NTD-CA?\n2. During the SG hallucination, other visual scenes are also introduced. How does the proposed method deal with the visual distraction issues?\n3. How to ensure that the objects, attributes, and relationships in the initial SGs are always remain in the imaged SGs? \n4. Has the main results in Table 1&2 adopted the scene sampling mechanism? So is this the average of multiple experimental results?\n5. In HIS, which semantic level has a significant impact on task performance?\n6. How to leverage the VG dataset to train the SGH module?\n\nLIMITATIONS:\nI do not foresee any potential negative impact from this work.", "TITLE": "Official Review", "RECOMMENDATION": 10, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new setting (or sub-domain) of text-to-image generation (T2I), namely abstract-to-intricate T2I. To tackle the new setting, the authors propose a method (Salad) to enrich the scene graphs parsed from the text prompts based on the discrete diffusion models. The enriched scene graphs are used as guidance to generate complicated scenes that align better with the initial concise prompts. They report quantitative results and analysis experiments to show the effectiveness of the two-stage system in multiple metrics.\n\nSTRENGTHS:\n- The paper addresses a practical and important problem. Users of the T2I system, in reality, have to write unreasonably long text prompts to generate images with plausible scenes/styles/semantics. Abstract-to-intricate T2I could benefit the users with a more faithful and efficient generation process. \n\n- The scene graph hallucination stage achieved with the discrete diffusion model is interesting. Enriching the text prompt in the scene graph space seems like a more controllable and stable process that can preserve the faithfulness of relations and attributes. \n\n- The authors conduct an extensive analysis of the components of the system to demonstrate its effectiveness.\n\nWEAKNESSES:\n- Task definition. While I understand the concept of abstract-to-intricate, I think the work lacks a more rigorous definition of the task. It seems unclear why places and progressive verbs are included and other nouns or verbs are discarded in the COCO-A2I. I tried the failure prompt in Fig. 1 on Stable Diffusion, and I got good results most of the time. So what makes these prompts an A2I problem instead of a faithfulness problem of all T2I models?\n\n- My major concern is about the experimental setup. It seems that the Salad system is using the pre-trained Stable Diffusion as the image generator compared to other methods like Frido and LDM-G. The comparison could be unfair as all these baselines are trained with much fewer images and inherently have higher FID scores on MSCOCO. In addition, the authors use Frido for text-based enrichment approaches. \n\n- The writing needs improvement. There are multiple typos throughout the paper and some inaccurate sentences. For instance, line 133 \"attribute (o)\", line 226 \"baselines: stable diffusioninspired by [6].\", line 235-236 \"For the SIS module, we load the parameters of Stable Diffusion3 (v1.4) as the initialization.\", line 238 \"UNet\" (UNet in Stable Diffusion?).\n\nQUESTIONS:\n- What is the SGH module's output and the SIS step's input? Fig. 2 seems like you are inputting the scene graph matrix into the UNet, while Sec. 3.2 and Fig. 3 show that you convert the graphs into prompts and feed them to the attention layers.\n\n- Did you fine-tune the Stable Diffusion model by saying \"For the SIS module, we load the parameters of Stable Diffusion3 (v1.4) as the initialization.\"? \n\n- Why do all models achieve such high CLIP scores? Is the CLIP score the cosine similarity between image-text pairs or the CLIP R-precision metric?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper aims at the abstractive setting for text-to-image generation (T2I). They propose scene-graph hallucination (SGH) to perform imagination over the scene graph of the input prompt and make up the missing information. Then SGH can perform better T2I with the completed scene graph. Experiments on the COCO dataset demonstrate that T2I can significantly bridge the gap of abstract-to-intricate T2I.

STRENGTHS:
+ This paper is well-written and easy to follow.
+ The goal of abstract-to-intricate T2I is important since we can not expect users always input their full prompt. This setting has the potential to improve practical applications.
+ The usage of scene graphs is effective and can lead to better imagination for T2I than the Chain-ot-Though (CoT) language prompt.
+ They provide detailed ablation studies from different aspects (Table 3/4 and Fig. 6/7) as well as rich qualitative examples (Fig. 5 and Fig. 12/13 in Appendix).

WEAKNESSES:
- Maybe I missed it, but what is the motivation for using scene graphs instead of LLM-completed prompts? Does the imagination over scene graphs work more robustly than LLM?
- Since they rely on imagination to deal with abstractive issues, the hallucination situation also raises. It will be better to evaluate this issue, and how is the trade-off between this and the T2I performance. How to avoid or mitigate this deficiency in the proposed framework?

QUESTIONS:
I do not have additional questions. Please see the Weakness and refer to other reviewers.

LIMITATIONS:
The hallucination issues over scene graphs should be carefully addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-11