SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Data and Resource We conduct T2I generation experiments mainly on the COCO [33] dataset. We also prepare the abstract-to-intricate SG pair annotations for training the SGH module, where we employ an external textual SG parser [46] and a visual SG parser [59] on the paired images and texts in COCO, to obtain the initial SG and imagined SG, respectively. To enlarge the abstract-to-intricate SG pairs, we further extend Visual Genome (VG) [30]. Besides, to simulate the abstract-to-intricate T2I scenario, we further manually extract a subset of text-image pairs from raw COCO data (named COCO-A2I), in which the texts are short and abstract, while the images are comparatively complex and intricate. Appendix §B.1 shows all the dataset details.
Baseline and Evaluation We make comparisons with three types of existing strong-performing T2I models. 1) GAN-based models: AttnGAN [52], ObjGAN [32], DFGAM [48], OPGAN [22]. 2) Auto-aggressive model: DALL-E[41], and CogView [10]. 3) Diffusion-based models: LDM [42], VQ-diffusion [19], LDM-G and Frido [14] with classifier-free guidance. Note that LDM-G and Frido are the current SoTA T2I synthesizers. In addition, two types of text-based enrichment approaches are included as baselines: stable-diffusion prompt generator (SD-PG) and SPY inspired by [6]. The enriched text prompts are then utilized as inputs for Frido to generate the final images. We adopt three standard metrics to measure image synthesis performance: 1) Inception score (IS) [44], 2) Fréchet Inception Distance (FID) [21] and 3) CLIP score. Moreover, we use GLIP [31] to measure the fine-grained 'object-attribute' grounding in images, and Triplet Recall (TriRec.) measure the 'subject-predicate-object' triplet recall between two SGs. We also adopt the Learned Perceptual Image Patch Similarity (LPIPS) [60] for diversifying generation evaluation. Detailed definitions of evaluation metrics are shown in Appendix §B.2.
4We select the texts that mainly include two types of words, i.e., Place Nouns (e.g., office, airport, bathroom) and Progressive Verbs (e.g., traveling, knitting, gardening).
Table 1: The T2I results on the overall COCO dataset.  ${}^{\mathrm{b}}$  : taken from Fan et al. [14];  ${}^{\mathrm{h}}$  : copied from Ding et al. [10],  ${}^{\sharp }$  : taken from Hinz et al. [22]. The best score is in bold, and the second best is underlined.
Model	FID ↓	IS ↑	CLIP ↑
GAN Model
AttnGANb	33.10	26.61	-
ObjGANb	36.52	24.09	-
DFGANb	21.42	-	-
OPGAN#	24.70	27.88	-
Auto-aggressive Model
DALLE#	27.34	17.90	-
CogView#	27.10	18.20	-
Diffusion Model
LDMb	17.61	19.34	65.00
VQ-diffusionb	14.06	21.85	67.70
LDM-Gb	12.27	27.82	69.27
Fridob	11.24	26.84	70.46
Salad	10.19	29.96	73.83
Table 2: Results on the COCO-A2I subset for the abstract-to-intricate T2I generation.  ${\mathrm{{SPY}}}^{ \dagger  }$  denotes enriched texts are parsed into SG, and then perform SG-to-image generation via Frido.
Model	FID ↓	IS ↑	CLIP ↑
T2I Baseline
AttnGAN	78.19	11.09	52.78
ObjGAN	75.33	13.16	55.20
DFGAN	71.24	15.56	56.91
DALLE	66.36	16.03	63.05
CogView	62.85	16.98	63.97
LDM	55.27	16.20	67.79
VQdiffusion	69.14	15.78	64.58
Frido	40.36	18.36	68.53
Text-based Enrichment (+Frido)
SD-PG	36.50	17.64	65.23
SPY	35.59	21.59	67.86
SPY†	39.41	22.16	66.93
Salad	31.25	28.63	71.29
Figure 5: Qualitative results by different methods, where the given prompts randomly selected from the test set are short and abstract expressions (marked in red), while describing intricate visual scenes.
Implementation We define the maximum number of SG object nodes as 30, and each object node has a maximum of 3 attributes. We set the timesteps  $(T)$  for SGH and SIS as 100. For the SIS module, we load the parameters of Stable Diffusion $^5$  (v1.4) as the initialization. We use the CLIP $^6$  (vit-large-patch14) as our text encoder. We optimize the framework using AdamW [34] with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.98$ . The learning rate is set to 5e-5 after 10,000 iterations of warmup. For the attention layer in SG decoder and UNet in SIS, we define a shared configuration as follows: 4 layers, 8 attention heads, 512 embedding dimensions, 2,048 hidden dimensions, and 0.1 dropout rate. We mainly follow the prior works [54, 18, 46] to acquire the visual scene graph (VSG) from the gold image and textual scene graph (TSG) from the text prompt.
4.2 Main Comparisons
Table 1 shows the main T2I generation results on the overall COCO data. We see that diffusion-based methods are consistently better than the other two types of T2I methods, especially on the FID metric. Most importantly, our proposed Salad model yields the overall best results on all metrics. For example, we outperform Frido by  $1.05\%$  FID,  $3.37\%$  on CLIP score, and surpass LDM-G by  $2.1\%$  IS score. This directly demonstrates the efficacy of our method.
To directly assess the capability of solving the abstract-to-intricate T2I synthesis, we further make comparisons on the collected COCO-A2I dataset. Also the two text-based enrichment methods are included. As shown in Table 2, with text enrichment, both SD-PG and SPY achieve better
<sup>5</sup>https://github.com/CompVis/stable-diffusion
$^{6}$ https://github.com/openai/CLIP
performance than the non-enriched T2I methods on both FID and IS metrics. However, we find that the two enrichment approaches fail to show superior results on the CLIP scores. This is mostly because text-based enrichment methods suffer the aforementioned issues of wrong binding or visual distraction, etc., which results in lower matching between the generated images and raw (abstract) texts. In contrast, our model presents the overall optimal performance, significantly. This directly verifies the effectiveness of our method with the scene graph hallucination mechanism, i.e., yielding reasonable imaginations of scenes and leading to high-quality abstract-to-intricate T2I synthesis.
Here we present the model ablations to quantify the contribution of each part of our system. The results are shown in Table 3. First, canceling the whole SGH module (w/o both SG guidance and imagination), the most significant performance drops are witnessed, indicating the pivot influence of the SGH mechanism. Also, the drops from  $\mathrm{SGH}\rightarrow \mathrm{Init.SG}$  directly reveal the importance of our scene enrichment mechanism. Interestingly, by comparing the ablating drop in COCO and COCO-A2I, we can notice that the SGH mechanism is especially more important under the abstract-to-intricate setup, while the SG features more stand out for the general T2I scenario. Besides, both properly modeling SG features during denoising and modeling the SG node-type dependence (NTD) contribute to the overall
Table 3: Ablation results (FID↓). 'SGH→Init.SG': using the initial SG of text without SG enrichment throughout the whole T2I generation process. 'w/o SGH': removing the overall SGH module, i.e., also meaning both without scene enrichment and SG guidance. 'HSI→GCN': encoding the SG with a GCN [35] instead of our HSI mechanism.
Item	COCO	COCO-A2I
Salad (Ours)	10.19	31.25
\( \overline{\mathrm{{SGH}}}\rightarrow\overline{\mathrm{{Init}}}\cdot\overline{\mathrm{{SG}}} \)	12.45(+2.26)	37.77(+6.52)
w/o SGH	16.98(+6.79)	34.32(+3.07)
w/o Graph-CA (Eq.4)	10.95(+0.76)	32.74(+1.49)
w/o NTD-CA (Eq.5)	11.19(+1.00)	34.35(+3.10)
\( \overline{\mathrm{{HSI}}}\rightarrow\overline{\mathrm{{GCN}}} \)	10.87(+0.68)	35.96(+1.71)
performance. Further, when we abandon the hierarchical manner of the scene feature integration (HSI) and instead use a GCN to encode the overall SG, there are also clear performance drops.
To gain a more intuitive understanding of our model's capability on the abstract-to-intricate T2I synthesis, here we show some qualitative results. We visualize the generated images by different methods, where the given input prompts are in short and abstract format but describe intricate visual scenes. As shown in Figure 5, we find that Frido fails to generate images that precisely reflect the prompt instructions. In contrast, the text-based enriched method, SPY, yields visual results with much more details, in which the visual semantics, unfortunately, deviates much from the raw inputs largely. Overall, Salad is able to produce high-quality images with rich visual scenes and sophisticated contexts, meanwhile ensuring semantic accuracy, i.e., coinciding with the abstract input texts.
4.5 Analyses and Discussions
Via the above experiments, we have demonstrated the technical efficacy of our model. Following we further explore how our methods advance.
Q1: How does SG guidance aid the generation of high-quality images? We first consider assessing the overall semantic matchness between the input texts and the generated images. We make comparisons with baselines without integrating SGs. By observing the CLIP scores in Figure 6, it is clear that our model can achieve higher semantics faithfulness of image synthesis. Further, we try to probe the semantic structural alignment between the inputs and outputs. We mainly measure the 'object-attribute' correspondence between input texts and generated images via GLIP metric; and also assess if the structural triplets in initial textual SG can also be retrieved in the SG of the generated image (i.e., TriRec. metric). As shown in the
Figure 6: Evaluating the matches between input texts and generated images on COCO. For comparison, our 'Salad'* is downgraded by removing the SG imagination ability (i.e.,  $\mathrm{SGH} \rightarrow \mathrm{Init.SG}$ ).
figure, our Salad system consistently performs much better than the baselines in terms of these two metrics. This significantly suggests that with the SG guidance, the structural controllability of the T2I process is greatly enhanced, and thus leads to high-quality generation.
Q2: Does SGH indeed induce intricate and reasonable SGs? Firstly, in Figure 7 we explore whether the SGH can produce new SG structures during the T2I process, on the COCO and COCO-A2I datasets separately. As seen, after scene enrichment, the average numbers of all three element types (object, attribute, and relation) substantially increase. Notably, the addition is more evident on the COCO-A2I set, where scene enrichment is more needed.
Next, we examine if the newly imagined SG structures provide reasonable scenes to the input prompt. We reach this by measuring the recall rate (TriRec.) of the 'subject-predicate-object' pairs between two SGs, i.e., the predicted and the gold SGs. Assuming the SGs (denoted as  $G^{\mathrm{I}}$ ) of gold images entail reasonable scenes, we consider two types of SGs from our 'prediction': 1) SGs induced by SGH ( $\overline{G}$ ), and 2) SGs parsed from our synthesized images ( $G^{\mathrm{I}}$ ). We also compare with the text-enriched SPY method. As shown in Table 4, we first observe that the induced SGs highly align with the gold SGs, with 82.01 TriRec. score, indicating the
Figure 7: The average number of three types of SG nodes.
Table 4: Comparing the glod SGs  $(G^{I})$  with our induced SG  $(\overline{G})$ , and the SG  $(G^{\overline{I}})$  of generated image with TriRec. metric.
G vs. G1	G1 vs. G1
SPY	-	78.61
Salad	82.01	86.04
induced SGs are sensible for generating high-quality images. Moreover, by comparing the TriRec scores  $(G^{\mathrm{I}}$  vs.  $G^{\mathrm{I}})$  between SPY and Salad (i.e., 78.61 vs. 86.04), we learn that the synthesized images coordinated by imagined SGs more correspond to the gold images in terms of semantic scene structure. This also suggests that the SGH can induce valid SGs which favorably guide the image generation process.
Q3: Does sampling strategy helps diversified T2I? To measure the effectiveness of the sampling strategy in diversifying image generation given the same prompt, we consider both the qualitative analyses and human evaluation, where the former calculates the LPIPS score [60] to assess the perceptual similarity between two images in deep feature space. As shown in Figure 8, our Salad model significantly outperforms LDM and Frido in terms of diversity score and human evaluation, and things go worse upon removing the scene sampling
strategy. This demonstrates that our proposed scene sampling mechanism is effective in helping diversified T2I. In Appendix §C.2 we show more examples of diversified generations via the scene sampling mechanism.
LDM Frido Salad Salad w/o Sampling
Figure 8: Comparison of diversity using diversity score (LPIPS) and human evaluation.
T2I is a long-standing topic in computer vision and multimodal communities. Arrays of explorations have been devoted to achieving stronger image synthesis performances with various deep generative models, such as GANs [4, 17, 48, 52, 61], VAEs [29], flow-based approaches [11, 12] and ARMs [7, 13, 39, 49]. More recently, the diffusion denoising probabilistic methods (DDPMs) have revealed the greatest potentials on image synthesis, in which the optimal density estimation is more naturally achieved with a fixed diffusion process to transform an image into a Gaussian noise [9, 19, 23, 42, 47]. This work follows the line of diffusion methods and takes the SoTA latent diffusion model (LDM) [42, 14, 19] as the T2I backbone.
Generating high-quality images with complex scenes is the key criterion of a sound T2I system [14, 15, 26]. Many efforts have been paid for synthesizing more realistic and nature-looking images in sophisticated scenes, yet most of which are conditioned on taking the detailed descriptions as inputs [6, 51]. Thus, how to generate high-quality images in intricate scenes from succinct and abstract prompts becomes a meaningful yet challenging task. In this work, we introduce a scene hallucination mechanism, which, built upon the SG structure, performs more accurate and controllable scene completion and eventually helps generate intricate images of higher quality. We consider the SG representations [27] for the input prompt texts as well as the guidance of image synthesis. SG advances in intrinsically describing the semantic structures of scenes for texts or images [28, 53], enabling more fine-grained control of complex scenes [53], and thus aiding the final image generation.
Scene graph (SG) is a type of structured data that represents multiple objects and their complex relationships in the vision or language scenes, wherein the nodes denote objects & attributes and the edges depict relationships between objects [27]. As intrinsically describing the semantic structures of scene semantics for the given texts or images, SG has been widely utilized as a type of external feature being integrated into downstream applications for enhancements, e.g., image retrieval [27, 56], image generation [28, 53] and image captioning [54, 36, 55]. In this work, we consider the SG representations for the input sentences. Compared to the linear sequential nature of the text, SG offers a more intuitive manner to represent the scene semantics in a structured format, enabling more fine-grained control of complex scenes [53], and thus aiding the final image synthesis.
Our SGH mechanism also relates to the research of SG enrichment or imagination. While existing methods mostly approach the task by incrementally parsing SG elements [16, 57, 58], such greedy-increment paradigm may largely suffer from trapping in locally optimal SG generation, thus leading to inferior SG imagination. Instead, in this paper, we implement SGH as a discrete denoising&diffusion process. Discrete diffusion technique [25] is the latest introduced method that replaces the continuous state in standard diffusion models with a discrete one. During each denoising step, the entire SG structure is updated and optimized from a global viewpoint, so as to yield a more reasonable enrichment of scene structure. Besides, both the T2I and the SGH are modeled as the same diffusion process in our framework, where the two processes are well synchronized, such that the underlying SG features can perfectly guide the T2I synthesis at each step. To our knowledge, we are the first to investigate SG induction using discrete diffusion models.
In this work, we explore the text-to-image synthesis task under the abstract-to-intricate setup. Drawing inspiration from human intuition, we propose a scene-graph hallucination mechanism, which carries out scene imagination based on the initial scene graph of the input prompt, expanding the starting SG with more possible specific scene structures. We then develop an SG-based hallucination diffusion system for the abstract-to-intricate T2I, which mainly includes an SG-guided T2I module and an SGH module. Specifically, we design the SGH module based on the discrete diffusion technique, which evolves the initial SG structure by iteratively adding new scene elements. Then, we utilize another continuous diffusion model as the T2I synthesizer, where the overt image-generating process is navigated by the underlying semantic scene structure induced by the SGH module. On the standard COCO dataset, our system shows great superiority in the abstract-to-intricate T2I generation. Further analyses demonstrate that our SG-based hallucination mechanism is able to generate logically sound SG structures, which in return helps produce high-quality scene-riched images.
Benefits The current text-conditioned image generation approaches largely fail to the abstract-to-intricate T2I due to a lack of necessary details of input prompts. In this work, we propose a novel scene-graph hallucination mechanism inspired by human imagination intuition, which expands upon the initial scene graph from the text prompts to generate more feasible and specific scene structures. Furthermore, the enriched timestep-wised SG is leveraged to navigate the T2I generation process, leading to synthesizing more intricate images. Our study demonstrates that hallucinating images based on scene graph structures offer scalability, and modeling these structures enhances the generation of high-quality images with improved semantic controllability.
Potential weakness There can be two potential weaknesses that warrant consideration in our system. Firstly, the effectiveness of our system relies heavily on the quality of scene graph hallucination (SGH), yet the absence of a dedicated dataset for the SGH task poses a challenge in training the SGH module. However, we can leverage the richly annotated Visual Genome (VG) dataset, commonly used for training visual SG parsers, to provide initial training for the SGH module under an unconditional setting. Secondly, the training process of a diffusion model for text-to-image (T2I) generation entails substantial computational resources, resulting in increased energy consumption,  $\mathrm{CO}_{2}$  emissions, and potential environmental pollution.
This research is supported by NExT++ Lab and CCF-Baidu Open Fund.
A Extension of Technical Details
We in this part extend the specific details of our method techniques.
A.1 Discrete Diffusion Model for Scene Graph Hallucination
Here, we detail the forward and reverse processes in the discrete diffusion model [1] for SGH.
Forward Process In the forward process, we consider a node  $s_{t,j}^{*} \in G_{t}, * \in \{o, a, r\}$  ( $t$  is diffusion step,  $j$  is the node index) takes a scalar discrete value with  $K^{*}$  categories,  $s_{t,j}^{*} \in 1, \dots, K^{*}$ . Without introducing confusion, we omit subscripts  $j$  and superscripts  $*$  in the following description. We define the probabilities that  $s_{t-1}$  transits to  $s_{t}$  using the matrices  $[Q_{t}]_{ij} = q(s_{t} = j | s_{t-1} = i)$ , then we can write the forward Markov diffusion process:
$$
q \left(s _ {t} \mid s _ {t - 1}\right) = \mathcal {B} ^ {\top} \left(s _ {t}\right) \boldsymbol {Q} _ {t} \mathcal {B} \left(s _ {t - 1}\right), \tag {10}
$$
where  $\mathcal{B}(s_t)$  denotes the one-hot column vector which length is  $K$ . The categorical distribution over  $s_t$  is given by the vector  $Q_{t}\mathcal{B}(s_{t - 1})$ . Due to the property of the Markov chain, starting from  $s_0$ , we can derive the cumulative probability of  $s_t$  at arbitrary  $t$ -阶梯:
$$
q \left(s _ {t} \mid s _ {0}\right) = \mathcal {B} ^ {\top} \left(s _ {t}\right) \cdot \bar {\boldsymbol {Q}} _ {t} \cdot \mathcal {B} \left(s _ {0}\right), \text {w h e r e} \bar {\boldsymbol {Q}} _ {t} = \boldsymbol {Q} _ {t} \boldsymbol {Q} _ {t - 1} \dots \boldsymbol {Q} _ {1}. \tag {11}
$$
Besides, by conditioning on  $s_0$ , the posterior of this diffusion process is tractable:
$$
q \left(s _ {t - 1} \mid s _ {t}, s _ {0}\right) = \frac {q \left(s _ {t} \mid s _ {t - 1} , s _ {0}\right) q \left(s _ {t - 1} \mid s _ {0}\right)}{q \left(s _ {t} \mid s _ {0}\right)} = \frac {\left(\mathcal {B} ^ {\top} \left(s _ {t}\right) \boldsymbol {Q} _ {t} \mathcal {B} \left(s _ {t - 1}\right)\right) \left(\mathcal {B} ^ {\top} \left(s _ {t - 1}\right) \bar {\boldsymbol {Q}} _ {t} \boldsymbol {v} \left(s _ {0}\right)\right)}{\mathcal {B} ^ {\top} \left(s _ {t}\right) \bar {\boldsymbol {Q}} _ {t} \mathcal {B} \left(s _ {0}\right)}. \tag {12}
$$
Note that the transition matrix  $Q_{t}$  is capable of controlling the data corruption and denoising process, thus it should be carefully designed such that it is not too difficult for the reverse network to recover the signal from noises. We follow [1, 19] that exploits a mask-and-replace strategy to design a  $Q_{t}$ , which can be defined as:
$$
\boldsymbol {Q} _ {t} = \left[ \begin{array}{c c c c c} \alpha_ {t} + \beta_ {t} & \beta_ {t} & \beta_ {t} & \dots & 0 \\ \beta_ {t} & \alpha_ {t} + \beta_ {t} & \beta_ {t} & \dots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ \gamma_ {t} & \gamma_ {t} & \gamma_ {t} & \dots & 1 \end{array} \right], \tag {13}
$$
where for each node  $s_t$ , there are three probabilities, i.e., a probability of  $\gamma_t$  to transition to [MASK] node, a probability of  $K\beta_t$  to be resampled uniformly over all the  $K$  categories, and a probability of  $\alpha_t = 1 - K\beta_t - \gamma_t$  to stay the same node. Notedly, the [MASK] node never transitions to other states. The aforementioned discrete state-space models assume that all the nodes are switchable by corruption. However, it is unreasonable that an object node, 'bed', transitions to a relation node, 'in'. To avoid such invalid transition, we propose to apply disjoint corruption matrices  $Q_t^o, Q_t^a, Q_t^r$  for object, attribute, and relation nodes, respectively.
Reverse Process In the reverse process, an SG decoder is introduced as a denoising network to estimate the posterior distribution  $p_{\theta}(\cdot)$ , which takes the node token  $s_t$ , time step  $t$  and text prompt  $y$ . Specifically, each layer of the SG decoder contains two parts: 1) an adaptive normalization (AdaLN), and 2) a transformer block. AdaLN is applied to inject the timestep information:
$$
\boldsymbol {H} ^ {*} = \operatorname {A d a L N} \left(s _ {t}, t\right) = \boldsymbol {a} _ {t} \text {L a y e r N o r m} \left(\operatorname {C L I P} \left(s _ {t}\right)\right) + \boldsymbol {b} _ {t}, \tag {14}
$$
where  $\mathrm{CLIP}(\cdot)$  denotes an encoding layer for projecting the node token, and  $a_{t}$  and  $b_{t}$  are obtained from a linear projection of the timestep embedding. Each transformer block contains 1) a full self-attention, i.e.,  $\pmb{H}^{*} = \mathrm{Softmax}(\frac{\pmb{H}^{*}\cdot\pmb{H}^{*^{\top}}}{\sqrt{d_1}})\cdot \pmb{H}^{*}$ . 2) a text-cross-attention (Text-CA) integrates the input prompt  $y$ :
$$
\boldsymbol {H} ^ {*} = \operatorname {T e x t - C A} (y, \boldsymbol {H} ^ {*}) = \operatorname {S o f t m a x} \left(\frac {\boldsymbol {H} ^ {*} \cdot \boldsymbol {H} ^ {y ^ {\top}}}{\sqrt {d _ {1}}}\right) \cdot \boldsymbol {H} ^ {y}, \tag {15}
$$
where  $H^{y}$  is a conditional feature sequence yielded by a text encoder first takes the text prompts  $y$  as input. Then, 3) a graph cross-attention (Graph-CA) is devised to take the induced SG  $(G_{t + 1})$  in the previous  $t + 1$  timestep:
$$
\tilde {\boldsymbol {H}} ^ {*} = \operatorname {G r a p h - C A} \left(G _ {t + 1}, \boldsymbol {H} ^ {*}\right) = \operatorname {S o f t m a x} \left(\frac {\boldsymbol {H} ^ {*} \cdot \boldsymbol {H} ^ {G ^ {\top}}}{\sqrt {d _ {1}}} \cdot \boldsymbol {E} ^ {G}\right) \cdot \boldsymbol {H} ^ {G}, \tag {16}
$$