SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shengqiong Wu $^{1}$  Hao Fei $^{1*}$  Hanwang Zhang $^{2}$  Tat-Seng Chua $^{1}$
$^{1}$ NExT++, School of Computing, National University of Singapore
$^{2}$  School of Computer Science and Engineering, Nanyang Technological University
swu@u.nus.edu {haofei37, dcscts}@nus.edu.sg hanwangzhang@ntu.edu.sg
In this work, we investigate the task of text-to-image (T2I) synthesis under the abstract-to-intricate setting, i.e., generating intricate visual content from simple abstract text prompts. Inspired by human imagination intuition, we propose a novel scene-graph hallucination (SGH) mechanism for effective abstract-to-intricate T2I synthesis. SGH carries out scene hallucination by expanding the initial scene graph (SG) of the input prompt with more feasible specific scene structures, in which the structured semantic representation of SG ensures high controllability of the intrinsic scene imagination. To approach the T2I synthesis, we deliberately build an SG-based hallucination diffusion system. First, we implement the SGH module based on the discrete diffusion technique, which evolves the SG structure by iteratively adding new scene elements. Then, we utilize another continuous-state diffusion model as the T2I synthesizer, where the overt image-generating process is navigated by the underlying semantic scene structure induced from the SGH module. On the benchmark COCO dataset, our system outperforms the existing best-performing T2I model by a significant margin, especially improving on the abstract-to-intricate T2I generation. Further in-depth analyses reveal how our methods advance.
The task of generating images from natural language descriptions, known as text-to-image (T2I) synthesis, has attracted significant attention [4, 17, 52]. To approach T2I, various generative models have been explored, including generative adversarial networks (GANs) [48, 52, 61], variational autoencoders (VAEs) [50], flow-based models [3], and auto-regressive models (ARMs) [10, 41], all of which aim to generate realistic images in high quality and high faithfulness. Most recently, diffusion-based models have been proposed, which simulate the physical process of gas diffusion for image generation [23]. Diffusion models have shown unprecedented performance in image synthesis over existing methods, becoming the current state-of-the-art (SoTA) T2I solution [2, 9, 19, 47, 42].
As a long-reached viewpoint [28, 53, 37], sound T2I systems should not only achieve high-quality image generation in simple straightforward visual scenery but be more capable of synthesizing realistic images with complex scenes. Typically, detailed textual descriptions are necessarily needed to prompt the synthesis process with adequate details for high-quality vision generation. However, in a realistic world, it could also be ubiquitous to produce intricate visions without relying on lengthy elaborate prompts. For example, users may prefer T2I systems to synthesize well-detailed images while not taking too much time to write descriptions in detail. More crucially, due to the natural modality asymmetry between language and vision, even some simple words can intrinsically describe or represent abstract visual scenes with rich and complex details. Whenever mentioning the words
*Hao Fei is the corresponding author.
$^{2}$ Code is available at https://github.com/ChocoWu/T2I-Salad
$\triangleright$  Original (abstract) Prompt: Two young men give a presentation in the office.
Enriched Prompt-I:
Enriched Prompt-II:
(a)
Figure 1: (a) An example of abstract-to-intricate T2I synthesis. All images are generated by the Latent diffusion Model (LDM) [42]. LDM fails to accurately render the abstract contexts, e.g., 'give a presentation' and 'office' of the original prompt. Raw prompts can be enriched via descriptive insertion [8], or addition [48]. Enriched contexts are in blue. (b) We illustrate the human intuition on the abstract-to-intricate T2I process: we always first grasp the semantic structure of the original prompt text, i.e., scene graph (SG), and then carry out imagination with more complete scenes based on the SG. Here the glowing nodes and edges are enriched ones.
with specific scenes, such as classroom, kitchen, office, or actional verbs e.g., traveling, shopping, there is always a picture with multifaceted scenes and rich-detailed backgrounds. In a word, it is worth investigating generating intricate images from succinct abstract prompts.
Yet existing prevailing approaches, even the SoTA diffusion models, may largely fail the abstract-to-truncate T2I, due to the lack of necessary details of input prompts (cf. Figure 1(a)). One intuitive workaround is to directly enrich the texts, i.e., adding more details for the prompts. Specifically, existing works either consider inserting additional adjectives and attributives to modify the original mentions and scenes [8], or concatenating raw sentences with more tangible explanations and contexts that are elicited from external large language models [48], e.g., ChatGPT [38]. Unfortunately, due to the intrinsic grammar and linguistics rules, such text-based prompt enrichment can be subject to the issue of lower controllability. One problem is the visual distraction, where the main focus of the resulting image is dominated by other newly-added trivial contents, when aggressively inserting intermediate descriptive components into the raw texts, as exemplified in Figure 1(a). Besides, directly appending new textual descriptions would increase the prompt length and then lead to incorrect binding of attributes or relations, i.e., making the image deviate from the original user intention.
As a reference, intuitively, we human beings would tackle the abstract-to-intricate T2I as a two-step painting process, i.e., from semantic interpretation to scene imagination. In the semantic interpretation, a painter always first translates the succinct textual prompt into a structured skeleton that represents the semantic scene of key mention objects and their relations. Then, based on the initial scene, the painter mentally completes the abstract scene with more concrete and valid details. With the enriched scene structure, the final vision can be more accurately and easily rendered. Motivated by such human intuition, in this work, we propose a scene-graph hallucination (SGH) method for achieving effective abstract-to-intricate T2I. As illustrated in Figure 1(b), we first investigate representing the input prompt with its scene graph (SG) [46]. SG advances in depicting the intrinsic semantics of texts (or vision) with structured representations, e.g., objects, attributes, and relationships, enabling fine-grained control of the semantic scene [53]. Based on the SG of input text we then carry out scene hallucination, expanding the initial SG with more possible specific scene structures. Also with the SG representation, the imagination process can be much more accurate and controllable.
To implement the overall idea, we develop an SG-based hallucination diffusion system (namely, Salad) for high-quality image synthesis. Salad is a fully diffusion-based T2I system, which mainly consists of a scene-driven T2I module and an SGH module. As shown in Figure 2, we first take advantage of the SoTA latent diffusion model [9] as our backbone T2I synthesizer, in which the overt image-generating process is controlled and navigated by the underlying semantic scene structure.
Figure 2: Overall framework of our proposed SG-based hallucination diffusion system (Salad).
On the other hand, we design the SGH module based on the modality-agnostic discrete diffusion model [25], which evolves and completes the initial SG structure of the input prompt by iteratively generating new scene elements (i.e., SG nodes). During the SG imagination process, the resulting structural representation at each step is fused into the T2I synthesizer via a hierarchical integration strategy. Further, we devise a scene sampling mechanism, via which the SGH module can generate various SG imaginations, and thus help achieve diversified image synthesis during inference.
We conduct experiments on COCO [33], the widely-used T2I dataset. Results show that Salad outperforms all the existing strong-performing T2I systems with significant margins. Further analysis reveals that modeling the SG structures helps synthesize high-quality images with stronger semantic controllability. Our proposed SGH mechanism is effective in inducing sound SG structures, helping produce more realistic images from short abstract text prompts. And the scene sampling strategy aids diversified T2I generation. In summary, this paper contributes in five aspects:
- We are the first to study the novel setup of intricate image synthesis from abstract texts.
- We solve the abstract-to-intricate T2I with a novel SG hallucination mechanism, which is implemented via discrete diffusion technique, performing scene enrichment with reasonable imagination.
- We propose a diffusion-based model with a hierarchical scene integration strategy for highly controllable and scalable image generation.
- We devise a scene sampling mechanism to generate various scene graphs for diversified image syntheses during inference.
- Our framework achieves new SoTA results in the abstract-to-intricate T2I generation.
2.1 Scene Graph Representation
The SG (denoted as  $G$ ) [45] represents the semantic relationships among scene objects in a structure, where there are three types of nodes, i.e., object, attribute, and relation, cf. Figure 1(b). We formulate the object node set as  $\{o_1, \dots, o_N\}$ , where  $o_n$  denotes  $n$ -th object node; the attribute node set as  $\{a_{1,1}, \dots, a_{N,M}\}$ , where  $a_{n,m}$  means  $m$ -th attribute node of the  $n$ -th object node; the relation node set as  $\{r_{1,1}, \dots, r_{N,N}\}$ , where  $r_{i,j}$  means object node  $o_i$  connects to the object  $o_j$ . All nodes come with a category label  $l^o / a / r$ , and each type of node has its own unique category vocabulary. For example as in Figure 2 right bottom, the object node  $o_1$  with category label  $l_1^o$  associated with two attribute nodes  $a_{1,1}$  and  $a_{1,M}$ , with category label  $l_1^a$  and  $l_{23}^a$ , respectively. And the object node  $o_1$  connects to the object node  $o_2$  through an edge  $r_{1,2}$  with the category label  $l_7^r$ .
Diffusion models (DMs) [23] learn to convert a simple Gaussian distribution into a data distribution. Technically, DMs consist of a forward (diffusion) process and a reverse (denoising) process. In the forward process, the given data  $\pmb{x}_0 \sim q(\pmb{x}_0)$  is gradually corrupted into an approximately standard normal distribution  $\pmb{x}_T \sim p(\pmb{x}_T)$  over  $T$  steps by increasingly adding noisy, formulated as
$q(\pmb{x}_{1:T}|\pmb{x}_0) = \prod_{t=1}^T q(\pmb{x}_t|\pmb{x}_{t-1})$ . The learned reverse process  $p_\theta(\pmb{x}_{0:T}) = p(\pmb{x}_T)\prod_{t=1}^Tp_\theta(\pmb{x}_{t-1}|\pmb{x}_t)$  gradually reduces the noise towards the data distribution. To improve the fit of a generative model to the data distribution, a variational upper bound on the negative log-likelihood is optimized:
$$
\mathcal {L} _ {v l b} = \mathbb {E} _ {q \left(\boldsymbol {x} _ {1: T} \mid \boldsymbol {x} _ {0}\right)} \left[ \log \frac {q \left(\boldsymbol {x} _ {T} \mid \boldsymbol {x} _ {0}\right)}{p _ {\theta} \left(\boldsymbol {x} _ {T}\right)} + \sum_ {t = 2} ^ {T} \log \frac {q \left(\boldsymbol {x} _ {t - 1} \mid \boldsymbol {x} _ {t} , \boldsymbol {x} _ {0}\right)}{p _ {\theta} \left(\boldsymbol {x} _ {t - 1} \mid \boldsymbol {x} _ {t}\right)} - \log p _ {\theta} \left(\boldsymbol {x} _ {0} \mid \boldsymbol {x} _ {1}\right) \right], \tag {1}
$$
where  $p_{\theta}(\cdot)$  is estimated by a denoising network, which can be a time-conditional U-Net [43]. Recently, latent diffusion models (LDMs) [42, 14, 19] are introduced to adopt DMs to operate in an efficient, low-dimensional latent space, where a pre-trained encoder  $\mathcal{E}$  maps the given data  $x_0$  into a latent code  $z_0 = \mathcal{E}(x_0)$ , and a decoder reconstructs the final output image from the denoised latent  $\mathcal{D}(z_0) \sim x_0$ . Due to the higher computation sufficiency, this work thus takes the LDM as T2I backbone. Besides, DMs also have been extended to operate in discrete state spaces [1], i.e., performing diffusion and denoising processes over discrete variables, which have demonstrated competitive performances for discrete data, such as text [24] and layout [5]. Hence, we also adopt the discrete DMs to realize the SG induction process. Appendix §A.1 gives more technical details to the discrete diffusion models.
Formally, T2I aims to generate an image  $x$  that faithfully reflects the desired content in the input prompt text  $y$ . To approach abstract-to-intricate T2I, we propose an SG-based hallucination diffusion system (Salad), which is shown in Figure 2. The salad consists of two major modules. First, the SGH (cf. §3.1) is responsible for enriching the initial SG of the text prompt via a discrete diffusion model. Then, built upon an LDM (cf. §2.2), the Scene-driven Image Synthesis module (SIS, cf. §3.2) performs denoising for image synthesis, during which the derived SG features is fused via a hierarchical scene integration mechanism. The underlying SGH closely collaborates with the upper SIS at each step, and thus the semantic scene skeleton takes fine-grained control of the overt vision rendering. We also describe the optimization (cf. §3.3), and the scene sampling strategy (cf. §3.4).
3.1 Scene Graph Hallucination (SGH)
As aforementioned, we formulate the SGH as a discrete denoising diffusion process [25] (cf. Fig. 2). Specifically, in the forward process, the SG of the gold image, marked as  $G_{0}$ , will be corrupted into a sequence of increasingly noisy latent variables  $G_{1:T} = \{G_1,G_2,\dots ,G_T\}$ , where each SG node  $s_{t,j}^{*}\in G_{t},* \in \{o,a,r\}$  ( $t$  is diffusion step,  $j$  is the node index) takes a discrete value with  $K^{*}$  category labels, and  $o,a,r$  denotes the nodes' type, i.e., object ( $o$ ), attribute ( $a$ ), and relation ( $r$ ). For simplicity, we omit subscripts  $j$  and superscripts * in the following description. The discrete diffusion process can be parameterized with a multinomial categorical transition matrix:
$$
q \left(s _ {t} \mid s _ {t - 1}\right) = \mathcal {B} ^ {\top} \left(s _ {t}\right) \cdot \boldsymbol {Q} _ {t} \cdot \mathcal {B} \left(s _ {t - 1}\right), \tag {2}
$$
where  $\mathcal{B}(s_t)$  denotes the column one-hot vector of  $s_t$ . And  $\pmb{Q}_t$  is the transition matrix, with  $[\pmb{Q}_t]_{ij} = q(s_t = j|s_{t - 1} = i)$  representing the probabilities that  $s_{t - 1}$  transitions to  $s_t$ . Due to the property of the Markov chain, the cumulative probability of  $s_t$  at arbitrary timestep from  $s_0$  can be derived as  $q(s_{t}|s_{0}) = \mathcal{B}^{\top}(s_{t})\cdot \bar{\boldsymbol{Q}}_{t}\cdot \mathcal{B}(s_{0})$ , where  $\bar{\boldsymbol{Q}}_t = \boldsymbol {Q}_t\boldsymbol {Q}_{t - 1}\dots \boldsymbol {Q}_1$ . Inspired by [1, 19], we employ a mask-and-replace strategy to design the  $Q_{t}$ . For each node  $s_t$ , we define three probabilities: 1) a probability of  $\gamma_{t}$  to transition to [MASK] node, 2) a probability of  $K\beta_{t}$  be resampled uniformly over all the  $K$  categories, and 3) a probability of  $\alpha_{t} = 1 - K\beta_{t} - \gamma_{t}$  to stay the same node. Notedly, the [MASK] node never transits to other states. Hence, the transition matrix  $\pmb{Q}_t$  can be formulated as<sup>3</sup>:
$$
\boldsymbol {Q} _ {t} = \left[ \begin{array}{c c c c c} \alpha_ {t} + \beta_ {t} & \beta_ {t} & \beta_ {t} & \dots & 0 \\ \beta_ {t} & \alpha_ {t} + \beta_ {t} & \beta_ {t} & \dots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ \gamma_ {t} & \gamma_ {t} & \gamma_ {t} & \dots & 1 \end{array} \right]. \tag {3}
$$
The aforementioned discrete state-space models assume that all the standard nodes are switchable by corruption. However, as stated in §2.1, there are three different SG nodes under separate categories. Hence, we apply three disjoint corruption matrices  $Q_{t}^{o} \in \mathbb{R}^{K^{o} \times K^{o}}$ ,  $Q_{t}^{a} \in \mathbb{R}^{K^{a} \times K^{a}}$ ,  $Q_{t}^{r} \in \mathbb{R}^{K^{r} \times K^{r}}$  for object, attribute, and relation nodes, respectively, where  $K^{o}, K^{a}, K^{r}$  denotes the size of category labels of three node types respectively.
$^{3}$ Appendix §A.1 provides detailed formulation of the discrete diffusion process.
In the denoising process, we introduce an SG decoder as the neural approximator to estimate the distribution  $p_{\theta}(s_{t - 1}|s_t,y)$ . As shown in Figure 2, SG decoder first employs an adaptive normalization (AdaLN) to inject the timestep information. A text cross-attention (Text-CA) integrates the input prompt  $y$ . Then, a graph cross-attention (Graph-CA) is devised to take in the induced SG  $(G_{t + 1})$  in the previous  $t + 1$  timestep:
$$
\tilde {\boldsymbol {H}} ^ {*} = \operatorname {G r a p h - C A} \left(G _ {t + 1}, \boldsymbol {H} ^ {*}\right), \tag {4}
$$
where  $H^{*}$  are the features yielded from Text-CA. Graph-CA consistently consults the overall picture of the last SG for a more coherent generation when making the current decision.
Intuitively, among the object, attribute, and relation nodes, the object nodes always come first to determine the scene subjects, followed by their modifier attributes, and then the relations between objects. Thus, instead of parallel induction of three node types, we follow this SG node-type dependence (NTD) intuition, and design an NTD cross-attention (NTD-CA) for the  $*$ -type node induction ( $*$  can be object or attribute):
$$
\hat {\boldsymbol {H}} ^ {*} = \left\{ \begin{array}{l l} \text {N T D - C A} \left(s _ {t} ^ {o}, \tilde {\boldsymbol {H}} ^ {*}\right), & * = a \\ \text {N T D - C A} \left(s _ {t} ^ {o} \oplus s _ {t} ^ {a}, \tilde {\boldsymbol {H}} ^ {*}\right). & * = r \end{array} \right. \tag {5}
$$
Note that we stack multiple layers of the above calculations as one SG decoder. For each state of node types  $\hat{s}_t^*$ , a softmax function is put on to obtain the category label distributions:  $\hat{s}_t^* = \mathrm{Softmax}(\hat{H}^*)$ .
Following [1], we optimize the SG decoder by minimizing the variational lower bound  $\mathcal{L}_{vlb}$  (Eq. 1). Also the parameterization trick [19] is leveraged to encourage the system to predict the noiseless node distribution  $p_{\theta}(\tilde{s}_0|s_t,y)$  at each reverse step, which is taken as an auxiliary learning objective to be incorporated with  $\mathcal{L}_{vlb}$ :
$$
\mathcal {L} _ {S G H} = \mathcal {L} _ {v l b} + \lambda_ {1} \log p _ {\theta} (s _ {0} | s _ {t}, y),
$$
$$
\mathcal {L} _ {v l b} = \mathcal {L} _ {0} + \mathcal {L} _ {1} + \dots + \mathcal {L} _ {T - 1} + \mathcal {L} _ {T},
$$
$$
\mathcal {L} _ {0} = - \log p _ {\theta} \left(s _ {0} \mid s _ {1}, y\right), \tag {6}
$$
$$
\mathcal {L} _ {t - 1} = D _ {K L} \left(\left(q \left(s _ {t - 1} \mid s _ {t}, s _ {0}\right)\right) \mid \mid \left(p _ {\theta} \left(s _ {t - 1} \mid s _ {t}, y\right)\right), \right.
$$
$$
\mathcal {L} _ {T} ^ {o} = D _ {K L} \left(q _ {(s T} | s _ {0}) | | p _ {\theta} (s _ {T})\right),
$$
where  $\lambda_{1}$  is a hyper-parameter for controlling the learning components.
3.2 Scene-driven Image Synthesis (SIS)
With the enriched SG from SGH in each denoising step at hand, the backbone T2I diffusion carries out the image synthesis with the guidance of that SG. We design a hierarchical scene integration (HSI) strategy to ensure the highly effective integration of SG features. Specifically, we consider the fusion at four different hierarchical levels, i.e., object  $(L^{o})$ , relation  $(L^{r})$ , region  $(L^{c})$ , and global levels  $(L^{g})$  with each focusing on different context scopes, as illustrated in Figure 3. We maintain the representations of these three levels as the keys  $K_{L^i}$  & values  $V_{L^i}$  via CLIP encoder [40], which are then integrated together via the Transformer attention of U-Net in LDM:
Figure 3: Hierarchical scene integration (HSI) fuses the SG features under multiple levels: 1) objects (with attributes), 2) relational triplets (i.e., subject-predicate-object), 3) regional neighbors, and 4) the whole SG.
$$
\bar {\boldsymbol {H}} = \sum_ {i \in \{o, r, c, g \}} \operatorname {A t t n} \left(\boldsymbol {H}, \boldsymbol {K} _ {L ^ {i}}, \boldsymbol {V} _ {L ^ {i}}\right) = \sum_ {i \in \{o, r, c, g \}} \operatorname {S o f t m a x} \left(\frac {\boldsymbol {H} \boldsymbol {K} _ {L ^ {i}} ^ {\top}}{\sqrt {d}}\right) \boldsymbol {V} _ {L ^ {i}}, \tag {7}
$$
where  $H$  is the visual query vectors from the ResNet block in LDM. The above hierarchical scene integration is carried out for both the downsampling and upsampling processes in U-Net. By denoising  $T$  steps, the system finally produces the desired image. Appendix §A.2 gives more details of this part.
3.3 Warm-start Training
To ensure stable learning of the overall system, we take a warm-start training strategy. Firstly, the SGH is separately updated via  $\mathcal{L}_{SGH}$  (Eq. 6) based on the abstract-to-intricate SG pair annotations,
until it has converged. Then, both the SIS and SGH modules are optimized jointly by minimizing:
$$
\mathcal {L} = \lambda_ {2} \mathcal {L} _ {S G H} + \mathcal {L} _ {S I S}, \quad \text {w h e r e} \mathcal {L} _ {S I S} = \mathbb {E} _ {\boldsymbol {z} \sim \mathcal {E} (\boldsymbol {x} _ {0}), \epsilon \sim \mathcal {N} (0, I), t} [ \| \epsilon - \epsilon_ {\theta} (\boldsymbol {z} _ {t}, G _ {t}, y, t) \| _ {2} ^ {2} ]. \tag {8}
$$
Here we follow [23] to optimize SIS with a simple surrogate objective that calculates the mean-squared error loss, and  $G_{t}$  is the intermediate SG by SGH at timestep  $t$ , which can be derived from the  $s_t^*$ ,  $* \in \{o, a, r\}$  (cf. Figure 2).  $\epsilon$  is the noise in SIS, and  $\epsilon_{\theta}(\cdot)$  denotes the U-Net (cf. Figure 3).
3.4 Inference with Scene Sampling
During inference, we further aim to endow the SGH with diversified SG enrichment, and thus lead to T2I diversification. Intuitively, given an abstract prompt, there is often more than one possibility of the potential scenes to imagine. Also, it can be observed that in the de
Figure 4: Illustration of the scene sampling mechanism.
noising process, the diffusion model has a larger potential of divergence only at its earlier stage, while the generation tends to be more stable and certain when the iteration grows. Correspondingly, we expect the scene sampling to start in the early denoising steps, and gradually be more determining. Thus, we design a scene-sampling mechanism. First, instead of picking the best prediction of the category of any node  $\hat{s}_{t,j}^*$ , we take the top- $A$  category candidates with corresponding probability distribution  $\Psi$  based on the category distribution  $\hat{s}_{t,j}^* \in \mathbb{R}^{K^*}$  of node  $s_{t,j}^*$ . Then, we perform sampling over these candidates with a dynamic probability:
$$
\rho_ {t, j} ^ {*} = e ^ {- \eta \cdot t} \cdot \Psi + (1 - e ^ {- \eta \cdot t}) \frac {1}{A}, \tag {9}
$$
where  $* \in \{o, a, r\}$ , and  $\eta$  is a temperature. It is an annealing process, i.e., when  $t = T$  (starting denoising) more random sampling is preferable, while  $t$  approaches 0 (denoising ends), SGH tends to be more decisive. Figure 4 exemplifies the mechanism with the object type of nodes  $(\hat{s}_{t,n}^o)$ .