SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 9: Different semantic levels in the SG: objects, relation, region, and global levels.
where  $H^{G}$  and  $E^{G}$  is the embedding representation of the nodes and edges in  $G_{t + 1}$ , respectively, Next, 4) a node-type dependency cross-attention (NTD-CA) is designed for the *-type node induction (* can be object or attribute):
$$
\hat {\boldsymbol {H}} ^ {*} = \left\{ \begin{array}{l l} \text {N T D - C A} \left(s _ {t} ^ {o}, \tilde {\boldsymbol {H}} ^ {*}\right) = \operatorname {S o f t m a x} \left(\frac {\tilde {\boldsymbol {H}} ^ {*} \cdot \left(\operatorname {C L I P} \left(s _ {t} ^ {o}\right) ^ {\top}\right)}{\sqrt {d _ {1}}}\right) \cdot \operatorname {C L I P} \left(s _ {t} ^ {o}\right), & * = a \\ \operatorname {N T D - C A} \left(s _ {t} ^ {o} \oplus s _ {t} ^ {a}, \tilde {\boldsymbol {H}} ^ {*}\right) = \operatorname {S o f t m a x} \left(\frac {\tilde {\boldsymbol {H}} ^ {*} \cdot \left(\operatorname {C L I P} \left(s _ {t} ^ {o}\right) \oplus \operatorname {C L I P} \left(s _ {t} ^ {a}\right)\right) ^ {\top}}{\sqrt {d _ {1}}}\right) \cdot \left(\operatorname {C L I P} \left(s _ {t} ^ {o}\right) \oplus \operatorname {C L I P} \left(s _ {t} ^ {a}\right)\right). & * = r \end{array} \right. \tag {17}
$$
Finally, a softmax function is put on to obtain the category label distributions:
$$
\hat {\boldsymbol {s}} _ {t} ^ {*} = \operatorname {S o f t m a x} \left(\hat {\boldsymbol {H}} ^ {*}\right). \tag {18}
$$
Following [1, 19], the SG decoder is optimized by minimizing the variational lower bound (VLB):
$$
\mathcal {L} _ {v l b} = \mathcal {L} _ {0} + \mathcal {L} _ {1} + \dots + \mathcal {L} _ {T - 1} + \mathcal {L} _ {T}, \tag {19}
$$
$$
\mathcal {L} _ {0} = - \log p _ {\theta} \left(s _ {0} \mid s _ {1}, y\right), \tag {20}
$$
$$
\mathcal {L} _ {t - 1} = D _ {K L} \left(\left(q \left(s _ {t - 1} \mid s _ {t}, s _ {0}\right)\right) \mid \mid \left(p _ {\theta} \left(s _ {t - 1} \mid s _ {t}, y\right)\right), \right. \tag {21}
$$
$$
\mathcal {L} _ {T} ^ {o} = D _ {K L} \left(q \left(s _ {T} \mid s _ {0}\right) \| p _ {\theta} \left(s _ {T}\right)\right), \tag {22}
$$
where  $p_{\theta}(s_T)$  is the stationary distribution at timestep  $T$ . To estimate the  $p_{\theta}(s_T)$ , we extend the forward process by appending a rank-one matrix  $Q_{t+1}$  that ignores  $s_T$  and produces a deterministic  $s_{T+1} \in G_{T+1}$  where  $G_{T+1}$  is the initial SG parsed from the text prompt  $y$ , then learning the reverse step  $p_{\theta}(s_T | s_{T+1}) = p_{\theta}(s_T)$ . Furthermore, we follow the reparameterization trick proposed in [19], which lets the network predict the noiseless token distribution  $p_{\theta}(\tilde{s}_0 | s_t, y)$  at each reverse step so as to gain better quality. Based on the reparameterization trick, an auxiliary objective is introduced:
$$
\mathcal {L} _ {s _ {0}} = - \log p _ {\theta} \left(s _ {0} \mid s _ {t}, y\right), \tag {23}
$$
To combine this loss with  $\mathcal{L}_{vlb}$ , the final training objects are defined as follows:
$$
\mathcal {L} _ {S G H} = \mathcal {L} _ {v l b} + \lambda_ {1} \mathcal {L} _ {s _ {0}}, \tag {24}
$$
where  $\lambda_{1}$  is a hyper-parameter to control the effect of the auxiliary loss  $\mathcal{L}_{s_0}$ .
A.2 Scene-driven Image Synthesis
As described in §3.2, we propose a hierarchical scene integration strategy to effectively integrate the SG features. Here, we give more details on this part. Firstly, as shown in Figure 9, the SG is able to represent four different hierarchical levels of semantics corresponding to the image:
- Object level. In the process of image synthesis, a crucial aspect lies in accurately generating each specified individual object, corresponding to the object node and associated attribute notes in SG, for example, black T-shirt, old building, young man, luggage bag, sunglass.
- Relation level. A high-quality image is not only the generation of objects but also their intricate relationships which connect two objects, akin to the subject-predicate-object triplets found in the SG, such as man wear T-shirt, man in front of building, man carrying on luggage bad, man wear sunglass.
- Region level. Regional image generation focuses on multiple objects and entangled relationships among them, which aligns with the presence of overlapping relation triplets in SG. Here we adopt two overlapped relation triplets as the region representation of the image, such as man wear T-shirt in front of building, man wear T-shirt carrying on luggage bag, man wear
Two overlapped relation triplets means there are the same subject or object in the two relation triplets.
Thirt wear sungalss, man in front of building carrying on luggage bag, man in front of building wear sunglass, man carrying on luggage bag wear sunglass.
- Global level. The whole SG provides global semantics to guide the generation of images.
Then, we technically extract a collection of concepts from the four semantics levels in the SG, denoted as  $L^o$  (object level),  $L^r$  (relation level),  $L^c$  (region level), and  $L^g$  (global level). We encode each concept separately:
$$
\boldsymbol {U} _ {L ^ {i}} = \left\{\boldsymbol {u} _ {1}, \boldsymbol {u} _ {2}, \dots \right\}, \quad \boldsymbol {u} _ {j} = \operatorname {C L I P} \left(c _ {j}\right), \tag {25}
$$
where  $c_{j}\in L^{i};j = 1,2,\dots ,|L^{i}|;i\in \{o,r,c,g\}$
We further maintain the representation of these four levels as the keys  $K_{L^i}$  and values  $V_{L^i}$  by linear transformations:
$$
\boldsymbol {K} _ {L ^ {i}} = \operatorname {L i n e a r} \left(\boldsymbol {U} _ {L ^ {i}}\right); \boldsymbol {V} _ {L ^ {i}} = \operatorname {L i n e a r} \left(\boldsymbol {U} _ {L ^ {i}}\right), \tag {26}
$$
where  $i \in \{o, r, c, g\}$ . Next, we integrate these features via the Transformer attention of U-Net in LDM, which is formulated in Eq. (7).
B Detailed Experiment Settings
COCO The training and validation data numbers in COCO are 83K and 41k, respectively. We note that, in the evaluation phase, models are evaluated on the full COCO 2014 validation set.
Visual Genome (VG) Visual Genome [30] version 1.4 (VG) comprises 108,077 images annotated with scene graphs. Following previous work [28], we use object and relationship categories occurring at least 2,000 and 500 times respectively in the dataset, leaving 178 objects and 45 relationship types, and we ignore small objects, and use images with between 5 and 30 objects and at least three relationships, this leaves us with 62,565 images with an average of 10 objects and 5 relationships per image.
Construction of COCO-A2I We elaborate on the process of constructing the COCO-A2I dataset in the following three steps:
- First, we consider Place Nouns and Progressive Verbs are two types of abstract words that can depict intricate images. Therefore, we pre-define the candidate list of Place Nouns: street, sidewalk, kitchen, restroom, bathroom, living room, bedroom, hostel, house, office, bank, and the candidate list of Progressive verbs: traveling, knitting, gardening, shopping, presenting, drawing, baking, studying, etc.
- Second, we select the COCO valid dataset in that captions of instances contain the words in the candidate list, obtaining the primary-filtering dataset.
- Third, we filter out the primary-filtering dataset in which the number of words in the captions of instances is more than 10 and the number of bounding boxes is less than 6, obtaining the final target COCO-A2I dataset.
After the three-step pipeline, we obtain 2,005 text-image pairs.
In Figure 10, we show some examples, where the images tend to contain intricate content, including multiple objects, attributes, and relationships while the corresponding text prompts are relatively short and abstract.
B.2 Evaluation Metric Implication
We employ Fréchet Inception Distance (FID) [21], Inception score (IS) [44], CLIP score [20], and GLIP [31] used in [14] to quantitatively evaluate the quality of the generated images, and Learned Perceptual Image Patch Similarity (LPIPS) [60] utilized in [42] to quantify the diversity of the generated images. Additionally, we introduce Triplet Recall (TriRec.) to measure the percentage of the correct relation triplet among all the relations. Technically, given a set of ground truth triplets (subject-predicate-object), denoted  $GT$ , and the TriRec. is computed as  $\mathrm{TriRec.} = |PT \cap GT| / |GT|$ , where  $PT$  are the relation triplets extracted from the generated images by a visual SG parser.
A man sits in a diner photographing his meal.
Figure 10: The examples from COCO-A2I dataset for evaluating abstract-to-intricate T2I generation.
B.3 Human Evaluation Criterion.
In §4.5, we conduct a human evaluation, i.e., user study, for more intuitive assessments of the diversification quality of T2I models. Specifically, we generate 10 images conditioned on a single text prompt and ask evaluators to cluster these images based on their contextual information, such as objects, attributes, and relationships present in the generated images. After evaluating 100 distinct text prompts, we compute the average clustering ratio:
$$
\text {A v e r a g e C l u s t e r i n g R a t i o} = \text {A v e r a g e} \left(\frac {\mid \text {C l u s t e r} \mid}{1 0}, \dots\right), \tag {27}
$$
where  $|Cluster|$  denotes the number of clusters for 10 images generated by a text prompt. Average(·) is an average function. Here, a higher ratio score signifies greater diversification in image generation.
C.1 Visualizations of SGH Process
In Figure 11, we visualize the SGH process with two examples, where we sample several time steps and plot the generated SGs and the corresponding images. As can be seen, our SGH is capable of imaging certain reasonable structures of SGs to enrich the initial SG.
C.2 More Cases
Here we showcase more examples of the generated images via the scene sampling mechanism in Figure 12 and 13. We synthesize 10 images for each given text prompt.
Text Prompt (a): A fancy chandelier hanging in a small kitchen.
Text Prompt (b): A small room with painting stuffs.
Figure 11: Visualization of T2I generation process at  $t = 100, 90, \dots, 0$ , along with the enriched SG at  $t = 100, 70, 50, 20$ .
Text Prompt: Sparse narrow kitchen.
Text Prompt: A person is fishing.
Text Prompt: A father and son play Wii in the living room.
Text Prompt: A bed in a room.
Figure 12: More samples of abstract-to-intricate T2I generation via scene sampling mechanism.
Text Prompt: A person is baking.
Text Prompt: A car and traffic light on a city street.
Text Prompt: A living room for painting.
Text Prompt: A giraffe in the zoo.
Figure 13: More generated samples by our model.