LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts
Authors: Hanan Gani
Shariq Farooq Bhat
Muzammal Naseer
Salman Khan
Peter Wonka
Venue: ICLR 2024 poster (2024)
Keywords: diffusion
LLM
Primary area: generative models

Abstract: Diffusion-based generative models have significantly advanced text-to-image generation but encounter challenges when processing lengthy and intricate text prompts describing complex scenes with multiple objects. While excelling in generating images from short, single-object descriptions, these models often struggle to faithfully capture all the nuanced details within longer and more elaborate textual inputs. In response, we present a novel approach leveraging Large Language Models (LLMs) to extract critical components from text prompts, including bounding box coordinates for foreground objects, detailed textual descriptions for individual objects, and a succinct background context. These components form the foundation of our layout-to-image generation model, which operates in two phases. The initial Global Scene Generation utilizes object layouts and background context to create an initial scene but often falls short in faithfully representing object characteristics as specified in the prompts. To address this limitation, we introduce an Iterative Refinement Scheme that iteratively evaluates and refines box-level content to align them with their textual descriptions, recomposing objects as needed to ensure consistency. Our evaluation on complex prompts featuring multiple objects demonstrates a substantial improvement in recall compared to baseline diffusion models. This is further validated by a user study, underscoring the efficacy of our approach in generating coherent and detailed scenes from intricate textual inputs. Our iterative framework offers a promising solution for enhancing text-to-image generation models' fidelity with lengthy, multifaceted descriptions, opening new possibilities for accurate and diverse image synthesis from textual inputs.

Published as a conference paper at ICLR 2024
LLM BLUEPRINT: ENABLING TEXT-TO-IMAGE GEN- ERATION WITH COMPLEX AND DETAILED PROMPTS
Hanan Gani', Shariq Farooq Bhat?, Muzammal Naseer's ,Salman Khan
Peter Wonka "Australian National University
MMohamed Bin Zayed University of AI "KAUST (hanan.ghami, muzammal naseer, salman khan @mbzuai.acc.a. ac shariq. bhat. Shat@kkust.edde sdu.sa, pwonka@gmail.com
Text Prompt
object descriptions golden refriever: resling on plush rug coal
spatial relationships
unfolds. friendly and affectionate Golden Retriever with soft, golden-furred coat rests contently on plush rug, its warm eyes with joy. Nearby, graceful elegant white stretches leisurely, showcasing its pristine and fluffy fur. A sturdy wooden table with polished edges stands gracefully in the center, adorned with vase of vibrant modern television
In cazy living
heartwarming
golden retrevvr and cat sitiing nearby
golden
table stands at center
graceful
elagant with fluffy table: woaden
flower vase kept the table
freshness. stands ready to provide entertainment. The ambiance with a sense comganiorship and relavation.
addlnng
inviting, and filled
television: sleek modem living room: ambbeneee
flower vase: vibrant
talevision foood
Diffusion
GLIGEN
LayoutGPT
L.M-Groomded
white cal, alook telavision, wooden
Missing:
Missing: white cat. sleek
Missing: white cat
white Retriever,
Missing:
Golden
table,
television, nower
Deformed: golden retriever
Figure 1: Current state-of-the-a text-to-image models (Columns 1-4) face challenges when dealing with lengthy and detailed text prompts, resulting in the exclusion of objects and fine-grained details. Our approach (Column 5) adeptly encompasses all the objects described, preserving their intricate features and spatial characteristics outlined in the white boxes.
ABSTRACT
Diffusion-batt generative models have significantly advanced text-to-image gener- ation but encounter challenges when processing lengthy and intricate text prompts describing complex scenes with multiple objects. While excelling in generating im- ages from short, single-object descriptions, these models often struggle to faithfully capture all the nuanced details within longer and more elaborate textual inputs. response, present novel approach leveraging Large Language Models (LLMs) to extract critical components from text prompts, including bounding box coordi- nates for foreground objects, detailed textual descriptions for individual objects, and succinct background context. These components form the foundation of our layout-to-image generation model, which operates in two phases. The initial Global Scene Generation utilizes object layouts and background context to create initial scene but often falls short in faithfully representing object characteristics specified in the prompts. To address this limitation, we introduce an Iterative Refinement Scheme that iteratively evaluates and refines box-level content to align them with their textual descriptions, recomposing objects needed to ensure consistency. Our evaluation on complex prompts featuring multiple objects demon- strates substantial improvement in recall compared baseline diffusion models. This further validated by a user study, underscoring the efficacy of our approach in generating coherent and detailed scenes from intricate textual inputs. Our code is available at httpss//github..o./ wwclluubbooohaaaaaa www.//aaansaat/lllllleeeeeee

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Modern generative diffusion models, e.gRombach et al. Ruiz et al. (2023), provided a massive leap forward in the problem of text-to-image generation and have emerged as powerful tools for creating diverse images and graphics from plain text prompts. Their success can be attributed to several factors, including the availability of internet-scale multi- modal datasets, increased computational resources, and the scaling up of model parameters. These models trained using shorter prompts and especially excel at generating images of one prominent foreground object. However, as the description length and the number objects in the scene increase, modern diffusion models tend to ignore parts of the prompt often leading to critical omissions, misrepresentations, or the generation of objects that do not align with the nuanced details described in the prompts. Fig. shows a scenario where existing sfate-of-the-art text-to-image diffusion models struggle to follow all the details. This failure can partly be ascribed to the diffusion model's CLIP text encoder (Radford et al., 202T) which can only process the first 77 text tokens, effectively truncating longer prompts and potentially omitting critical details. Indeed, a single prompt describing a complex scene can span far beyond these token limits, making it a challenge for existing models to process and translate long prompts comprehensively.
(2022): ): Ho et al. (2020) Saharia et al. (2022)
:
Recent efforts (Epstein et al. 2023; Kang et al., 2023) have been dedicated to improving the capabilities of pre-trained diffusion models to faithfully follow the intricate details within text prompts. These works predominantly revolve around aspects such as object count (e.g. "2 oranges and 4 apples on the left of an apple"), " In the context of longer and more complex prompts, these models still tend to struggle to generate coherent images that faithfully reflect the complexity of the text prompts, especially when tasked with the placement of object instances at considerable spatial separations, often falling short of comprehensively capturing all instances of objects as intended. More recently layout-bur based diffusion models (Feng et al.) 2023 Li et al. 2023 Yang et al., (023b) have proven to be effective in capturing the count and spatial characteristies of the objects the prompt. Such models first generate bounding boxes of all the objects and then condition the diffusion model jointly on the bounding boxes and the text prompt to generate the final image. While effective in the case of small prompts, these models still struggle when presented with long text descriptions that feature multiple diverse objects and hence fail to generate the desired output (See Fig. D
the table"), and/or capturing spatial relationships among objects (e.g. "an orange
To address these challenges, our approach seeks to improve text-to-image generation from lengthy prompts. We introduce a framework that divides image generation into two phases: generating a global scene, followed by iterative refinement of individual object representations. We exploit LLMs to break down long prompts into smaller components organized in a data structure that we call Scene Blueprint. This allows to generate the image in step-wise manner. Our framework ensures that the final image faithfully adheres to the details specified in lengthy and complex text prompts.
We evaluate our framework on challenging prompts containing 3 to 10 unique foreground objects in varied scenes. Our results showcase a significant improvement in recall (~-85% compared to the baseline Feng al. (2023) (~69%) (+16 % improvement). We also include user study that demonstrates that our proposed method consistently produces coherent images that closely align with their respective textual descriptions, whereas existing approaches struggle effectively handle longer text prompts (See Fig.
In summary, our main contributions are:
Iterative image generation from long prompts: We introduce a two-phase framework for generating images from long textual descriptions, ensuring faithful representation of details.
We propose Scene Blueprints as a structured scene repre- sentation encompassing scene layout and object descriptions that enable coherent step-wise generation of images from complex and lengthy prompts.
Scene Blueprints using LLMs:
State-of-the-art results:
We present quantitative and qualitative results showcasing the effectiveness of our method, in terms of adherence to textual descriptions, demonstrating its applicability and superiority in text-to-image synthesis from lengthy prompts.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Text-to- tosttto--mmmeee Difftusion. Over the years, Generative Adversarial Networks (GAN) (Goodfellow
et al. 2014) have been the default choice for image synthesis (Brock et al. 2018 Reed al. 2016
Xu et al. 2018] Zhang et al.] 2017 2021 Tao et al. 2022 /Zhan et al. 201 8a Karras et al. 2019).
However, more recently, the focus has shifted towards text conditioned autoregressive models (Ding
ct al. 2021 Gafni et al. 2022 Ramesh et al. 2021 Yu et al. 2022) and diffusion models (Rombach
ct al. 2022 Gu et al.] 2022; Nichol al. [022]] Ramesh al., [022]] Saharia et al. 2022) which
have exhibited impressive capabilities in producing high-quality images while avoiding the training
challenges, such as instability and mode collapse, commonly associated with GANs (Arjovsky et al.,
2017, Gulrajani et al., 2017; Kodali et al., 2017). In particular, diffusion models are trained on large-scale multi-modal data and are capable of generating high-resolution images conditioned on text input. Nevertheless, effectively conveying all the nuances of an image solely from a text prompt can present a considerable hurdle. Recent studies have demonstrated the effectiveness of classifier-free guidance (Ho & Salimans 2022) in improving the faithfulness of the generations in relation to the input prompt. However, all these approaches are designed to accept shorter text prompts, but they tend to fail in scenarios where the prompt describing a scene is longer. In contrast, our proposed approach generates images from longer text prompts, offering an efficient solution to address this challenge.
Layout-to-Image Generation. Generating images from layouts either in the form of labeled bounding boxes or semantic maps was recently explored in (Sun & Wu 2019 Sylvain al. 2021 Yang et al. 2022 Fan et al. 2023 Zhao et al., 2019; Park et al. 2019]. Critically, these layout to image generation methods are only conditioned on bounding boxes and are closed-set, i.e., they can only generate limited localized visual concepts observed in the training set. With the inception of large multi-modal foundational models such as CL.IP (Radford et al. 202T). has now been possible generative models can be conditioned
to generate images in an open-set fashion. Diffusion- on multiple inputs, however, they have been shown to struggle in following the exact object count and spatial locations in the text prompts (Chen et al. 2023 Kang et al. 2023). More recently layout conditioned diffusion models have been proposed to sol ve this problem (Chen et al. 2023: Li et al.
2023 Yang et al.) 2023b Phung al.) [202)). Chen et al. (023) manipulates the cross-attention layers that the model uses to interface textual and visual information and steers the reconstruction in the desired user-specified layout. GLIGEN (Li et al., 2023) uses a gated self-attentio layer that enables additional inputs (e.g., bounding boxes) to be processed. ReCo (Yang et a (202bb) achieves layout control through regional tokens encoded as part of the text prompt. Zhenng et al. (2023) introduce LayoutDiffusion which treats each patch of the image as a special object for multimodal fusion of layout and image and generates images with both high quality and diversity while maintaining precise control over the position and size of multiple objects. In addition to this, there have been few works on LLM-based layout generation (Feng et al. [023 [Lin et al. (023). These works exploit the LLMs' abilities to reason over numerical and spatial concepts in text conditions (Li et al. 2022a). Building upon these works, we extend LLMs' powerful generalization and reasoning capabilities to extract layouts, background information, and foreground object descriptions from longer text prompts.
Diffusion Based Image Editing and Composition. Diffusion-bart image editing has received overwhelming attention due to its ability to condition on multiple modalities. Recent works utilize text-based image editing using diffusion models to perform region modification. DiffusionCLIP (Kim) 2022) uses diffusion models for text-driven global multi-attribute image manipulation varied
et al. domains. Liu et al.)
2023) provides both text and semantic guidance for global image manipulation. In addition, GLIDE Nichol et al. 202I) trains diffusion model for text-to-image synthesis, as well as local image editing using text guidance. Image composition refers to a form of image manipulation where foreground reference object is affixed onto a designated source image. naive way to blend a foreground object on a background image may result in an unrealistic composition. However, more recent works (Avrahami et al., 202T, Yang et al. 2023a Ho & Salimans 2022 Lu et al., 2023) use diffusion models to overcome the challenges posed due to fusion inconsistency and semantic disharmony for efficient image composition. Avrahami et al. (202I) takes the target region mask and simply blends the noised version of the input image with local text-guided diffusion latent. Yang et al. (2023a) trains a diffusion model to blend an exemplar image on the source image at the position specified by an arbitrary shape mask and leverages the classifier-free guidance (Ho & Salimans 2022) to increase the similarity to the exemplar image. Lu et al. (2023) introduces TF-ICON which leverages off-the-shelf diffusion models to perform cross-domain image-guided composition without
3

---

Published as a conference paper at ICLR 2024
Global Generation
Iterative Refinement
descrptions
LLM
Bics Layeuts
Layout Generation
Barative Rulinement Schieme
Leyout Interpdiaionn
Bacigroand
Ougput
Loop through all box proposals
Iterative Refinement Scheme
long Pronepe
a copy livig ioom,
Sampling
untdcc A Mordd and
24
Retiover with rosta confentll top, Filled byy Neamy, a grapeht oogant
Lacoet
alice
FIME
UNET
Guidance
Loss
pitfuston
Figure 2: Global Scene Generation: Our proposed approach takes long text prompt describing a complex scene and leverages L.I to generate & layouts which are then interpolated to single layout, ensuring the spatial accuracy of ohject placement. Along with the layouts, we also query L.LM generate object descriptions along with concise background prompt summarizing the scene's s essence. Layout-to- Image model is employed which transforms the layout into an initial image. Iterative Refinement Scheme: The content of each box proposal is refined using a diffusion model conditioned on box mask, (generated) reference image for the box, and the source image, guided by multi-modal signal.
requiring additional training, fine-tuning, or optimization. Based on these works, we build an iterative refining scheme, shape and modifies each layout based on the object characteristics guided by multi-modal loss signal.
which performs region-based composition the layout level, utilizing given mask
METHODOLOGY
3.1 PRELIMINARIES ON DIFFUSION MODELS.
Diffusion models are generative models that learn the data distribution of complex datasets. They consist of forward diffusion process and reverse diffusion process. During the forward process, noise is added to the input data point zo for 7 steps, until the resulting vector 2, is almost distributed according to a standard Gaussian distribution. Each step in the forward process a Gaussian transition q(x; xt-1) ((v1- - - BrI), where { )7 is fixed learned variance schedule. The resulting latent variable æ, can be expressed as:
xt Vaiao + V1 11-a-6 016; E 6~ IO,,
where a2 : IT- (1 (.). The reverse process q(x1-1 x;) is parametrized by another Gaussian transition Pa (z,-1 x.) := A (xx-1i po (x,, t), oo (2, I).. (I,, t) can be decomposed into the linear combination of ar; and noise approximation model en (are, 1), which is trained that for any pair (zo, 1) and any sample of 6,
Et
/ Va,B0
Ea (0tt, (0
V1
After training 6ø (x. f), different works (Song et al. study different approximations of the unknown g( - Lt Fo perform sampling. In our work, we utilize the denoising diffusion implicit model (DDIM) introduced by Song et al. (020a) to predict the clean data point.
2020a Song & Ermon 2019 Song et al. 2020b)
3.2 OVERVIEW
Our core objective is generate images from long textual descriptions, ensuring that the resulting images faithfully represent the intricate details outlined in the input text. We generate the output image in multi-step manner; generating initial image that serves template at global level, followed
4

---

Published as a conference paper at ICLR 2024
Figure Effect of interpolation factor my We interpolate the l bounding boxes for each object and control the interpolation by the factor 7/. We visualize the change in the bounding box location of "a white cat" highlighted in the text for different values from 0.1 to 0.9 with increments of 0.1. Best viewed in zoom.
leyoate
Fest pracee
vit aider Srow
Linear Interpolation
0=02
0-02
q- to
n-dz
by box-level refinement phase that serves corrective procedure. 1) Global scene generation: We begin by decomposing lengthy text prompts into "Scene Blueprints." " Scene Blueprints provide a structured representation of the scene containing: object bounding boxes in image space, detailed text descriptions for each box, and a background text prompt. We use an LLM to extract the Scene Blueprint from the given long prompt and use layout-conditioned text-to-image models to generate the initial image. Unlike prior works, we support additional user control on the box layouts by generating multiple proposal blueprints and providing an option to smoothly interpolate between the candidate layouts. This interpolation not only facilitates (optional) user control but also helps mitigate any potential errors introduced by the L.M. 2) Box-level content refinement: In the second phase, we iterate through all the boxes and evaluate and refine their content in terms of quality and adherence to the prompt. We use multi mollll guidance procedure that maximizes a quality score for each box.
3.3 GLOBAL SCENE GENERATION
Textual representation of scene contains various characteristics that provide information about objects, their spatial properties, semantics, attributes, and more. To coherently capture these properties, we employ an off-the-shelf pre-trained large language model (penAi) 2021). We instruct
the LLM with appropriately engineered prompt (see supplementary Sec. A.4 to generate Scene Blueprint containing the following three components:
1. Layout: Bounding box coordinates for each object {object (x, y. width, height)
2. Object description: Description associated with each object {object ; description)
3. Background Prompt:
A general prompt describing the overall essence of the scene.
The layout and the background prompt generated by the LLM are then used to condition the diffusion model to generate an image. We follow recent work by [ian et al. (2023) which generates the image from the layouts in two steps. 1) generating masked latent inversion for each object bounding box, and 2) composing the latent inversion as well as generating the corresponding background from the background prompt.
Layouts Interpolation and Noise Correction. While LLMs have advanced spatial reasoning abilities, observed that they struggle model the spatial positions of the objects when presented with longer descriptions, often resulting in abnormal relative sizes of the objects and unnatural placement of object boxes in the scene (Sec 4.1 A naive approach to resolve these issues is to fine-tune LLM on handcrafted data of text descriptions and bounding boxes. However, this approach requires extensive resources terms of human annotators and compute requirements, and risks catastrophic forgetting (Luo et al. 2023). On the other hand, correcting these errors manually by adjusting the boxes is also daunting task and defeats the purpose of using an LLM extract layouts. To address these challenges, we propose simple yet effective solution layout interpolation. Instead of generating only one proposal layout, we query the I.LM to generate i layouts. Subsequently, we employ linear interpolation to compute the coordinates for each object's final bounding box, denoted as (1)).
0j (z), úر. ú hy)) where for any coordinate & E oj, 2, =Interpolation(2) 22(2),((3)
The interpolation function recursively updates each coordinate such that at any iteration / of bounding boxes. 2( := (1 (1-ny n7) 2f(-1)
1) + 1 (!) where is the interpolation factor which controls the influence of the individual bounding boxes the final interpolated box (see Fig.
5

---

Published as a conference paper at ICLR 2024
As the long complex prompts may result in a large number of boxes, the images generated by the layout guidance tend to have color noise and small artifacts (See Fig. g in supplementary). Therefore, we optionally perform image-to -mmae translation step using stable diffusion (Rombach et al.) 2022), resulting in a cleaner image while preserving the semantics. We refer to this image as Vinitial-
Despite conditioning on layout, we observe that the diffusion model is unable to generate all scene objects effectively. It struggles to compose multiple diverse objects having varied spatial and semantic properties in one shot. Consequently, Finitiol often has missing objects or fails to represent them accurately in accordance with their descriptions. Therefore, we employ a box-level refinement strategy that evaluates and refines the content within each bounding box of the layout in terms of quality and adherence to the prompt.
3.4 BOX-LEVEL REFINEMENT
Current diffusion models have certain limitations in terms of composing multiple objects in the image when presented with longer text prompts, problem that is still unexplored. To overcome this issue and ensure faithful generation of all the objects, we introduce an Iterative Refinement Scheme (IRS). Our proposed IRS works at the bounding box level and ensures the corresponding object at each bounding box is characterized by its properties given in the textual description. To achieve this, we iterate across each object's bounding box in Fiiitiol and compare the visual characteristics of the object with its corresponding description extracted from the text prompt. Consider an object j with its bounding box 0 = (Øzz M/j vūj, hy) and its textual characteristics denoted by dj. we use CLIP score (Hessel al. 2021 as metric to get the similarity between the object and its description such that, (core(s) CLIP ( (ooject()),, dz). ). If the CLIP score s is below a certain threshold, we modify the content of the bounding box such that follows its corresponding description.
Any reasonable modification of the content within a bounding box must improve its fidelity and adherence to the prompt. Since diffusson models, e.g. stable diffusion (Rombach et al. 2022). are already good at generating high-fidelity images from shorter prompts, we exploit this ability and follow haint-by-example approach. We generate a new object for the designated box by passing the object description d, to a Text-to-Image stable diffusion model. The generated image x/// acts reference content for the bounding box Oj. We then use pretrained image composition model Yangg et al. 2023a) conditioned on the reference image x1 mask my (extracted from the bounding box), and source image Tinatiol to compose the reference object at the designated position on the source image specified by the mask. To ensure the composition generates the object that follows the properties described in the text prompt closely as possible, guide the sampling process by an external multi-modal loss signal.
Box-level Multi-modal Guidance. Given the initial image Timitial a reference image x"j ref text prompt d, and a binary mask my that marks the region of interest in the image corresponding to bounding box Oj, our goal is to generate modified image F, s.t. the content of the region ④ my is consistent with the prototype image gef and adheres to the text description d,, while the complementary area remains as close as possible to the source image, i.e., Tinitial O (1 my) P f o (1 m,) ), where @ is the element wise multiplication. Dhariwal & Nichol (2021) use classifier trained on noisy images to guide generation towards a target class. In our case, we use an external function in the form of CLIP to guide the generation in order to adhere to the prototype image a5/// and text description dy. However, CLIP is trained on noise-free data samples, we estimate a clean image Di from each noisy latent 24, during the denoising diffusion process via Eqn. [ as follows,
a guiding
3t (V1
04 (n Vou
( (2z,1)
Our CLIP based multimodal guidance loss can then be expressed as,
LCLIP = Lcosine (CLIPimege image (ão @ my), CLIP (d,))
(4)
A Leasine CLIIP (To C my), CLIP
ELIPimago (10 (o)e1)).
where Leosine denotes the cosine similarity loss and À is a hyperparameter. The first part of the equation measures the cosine loss between the composed object at the region specified by the mask

---

Published as a conference paper at ICLR 2024
2458
72.74
03.5%
Figure 4: User study. A majority of users picked our method compared to prior works when presented with 2-AFC task of selecting the image that adheres to the given prompt the most.
my and its corresponding text characteristics dz. The second part of the equation measures the cosine loss between the composed object and its corresponding prototype xf//_ A similar approach using CLIP text-based guidance is used in Avrahami et al. (2021) for region-based modification. However, in contrast, we also include CLIP image-basee based guidance steer the generation toward the prototype image x; to account for the fine-graine details that may not be captured in the text description. In order to confine the modification within the given bounding box, we optionally employ a background preservation loss Lig which is a summation of the L2 norm of the pixel-wise differences and Learned Perceptual Image Patch Similarity metric Zhang et al.. (2018b) between To @ (1 m,) and Finitial @ (1 mj) The final diffusion guidance loss is thus the weighted sum of LCLIP and Lbg given as,
Lguidance = LCLIP + Lbg-
(5)
The gradient of the resultant loss Vz Cysidance is used to steer the sampling process to produce an object at the bounding box og which follows the properties of prototype ref and description dj. Additionally, refer to the supplementray section A.8 to understand how guidance loss influences the generation process and algorithm A. for a full end-to-end pipeline.
4 EXPERIMENTS
Settings: Our framework uses a combination of several components. For acquiring the long text descriptions, we ask ChatGPT to generate scenes on various themes. In addition this, also use the textual descriptions from some COCO (Lin et al. (014)) and PASCAL (Everingham et al. 2010) images by querying an image captioning model (Zhu et al. (023) to generate a detailed description spanning 10-100 words. For extracting layouts, bounding boxes, and background prompt, we make use of ChatGPT completion api (OpenAl 2021) with an appropriate instruction template (See Supplementary Sec. A.4).. We generate 3 layouts for each text prompt and interpolate them to single layout to account for the spatial location correctness. To avoid layout overlap, we push the boxes away from each other until there is minimal contact wherever feasible. For base layout-to- image generation, we use the work Lia et al. (2023) and scale it for longer text prompts. We use 20 diffusion steps at this point. For box refinement, of Yang et al. (2023a) which conditions on a reference image. For each box refinement, we use 50 diffusion steps. For implementation, Nvidia A 100 40GB GPU.
use the pre-trained image composition model
Pytorch 2.0. Finally, our entire pipeline runs on single
Our work stands first effort to address the challenge of generating images from extensive text prompts. As discussed in Section | current siffusion-based texl-to-image generation methods typically utilize the CLIP tokenizer condition the diffusion model for image generation. However, this approach can lead to inconsistent images when confronted with lengthy text prompts with intricate details. To the best of our knowledge, there is currently no established metric for assessing the performance of diffusion models in handling lengthy text descriptions. Hence we propose to use the Prompt Adherence Recall (PAR) score to quantify adherence to the prompt defined as Mean of object presence over all objects over all prompts where we use an off-the-shelf object detector (Li et al. 2022b) to check the object actually present in the generated image such that object presence is if present and 0 otherwise. We achieve PAR score of 85% which significantly better than Stable Diffusion (49%), GLIGEN (57%) and LayoutGPT (69%). We also conducted extensive user study assess the effectiveness of our method in comparison to four established bascline approaches: GLIGEN (Li et al., 2023), LayoutGPT (Feng et al. (022)) and L.M--Grudd Diffusion (Lian et al.
Quantitative Results:
2023 To mitigate any bias, participants were instructed to
7

---

Published as a conference paper at ICLR 2024
Long Textual prompt
Stable Diffusion
GLIGEN
LayouGGPT
LLM-Grounded
Ours
The image shows liing room with chaisimas
caree There
couth a chall, and coffe labll in the The aastl cattled while there larse window with ouraitt on one There a bisscle pontedt in the comee of the
room. The room is well by shandeser nandang from the celling The overall atmosphere of the room oszy and festive.
chair, table, whitel
chrietmas tree, chandelier, bicycle
bicycle, coffee table christmas tree
painted walls, bicycle tree
bicycle christmas tree
the quiet countryckde, a cod farmouuse stands with als--aahioned charm. Nearby. weathered acckot fance surounds garden of widfoowers An atilousl fracter though
reminndr work. scanectoze watcheia over feids swaying arops. The air carres the scent of earth hay. against rolling hills this tamhouse story connection the land and ite traditinns
Unrealistic image
scarecrow, fence, antique tractor
scarecrow, fence red color of farmhouse
scarecrow, farm house, modem tractor antique tracior
scarecrow, fence
visiers
Kactis Arr this natural cana, streis banth beckons, somncourry
from the
Nouety, quaest buschag Hends searlwody with the park's lardhccpp,,
touth archteettmm charm harmorious fuston of lush greenery and tranqul spaces. where the chair oftere solaco, the gross whispers secrets, and the para issit is toven of sorenity
The
bench, sun's warm embrace, quaint building.
sun's warm embrace, quaint building
warm embrace
park bench, sun's warm embrace
The image:
with atble ooffee Intile The table has drawees and bottle in clecert on tog of There is delessition urr ar mtbtle alond in the conner the rom, a wiite cobinot Just beside The walls palited and the se
mnndow
white walls, television, cabinet, white coffee table. while stand
on one wall. The overall effect is one of calm and serentty
bottle, white cabinet
white cabinet, white stand
bottle, television, white cabinet
eath
mentine
pateerly conbasting the notull sumourding with modemity An and onnce occeadh the abide dasnncy noe otter trance spot reflectoon High above a scluts kis zoam puccetyyy in the scan ale suptainn the canehee apint of the day. This hammoiious blend reture and numann eeeents makes the part perfect pase for leturcy strals and momontt roloaation.
Nenryy pached wibe
Unrealistic image white car, colorful kite, empty bench
brown cow, white car, colorful bench
Unrealisticc image, deformed cow; brown cow, shade of free
white car, bench; position of cow
Figure 5: Qualitative comparisons: We compare our image generation method to state-of of-te-of-the-art art baselines, including those using layouts. The underlined text in the text prompts represents the objects, their characteristics. and spatial properties. Red text highlights missing objects, purple signifies inaccuracies in object positioning, and black text points out implausible or deformed elements. Baseline methods often omit objects and struggle with spatial accuracy (first four columns), while our approach excels in capturing all objects and preserving spatial attributes (last column).
select one image from pair of images randomly selected from two distinct approaches. Their goal was to choose the image that most accurately represented the provided textual descriptions regarding spatial arrangement, object characteristies, and overall scene dynamics. The outcomes of the user study are presented in Fig. 4 Our findings demonstrate that, on average, our proposed method consistently produces coherent images that closely align with their respective textual descriptions, whereas existing approaches struggle to effectively handle longer text prompts. For a detailed procedure and user-study results on the fidelity, refer to section A.9 of supplementary.
Qualitative Analysis: Fig. 5 presents qualitative assessment of our method in comparison to established state-of-thh-aar methods. The text descriptions include specific phrases denoted by underlined italics, conveying information about objects, their attributes, and spatial arrangements.
8

---

Published as a conference paper at ICLR 2024
Figure 6: Effect of Layout Interpolation: Our layout interpolation method (last column) sig- nificantly improves object spatial positioning compared to non-interpolated cases (first two columns). Best viewed in zoom.
Without intorpoaiion
wish Enpopoation S
kagge domn Veey mae
colbe sele theren The
san sik Alcic Sela
Figure 7: Effect of Guidance: Without guid- ance signal, the composed image does not follow the properties corresponding to its de- scription and visual appearance. contrast, the one with the guidance (right) adheres to the visual prototype and description.
skgnals
klance
gakannca
Cat description:
groceful white cat gracefully stretching, showing off its luffy, pritine
Notably, the red text beneath each image highlights instances of missing objects, while the purple text indicates spatial inaceuracies, and the black text identifies elements of implausibility or distortion. From the figure, the stable diffusion baseline (column 1) frequently falls short in incorporating certain objects from the prompt due to its inability to efficiently handle lengthy text prompts. In some instances (rows 2 and 4), the generated images exhibit unrealistic features. Layout-based approaches (columns 2, 3, and 4) also encounter difficulties in fully capturing the nuances of the text prompt, resulting in occasional omissions of objects and instances of deformations (column 3, row 5). In contrast, our approach excels by accurately capturing all objects from the text, including their intricate details and precise spatial positions. Refer sections A.3 and A.5 for further results and comparisons with additional baselines.
4.1 ABLATIONS
Effect of Layout Interpolation. We query the LLM to generate multiple layouts from the input text prompt and then employ linear interpolation to merge them into a single layout. However, for lengthy prompts, LLMs can occasionally generate layouts with random object placements, resulting in unnatural images. As shown in Fig. [6] the first two columns depict images without layout interpolation, while the last column shows the interpolated image. The underlined phrases in the text prompt indicate object spatial characteristics. In contrast, the last column demonstrates the improved result with interpolation, aligning nearly every object with its textual spatial description.
Effect Guidance The external guidance the form of CLIP multi-mode loss used the refinement stage steers sampling of the specific box proposal towards its corresponding description and reference prototype. We present a visual illustration of this phenomenon in Fig. 7 As seen from the figure, the properties of the cat get more aligned with the prototype image and text description in the presence of a guidance signal.
5 CONCLUSION
In this work, we identified the limitations of prior text-to-image models in handling complex and lengthy text prompts. In response, we introduced a framework involving a data structure (Scene Blueprint) and multi-step procedure involving global scene generation followed by an iterative refinement scheme to generate images that faithfully adhere to the details in such lengthy prompts. Our framework offers promising solution for accurate and diverse image synthesis from complex text inputs, bridging a critical gap in text-to- eoxt-to-image synthesis capabilities.
While we presented a simple interpolation technique to combine various bounding box proposals, we maintained fixed box layouts for the second phase. A promising avenue for future research lies in exploring dynamic adjustments of boxes within the iterative refinement loop. Another area warranting further examination pertains to the handling of overlapping boxes. While we currently address this challenge by sorting boxes by size prior to the box-level refinement phase, there is an opportunity to explore more advanced techniques for managing overlaps. Additionally, box-level refinement treats each object in isolation, overlooking the relationships that exist among objects within a scene. A compelling avenue for future research is to incorporate and leverage these object relationships, with the aim of achieving more comprehensive and contextually aware image generation.
current approach to

---

Published as a conference paper at ICLR 2024
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18187-18197, 2021.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 843-852, 2023.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 2018.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.0337. 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 079008794,22 2021.
Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang. et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34: 19:22 19835, 2021.
Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. arXiv preprint arXiv:2306.00996 2023.
M. Everingham. L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303 June 2010.
Wan-Cyuan Fan, Yen-Chun Chen, DongDong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. In Proceedings of the AAAl Conference on Artificial Intelligence, volume 37, pp. 579-587, 2023.
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305. 2023.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. In European Conference on Computer Vision. pp. 89-106. Springer, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10696-10706, 2022.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint urXiv:2104.08711 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint 1rXiv:2007.12598 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Adyances in neural information processing systems, 33:6840-6851, 2020.
10

---

Published as a conference paper at ICLR 2024
Wonjun Kang, Kevin Galim, and Hyung 11 Koo. Counting guidance for high fidelity text-to-image synthesis. arXiv preprint arXiv:2306.17566 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVP conference computer vision and pattern recognition, pp. 4401-4410, 2019.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2426-2435, 2022.
Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung- Jung-Woo Ha, and Jun-Yan Yan Zhu. Dense text-to-image generation with attention modulation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7701-7711, 2023.
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv preprint arXiv: 705.07215.5 2017.
Boyi Li, Rodolfo Corona, Karttikeya Mangalam, Catherine Chen, Daniel Flaherty, Serge Belongie, Kilian Q Weinberger, Jitendra Malik, Trevor Darrell, and Dan Klein. Does unsupervised grammar induction need pixels? arXiv preprint arXiv: 10564, 2022a
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jeng-eengggggggeeeeeeeeeeeeeeeae Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training, 2022b.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511-22521, 2023.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Lim-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint 23555 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Xingchao Liu, Dahun Hwang Park, Samanch Azadi, Guandao Zhang, Armen Chopikyan, Yizhe Hu, ... and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.
289-299, 2023.
Shilin Lu, Yanzhu Liu, and Adams Wii-Kin Kong. Tf-icon: Diffusion-based training-free cross- domain image composition. In Proceedings of the IEEF/CVF International Conference on Com- puter Vision, 2023.
Yun Luo. Zhen Yang. Fandong Meng. Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint SBTi:2300.0874477 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 12. 10741, 2021.
OpenAl. Chatgpt: A large-scale generative model for conversations. 2021.
Tacsung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptiv normalization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2337-2346, 2019.
Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. arXiv preprint arXiv:2306.05422 2023.

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 9748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 1(2):3 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069. PMLR, 2016
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer ence on computer vision and pattern recognition, pp. 10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-3666 -36444, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020b.
AtabilityAI. Deepfloyd if, 2023.
Wei Sun and Tianfu Wu. Image synthesis from reconfigurable layout and style. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 10531-10540, 2019.
Tristan Sylvain, Pengchuan Zhang, Yoshua Bengio, Devon Hjelm, and Shikhar Sharma. Object- centric image generation from layouts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2647-2655, 2021.
Ming Tao, Hao Tang, Fei Wu, Xiao Yuan Jing, Bing-Kun Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine- grainer text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1316-1324, 2018.
Binxin Yang. Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar- based image editing with diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 18381 1338111839,, 2023a.
12

---

Published as a conference paper at ICLR 2024
Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region controlld text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14246-14255, 2023b.
Zuopeng Yang, Daqing Liu, Chaoyue Wang, Jie Yang, and Dacheng Tao. Modeling image compo- sition for complex scene generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7764-7773, 2022.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to- generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan+++; Realistic image synthesis with stacked generative adversarial networks. 8a.
IEEE Iransactions patlern analysis and machine intelligence,
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for sext-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 833-842, 2021.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric, 2018b.
Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8584-8593 2019.
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22490-22499, June 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En hancing lision-language understanding with advanced large language models. arXiv preprint a3Xiv:2304. 10592, 2023.
13

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.I ALGORITHM
In Sec. we describe our proposed approach LLM Blueprint. Our method uses an LLM to first generate bounding box layouts and descriptions pertaining to each object in the description, along with background prompt that describes the overall scene in the text. We use specific instruct prompts to do so (see Appendix A.4). Due to possibly inconsistent box positions, we query the LLM to generate & layouts giving &o bounding boxes for each object in the layout. We then linearly interpolate the coordinates to obtain final bounding box layout denoted O. We use layout-to-imen image generation model M to obtain an initial image. However, as described in Sec of the main paper, apply additional noise correction step by passing the generated image to image-to-image diffusion model SD 5Dimage-imaag while retaining the semantics of the scene. We call the image generated until this point as Global Scene Generation and denote it as Pinitial- However, the image generated after the first phase Tinitial still has inconsistencies in generating objects as per their characteristics. To overcome this issue and ensure faithful generation of all the objects, we introduce an Iterative Refinement Scheme (IRS). Our proposed method works at the bounding box level and ensures the corresponding object at each bounding box characterized by properties given the textual description. We achieve this by looping across each bounding box and iteratively modifying the regions based on a CLIP metric. In other words, we generate a new object for the designated box by passing the object description dj text-to-image stable diffusion model S image- The generated image z/ acts as a reference content for the bounding box oj. We further create a binary mask mj from the bounding box coordinates. We then use image composition model Moomp (mo( Eg (2+)) conditioned on the reference image tef, mask m, and source image Fenitial to compose the reference object at the designated position on the source image specified by the mask mj. To ensure the composition generates the object that exactly follows the properties described in the text prompt, we use an external function in the form of CLIP to guide the generation in order to adhere to the prototype image ref and text description dj. However, since our prototype 2//e/ is in the input space and CLIP is trained on real- wol-world data samples, we estimate a clean image 2o from each noisy latent I₃ during the denoising diffusion process via Eqn. [ (main paper). We optionally add a background preservation loss Lig to avoid the composition process affecting the background. We provide pseudo code of our algorithm in Algorithm
A.2 NOISE AND ARTIFACT CORRECTION
As discussed in Sec. 3.3 the generated image after the first phase sometimes suffers from noise or artifacts. We do an optional noise correction step after Global scene generation, which essentially removes the unwanted noise and artifacts. Specifically, we utilize the amage-to-imaat translation method of stable diffusion Rombach et al. (2022) which instead of starting from random noise, starts from input image, adds noise to it and then denoises it in the reverse process. The idea is to enhance the quality of the image while maintaining its semantics. We notice that this process removes the unwanted noise and artifacts present in the image (See Figure g
Input
Noise corrected
Input
Noise corrected
Input
Noise corrected
Figure& 8: Noise Correction: The noise correction strategy removes noise and certain redundant artifacts from the image that unnecessary.
A.3 FURTHER QUALITATIVE COMPARISONS
We provide further qualitative comparisons of our approach with state-of-the-art baselines in Fig.
14

---

Published as a conference paper at ICLR 2024
Long Textual prompt
Stable Diffusion
GLIGEN
LayourGPT
LLM Grounded
Ours
labreau comes to Ho. auuty cou with de datter blach andaan marksgg grades ocrenely 30l cocasionely awating ocacety hooee stand matesscally,
Noanthy
caseniorally Blnng heed survey the surmunncing nclings. This maqqui ouidoor moment capti the esserca hamonious coenioennce oo omong complemenay
unreslistic image chestrut colored
dog. chestrut colored dog, chestnut colored horso,
big one dog. chestnul colored
these
horse
tree
trangul park bench under geresnn cal tres where a forgotion doen cool on the noodot cost Nooray, acattitio calerd aonums leava
lege.
un asaem. ciecet sound shiping bidds
gentle breeze
through the laaves create a screne atmosphero this queee corner of the park, inviting passersby sayy to pauss and heaidy.
book, squirel
scuirrel, book.
squirel
squirrel
mon with
totee and shalle The tabie made orunnd There cotted pant toblle mala taiieed snloe there are asddias milt wtile surains-
overneed tamp. The floor bied The overall stmosphere and inviing.
white curtains, two large windowe. overnead lamp
yelloww curteins
wells yellow walls, overhead lamp
ysllow wells, while curtains, evernead lamp
small windows
potted plant
a manqul backyard parden
sunch beneeth
cbeny
hasetry
bnd
with costull-kaar water, serves rotresting ig coss for toathored ttstors Keeping wasch over wome, aciott colond
goccnes
As perde tresus custle the leavess nearby shruba, the gardon bookers with its amteence. mnvting fnn soloce
bird bath, tro gerden gnomes
bind bath, trio of gerden gromes
garden gnomes, cherry
garden
bird both
roreat
The imago shows deek oill littie costouter trotoo soce sopwers to be tomnel un. Therw gietuw farm on woll betind the deek. there
mater died doat Iacipe. deat. made of wood and haoer stenee is painted a light color.
laptop
lamp, glass of water, crawer
drawer, glass water worden table, drawer. picture frame, draweer pioture frame, lamp lamp
glass picture frame
picture frame
of shows loo shaioos moder aut0 of white lbuers on the tab The basharound wooded, mall with bright and evenly diariuuees. costing shodows on the tadle and builees The nownll muxd relased suphisdicated.
extra glasana vodka bollles id biluee slriped wooden wall. flower
emply 2 glasses. vodka
two glasses, while and blue striped wall
vodka bottles, 2 glasses, vase of floerss, wooden wall
Figure 9: Qualitative comparisons: We provide further qualitative comparisons ofte-of-thh-as art baselines. The underlined text in the text prompts represents the objects, their characteristics. and spatial properties. Baseline methods often omit objects and struggle with spatial accuracy (first four columns), while our approach excels in capturing all objects and preserving spatial attributes (last columa).
approach against the
15

---

Published as a conference paper at ICLR 2024
Algorithm 1 LLM Blueprint
Input: Long textual description C, diffusion steps k, sampling iterations n, layout-to-image model
ML. LLM, image composition model Mcomp (21((2)), Eo(r:)), LLM queries q, stable diffusion model SD. CLIP model, Interpolation function.
Output: Composed image F encompassing all the elements of the textual description C.
// Get layouts, object details, and background prompt from LLM.
0 k. D, Pi, + (C)(C)
// Interpolate layouts.
O Interpolation(N))
// Generate initial image from interpolated layout.
Finitial
// Optional noise correction.
Tinitial DDimage-immss (Jinitial)
/ Iterative Refinement.
foreach (o,, dj) in {O, do
zj ref + S (dz)
my - GetMask 0,))
Ik ~N(āaro
ā+) I)
for I. from T to do
for iter from 1 10 do
11 Σ uo (
Ex( It (m)
vā
Voz
L + Laur (To, m)) ACuur(To, x0 mm))
,m,,) + + 7Lor (2, Fo.m. m3)
+ (I,)) v1 - Vz,
Fig + va--1 2: vI-vi + +1=1= - Ō2-1
Vol
It-1 t my O Big + (1 - m2) O. ON (1
20 f- Tt-1
return 20
A.4 INSTRUCT PROMPTS FOR LLM
Our approach utilizes LLMs' advanced spatial and reasoning capabilities derive layouts, object descriptions, and background prompt from long textual description. For extracting layouts and background prompts, we use the prompt designed by [Lian et al. (2023) and scale it to work on longer textual descriptions. For extracting object descriptions, we designed our own unique prompt. Below are the instruct prompts utilized our work.
Instruct prompt for extracting layouts and background prompt.
You intelligent bounding box generator. will provide you with caption for photo, image, a detailed scene, or a painting. Your task is to generate the bounding boxes for the objects mentioned scribing the scene. The images are of size 512x512. The top-left corner has coordinates [0, 0). The bottom-right corner has coordinates (512, 512]. The bounding boxes should not overlap or go beyond the image boundaries. Each bounding box should be in the format of (object name, [top-left x coordinate, top-left y coordinate, box width, box height]) and include exactly one object (i.e., start the object name with "a" or "an" if possible). Do not put objects that are already provided the bounding boxes into the background prompt. Do not include non-existing excluded objects in the background prompt. If needed, you can make reasonable guesses. Please refer to the example below for the desired format.
the caption, along with background prompt
Caption: In the quiet countryside, red farmhouse stands with an old-fashioned charm. Nearby, weathered picket fence surrounds garden of wildflowers. An antique tractor, though worn, rests as a reminder of hard work. scarecrow watches over fields of swaying crops. The air carries the scent of earth and hay. Set against rolling hills, this
16

---

Published as a conference paper at ICLR 2024
farmhouse tells story of connection to the land and its traditions
Objects: [('a red farmhouse', [105, 228, 302, 245]), ('a weathered picket fence', [4, 385, 504, 112)), ('an antique tractor', [28, 382, 157, 72)), ('a scarecrow', [368, 271, 66, 156)) Background prompt: A realistic image of a quiet countryside with rolling hills
Caption: A realistic image of landscape scene depicting green parking on the left of a blue truck, with a red air balloon and a bird in the sky
Objects: [('a green car', [21, 181, 211, 159)), ('a blue truck', [269, 181, 209, 160)), ('a red air balloon', (66, 8, 145, 135)), ('a bird', (296, 42, 143, 100))]
Background prompt: realistic image of a landscape scene
Instruct prompt for extracting object descriptions.
You are intelligent description extractor. I will give you list of the objects and corresponding text prompt. For each object, extract its respective description or details mentioned in the text prompt. The description should strictly contain fine details about the object and must not contain information regarding location or abstract details about the object. The description must also contain the name of the object being described. For objects that do not have concrete descriptions mentioned, return the object itself in that case. The output should be a Python dictionary with the key as object and the value as description. The description should start with 'A realistic photo of object' followed by its characteristics. Sort the entries as per objects that are spatially behind (background) followed by objects that are spatially ahead (foreground). For instance object "a garden view" should precede the "table". Make an intelligent guess if possible. Here are some examples:
list of objects: Golden Retriever, white cat,a wooden table,a vase of vibrant flowers, sleek modern television]
text prompt: In a cozy living room, heartwarming scene unfolds. A friendly and affectionate Golden Retriever with soft, golden-furred coat rests contently rug, its warm eyes filled with joy. Nearby, a graceful and elegant white cat stretches leisurely, showcasing its pristine and fluffy fur. A sturdy wooden table with polished edges stands gracefully in the center, adorned with a vase of vibrant flowers adding touch of freshness. On the wall, sleek modern television stands ready provide entertainment. The ambiance is warm, inviting and filled with a sense of companionship and relazation.
plush
output: {a sleek modern television: A realistic photo of a sleek modern television., wooden table: realistic photo of a sturdy wooden table with polished edges., vase of vibrant flowers: A realistic photo of vase of vibrant flowers adding touch of freshness., a Golden Retriever: 'A realistic photo of a friendly and affectionate Golden Retriever with soft, golden-furred coat and its warm eyes filled with joy., white cat: 'A realistic photo of graceful and elegant white cat stretches leisurely, showcasing its pristine and fluffy fur.}
A.5 ADDITIONAL COMPARISONS WITH DEEPFLOYD AND DENSEDIFFUSION
As recommended we provide visual comparison of our method with DeepFloyd Stability Al (2023)
and DenseDiffusion Kim et al. (2033) in Fig. 10 As seen from the figure, DeepFloyd with a strong T5 text encoder struggles to generate coherent images with complex compositional prompts. The same is true for DenseDiffusion. We conclude that our scene blueprint augmented with iterative refinement is necessary for generating coherent images from complex compositional prompts.
We further present quantitative comparison in terms of Prompt Adherence Recall (PAR) (see Sec. Fof main paper) of our approach with all the baselines Table I We also report average inference time for each approach. As seen from the table, DeepFloyd with PAR score of 60% is highly inefficient as it takes around 8 min to generate a 256x256 image from a long textual prompt. We notice that while other approaches are slightly efficient in time, they report lower PAR score, thus rendering them ineffective on complex compositional prompts. Our approach with an average inference time of around 3 min (including blue print generation and iterative refinement process) has the highest PAR
17

---

Published as a conference paper at ICLR 2024
Long Textual Prompt
DeepFloyd
DenseDifuusin
a quiet rural scene, a picturesque to life. stuntty cow with distrnet blackandduhhte maninng grazes serenely, Its tell occasionally awatting
comes
graceful horse stands majestically, ghestnut coat olinting in the sunlight, A loyel dog loungse in the shade of tree, occasionally litting surroundings. This tranquil outdoor moment caplures the essence of harmonious coexistence among these distinct yet complementary creatures.
Nearby,
head survey the
Unraaiate image Missing: dog. horee, black and white patched cow
Massing: dog, harsa cow free
Miksing: Tree
cazy living unfolds. A firenly and affectionate Golden Relriver with snft. golden-fumm coaf rests contently plush nag, its warm eyes filed with joy. Nearby, graceful and elegant white cat stretches Inisurely
heartwarming
shovcasing its pristize and fiufyy wooden table with polished edges stands gracefully the center, adomadd with adding a touch freshness. On the wall, a steek modem feleviaton stands ready entertainment. The ambiance is warm, inviting and filled with a sense of companionship and relaxation.
sturdy
of vibrant flowers
provide
Missing:
Missing: white cat. table
In serene park aree, nature's beauty flourishes. A sprawling grenn lawn stratches Inviting visitora to revel in its vibrancy. Towening trees gravide a soothing canpgy, their branches affering refuge from the sun's warm embrace Amidst this natural ansis, simple hench backons, quiet sanctuery for those soeking resoite, Nearby, quart building blends seamlessly with the park's landscape. adding touch architectural charm. The scene harmonious fusion of lush greenery and tranquil spaces, where chair offers solace, the grass whispers secrets, and the park itself is haven of serenity
Missing: building
Missing: building, sun
In the quist countryalde red farmbouse stands with an old-fashioned charm. Nearby, a weathered nicket fence surrounde garden widffowerss An antiose fractor, though wom, rests as reminder of hard work A scæeerrzw wetches over fields of swaying crops. The air cerries the and hay. against rolling this farmhouse tells story of connection to the land and its traditions.
60
Unrealstic Isti Image Missing: white fence untique tractor, scarecrow
Missing:
In serene park, a gentle breaze creates a tranquil atmosphere.
contented krown cow guazes penoetully in a grassy meadow in the park. Nearby. parked white car waits pationtly, contrasting the natural surroundings with its madernity. An emuty bench beneath the shade af a lowering free offers tranquil spat for reflection. High above, calorfal kite soars groonfully in the seen sky, capturing the carefree spint of the day. This harmonizus blend of nature and human elements makes park perfect place leisurely strolls and moments of relaxation.
Missing: white car, bench. colarful kite
unroalistic Missing: while car, DOW
We provide qualitative comparisons our approach against DeepFloyd and DenseDiffusion. The underlined text in the text prompts represents the objects, their characteristics, and spatial properties. Bascline methods often omit objects and struggle with spatial accuracy (first 2 columns), while our approach (last column) excels in capturing all objects and preserving spatial attributes. The text in black below images (if present) shows the unrealistic nature of image, red text enlists the missing objects in image and pink text refers to location misalignment of the object.
Figure 10:
18

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative comparison of Our approach with baselines in terms of PAR score and average inference time. Our approach has the best PAR score while having relatively decent efficiency in terms of inference time.
Method
PAR score (%) 1
Inference time (min.) +
Stable Diffusion
49
0.18
GLIGEN
57
0.50
LayoutGPT
69
0.83
DenseDiffusion
52
2.50
DeepFloyd
60
8.33
Ours
85
3.16
Text prompt
cazy liing room, heartwarming scena unfolds, A friendlly and aftectioate Golden Retriever with soft, golden-turred cost rests contenty on plush warm filld with joy. Nearbhy the right gracefu and diegant white strotches lesurely ahowsising its pristine and flufly fur A sturdy wooden tabie with ishod cages stands gracchully the adomed with vibrant adding of hessneesa On the warm, invling,
modem talevision
redy provice
companionsship
K=1
K=2
K=3
K=4
K=5
Effect of number of layouts final generated image. We notice that final generated image is coherent in all the cases and aligns well with textual prompt terms object altributes and spatial positions. Note that the position of cat and dog are well defined from the text i.c. cat is on the right of dog.
Figure 11:
score of 85%, thus validating the effectiveness of our approach. Therefore, a discernible trade-off emerges between addressing complexity and ensuring the faithful reproduction of images.
A.6 EFFECT OF NUMBER OF LAYOUTS ON FINAL GENERATED IMAGE
We present an analysis to study the effect of number of layouts on the final generated image in Fig IT Consistent with the findings Li et al. (0226), the layouts generated by LLM (ChatGPT) most of the times align with the textual prompts. The interpolation of multiple layouts (K> 1) produces coherent images preserving the spatial relationships between the objects i.e. dog is always towards
Long Prompt
Layout (K-1)
Generated image
cozy living room, heartwasrming soene unfolds, A friendl and affectionaie Golden Reiries with soft. cotdenefured coat reets contendy on eyes guaczful and resognt ghile stretabes Inisunly abouccaing ihe gustine and fufly fur A sturdy aesddon table with polished adges stands gracntaly the center, adorned adding touch of freshness. On the wal, sibee modem tolaviaion stands ready to provide entertainment. The amblance is warm. inviting, and 1iled sense companionship and relaxation.
aleeet modert Delvessan
plueh rug, its
Nearby,
Wooden table
cullen notilver
yase of wibrant fowers
Effect of Interpolation: While ChatGPT is good at generating layouts from textual prompts, however, in few cases it can generate misaligned layouts (2nd row), leading to images missing certain objects such as cat (last row). We notice that the position of cat and dog are not well defined from the textual prompt.
Figure 12:
19

---

Published as a conference paper at ICLR 2024
Table Effectiveness of ChatGPT in modeling spatial relationships. We observe that ChatGPT perfectly follows the object positions given prompt with clearly defined object positions. While as it shows inherent bias for the prompts with ambiguous object poistions.
Prompt
Object
right
left
arbitrary position
A living room with cat and a dog sitting one on each side
dog
0.3
0.6
0.1
cal
0.7
0.2
0.1
A living room with cat sitting towards right and a dog sitting towards left
dog
1
cat
0
A living room with dog sitting towards right and a sitting towards
dog
cat
the left while cat is the right. However, in extreme cases with only one layout, such as in Fig 12 the ChatGPT can sometimes generate spatially incorrect boxes, such as that for dog and cat, leading to missing objects or incorrect spatial positions of the objects in the final generated image (Fig. of main paper)
A.7 EFFECTIVENESS OF CHATGPT IN MODELING SPATIAL RELATIONSIIIPS
Consistent with the findings of Li et al. 2022b), we observed that proprietary LLMs such as ChatGPT are exceptionally good at following the object positions from the textual prompt. We conducted an analysis in Table 2 with ChatGPT to verify its effectiveness on well-defined and ambiguous prompts (position of some objects is unclear). Specifically we prompted ChatGPT to generate bounding boxes of cat and dog with three different prompts: "A living room with cat and dog sitting on each side", "A living room with a cat sitting towards right and dog sitting towards left", "A living room with a dog sitting towards right and a cat sitting towards left". Our analysis reveals that on an average 60% of the times for the ambiguous prompt "A living room with cat and dog sitting on each side", the chatGPT generates bounding box on the right for the cat and on left for the dog. While for other two unambiguous prompts, the ChatGPT generates correct location of bounding boxes for cat and dog. This shows that ChatGPT works exceptionally well for unambiguous prompts with clearly defined spatial relationships. For the ambiguous prompt, we notice an inherent bias inside ChatGPT, which leads to it generating dog on the left and cat on the right. To account for the errors and to provide a meaningful fix to the inherent bias inside ChatGPT, we provide a hyperparameter r in the interpolation, which can be controlled to adjust for the bounding box of each object (see Fig. in the main paper for a visual illustration of effect of 1) parameter).
A.8 SAMPLING PROCESS WITH GUIDANCE
Kindly note that the guidance function enables diffusion models to be controlled by arbitrary guidance modalities without the need to retrain any specific components (Bansal et al. 2023). Since ours is a training-fre -free approach, therefore, to ensure the objects at each bounding box location follow their corresponding description in the long textual prompt, we employ CLIP-based loss LCLIP at each step & during sampling process (Eq. 5 main paper). During sampling, we first compute the gradient of this loss with respect to the estimated clean sample Tu from Eq. 3f the main paper and denote it as V COLIP. Using this gradient, we update the noise e such that the updated noise at time step t is expressed as,
é c V1 at V2, foLI Based on the ubove-updated noise, the sampling step is given as,
It vI āge
Lfg vār-i
where x l represents the latent variable of the image containing the specific foreground object being refined, This is combined with the unaltered background to get the final output latent shown below,
It - m @ Fig + +(1 - m) O WWWNN(aaaaaa ((Vārzo (1 I.
(8)
where represents the mask for the object being refined. The process is repeated n iterations across T time steps and finally decoded to get the image with refined object. Further please refer to the Algorithm A1 in Appendix Al for full end-to-end pipeline.
20

---

Published as a conference paper at ICLR 2024
1CW
25%
85.7
Figure 13: User study on quality. A majority of users picked our method compared to prior works when presented with 2-AFC task of selecting image with highest fidelity.
A.9 STUDY ON FIDELITY
Details of user study. The subjects for human study were recruited from a pool of academic students with backgrounds in computer science and engincering. To ensure an unbiased evaluation and maintain objectivity, the subjects participating in the evaluation remained anonymous with no conflicts of interest with the authors. Furthermore, we prioritized user confidentiality by refraining from collecting personal details such as names, addresses etc. The entire survey process was anonymized, ensuring that there was no leakage of data to any user. Following a 2-AFC (two- alternative forced choice) design, each participant was tasked with selecting one image out of pair of images, having highest fidelity. Each pair of images was randomly sampled from a pool containing images from baselines well
approach. This system mitigates biases which from user-curated surveys. Further, the pool of images was shuffled before sampling each time. The system was further equipped with the feature of randomly flipping the position of the images, to remove the user-specific position bias in case any pair of images is repeated. Additionally, 2-second delay between questions was introduced facilitate more thoughtful decision-making.
originate
Following above procedure, we present an extensive user study in Fig. 13 on the fidelity of generated images. For the user study, we compared our method with Stable Diffusion, GLIGEN, LayoutGPT and LLM Grounded Diffusion. The system yielded approximately 90 responses from 15 subjects, with each user answering an average of questions. From the Fig. [3 it's clear that our approach compares favorably against baselines in terms of image fidelity while maintaining highest alignment with the text.
21


Reviews:
[{"review_id": "GSQu5jx5YE", "reviewer": "Reviewer_XXwY", "summary": "The paper extends recent works which leverages layouts to generate scenes corresponding to complex text-prompts. This work first shows that for complex prompts, existing layout to image generation methods have certain failure modes and proposes some practical modifications which are augmented with existing layout to scene generation methods.  First, the authors propose a scene blueprint to represent complex text-prompts; Secondly the authors design an iterative refinement process which improves the alignment of generated images with the complex text-prompts.", "questions": "See Cons/Questions;\nOverall, the paper is practical, but the various components though intuitive are not technically novel. While I do agree that not everything in a paper needs to be novel — the authors should provide solid justifications on the design of each component.  \n\nI am happy to revisit my scores after the rebuttal!", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The research question is a very practical problem — usually most of the text-to-image generation models are not good at coherent images corresponding to complex prompts, so providing a solution for it is important.\n- The method consists of various components (a lot of these components are existing though) and is conceptually intuitive!\n- The framework obtains strong results on human-study for fidelity of images generated for long prompts.", "weaknesses": "Cons / Questions\n\n- While the writing is satisfactory, it can still be improved! The authors should provide more information in the paper on how Eq. (5) is used to guide the sampling process.\n- Can the authors provide more intuition on the interpolation step? \n- Given that there are stronger open-source diffusion models (e.g., DeepFloyd) — the authors should provide some context on how long prompts work in those cases, as they use a stronger text-encoder like T5. \n- While the authors comment that the size of the tokens (77 in CLIP) is one of the potential reasons on why SD cannot generate compositional prompts — I believe this is only partially true. Even for non-complex compositional prompts, SD is not able to generate coherent images. Can the authors comment in general on some potential reasons why these text-to-image models are not able to generate images corresponding to simple compositional or complex prompts? I think both are related somehow and it will be beneficial to provide some context regarding it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CiOJLmPhWz", "reviewer": "Reviewer_c4Qg", "summary": "This paper presents a novel approach for improving text-to-image generation in diffusion-based models when processing complex scenes with multiple objects and intricate text prompts. The authors leverage Large Language Models (LLMs) to extract the layout information, detailed text descriptions, and background information from text prompts. Then the proposed layout-to-image generation model is composed of two stages: Global Scene Generation and Iterative Refinement Scheme. The Global Scene Generation phase uses object layouts and background context to create an initial scene which roughly represents the target image layout but not very accurate. Then the Iterative Refinement Scheme iteratively evaluates and refines box-level content to align them with their textual descriptions and recompose objects to ensure consistency. Extensive experiments are conducted to validate the effectiveness of the proposed approach.", "questions": "Please refer to the weakness section.\n\nPost-rebuttal: I have read the author response and other reviewers' comments. I decide to keep my initial rating unchanged, but I won't fight for acceptance of this paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach can handle the complex scenarios of text-to-image generation with long text prompts very well.\n2. The paper is well-written and easy to follow.\n3. The iterative refinement loop provides a possible solution to generating images of complex scenes.", "weaknesses": "1. I think the major limitation of the proposed approach is efficiency. The complexity of the proposed approach increases as the number of objects increases for complex scenes. This might also be the major issue presenting this approach to be applied in real usage.\n2. It seems that although the generation of objects can be iteratively refined, the bounding box locations cannot be refined. If the LLM predicts unreasonable bounding box layouts at the first stage, it cannot be corrected. Have the authors think of introducing the refinement of bounding box locations into the pipeline?\n3. How many layouts are needed to interpolate? How does the number of layouts for interpolation affect the results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fsv6fsKfEV", "reviewer": "Reviewer_Br9r", "summary": "This work aims to solve the problem that prior text-to-image models cannot accurately follow the object specifications in lengthy prompts. The work proposes a novel two-step pipeline that first uses a scene blueprint, which is an LLM-generated layout with object descriptions, to generate the overall image. A CLIP-based guidance is then applied to perform iterative refinement in order to make sure the content of each box is correct. The method enables accurate and diverse image generation with intricate and lengthy input text prompts. The user study and qualitative comparison indicate a non-trivial improvement over baseline methods.", "questions": "The authors are encouraged to respond to and address the weaknesses in the section above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. In addition to previous methods using the layouts for initial image generation, the method further proposes box-based refinement to improve the ability to generate all the objects mentioned in a lengthy and intricate prompt.\n2. The method points out the fact that there currently lacks a pipeline for benchmarking text-to-image methods with lengthy prompts and proposes a metric called prompt adherence recall (PAR) to evaluate their method and several baselines.\n3. Their method has better prompt adherence compared to baselines in prompt generation. The user study also confirms that the method can faithfully generate objects in the prompt.", "weaknesses": "1. This work builds on LLM-grounded Diffusion, as mentioned in Sec 3.3. However, the difference between LLM-grounded Diffusion and the current work is not clearly explained. This work lists \"scene blueprints using LLMs\" as one of the contributions, but both LLM-grounded Diffusion and LayoutGPT (with both works cited in related work section) propose using LLM to generate the scene layouts with object descriptions. The authors show that the scene representation enhanced by the proposed method works better for long text prompts, but this claim of \"proposing a structured scene representation\" is not a contribution as it has been proposed and used in previous works.\n2. The interpolation of k generated layouts could potentially cause degenerate or undesired results. For example, for \"a cat and a dog, one on each side\", the LLM can generate 1) a cat on the left, a dog on the right, or alternatively 2) reverse the position for the cat and the dog. The interpolation of layout 1) and 2) results in two objects placed both at the center.\n3. Missing details: In Sec 3.3, the authors mention \"image-to-image translation\" with latent/stable diffusion that leads to generation with higher quality, but no details such as the prompts or the exact approach are given. Is this simply adding some noise and denoise?\n4. The evaluation of the work does not investigate whether the proposed method degrades the fidelity of the image. The user study only involves asking whether objects are present and layouts are accurately generated rather than comparing the overall image quality. The author needs to present evaluation results that show their method does not have significant degradations on quality. If only the presence of objects is considered, a simple \"copy-and-paste\" from baseline text-to-image model with individual object generation using the generated layout would also result in high object presence without bringing much utility to the research community and potential applications.\n\nA typo that does not affect the rating:\n1. api -> API (page 7)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lp4QnFFDqI", "reviewer": "Reviewer_kt1C", "summary": "This proposed a framework for generating images from complex and detailed prompts, which took more work for previous models or frameworks. They utilize LLM's capability to generate augmented textual and visual prompts for better generations. And the framework comprises global scene generation and an iterative refinement scheme to align with conditional cues.", "questions": "- Minors:\n  - In Sec. 3.1, Models. -> Models (Please exclude the period in the section title.)\n  - In Fig. 2, it would be inappropriate to include the logo of a commercial product (ChatGPT from OpenAI) in an academic paper. And the company may not allow the usage of their logo to promote the work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This work steps toward longer textual descriptions for image generations to ensure the fidelity of complex textual prompts. \n\n- Scene blueprints with the iterative refinement step ensure high prompt adherence recall, quantitatively validating its effectiveness.", "weaknesses": "- Missing information on human study. Detailed information on human study could be provided to assess the reliability of the outcome. How do you recruit and select them based on what qualification? Isn't there any conflict of interest for the subjects and authors? How many subjects are recruited? What was the confidence of the votes? This issue is the major reason for leaning toward rejection. I am eager to see the author's feedback for reassessment.\n\n- This framework relies on recently proposed models like a strong LLM, CLIP, and an image composition model. The collection of previous works provides shallow techniques compared with them.\n\n- A missing related work. DenseDiffusion [1] may be worth being included in the layout-to-image generation subsection in Sec. 2 and the box-level multi-modal guidance in Sec. 3.4. DenseDiffusion tried to manipulate attentional weights to control the regions for layout guidance selectively. It would be appreciated if you could compare your method with it for readers in the upcoming revised manuscript. Note that, due to the narrow accessibility to this work at the time of submission, this is not considered in the score evaluation.\n\n[1] Kim, Y. et al. (2023). Dense Text-to-Image Generation with Attention Modulation. http://arxiv.org/abs/2308.12964", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)