LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
Authors: Long Lian
Boyi Li
Adam Yala
Trevor Darrell
Venue: Accepted by TMLR ()

Abstract: Recent advancements in text-to-image diffusion models have yielded impressive results in generating realistic and diverse images. However, these models still struggle with complex prompts, such as those that involve numeracy and spatial reasoning. This work proposes to enhance prompt understanding capabilities in diffusion models. Our method leverages a pretrained large language model (LLM) for grounded generation in a novel two-stage process. In the first stage, the LLM generates a scene layout that comprises captioned bounding boxes from a given prompt describing the desired image. In the second stage, a novel controller guides an off-the-shelf diffusion model for layout-grounded image generation. Both stages utilize existing pretrained models without additional model parameter optimization. Our method significantly outperforms the base diffusion model and several strong baselines in accurately generating images according to prompts that require various capabilities, doubling the generation accuracy across four tasks on average. Furthermore, our method enables instruction-based multi-round scene specification and can handle prompts in languages not supported by the underlying diffusion model. We anticipate that our method will unleash users' creativity by accurately following more complex prompts. Our code, demo, and benchmark are available at: https://llm-grounded-diffusion.github.io

Published in Transactions on Machine Learning Research (02/2024)
LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-lmage Diffusion Models with Large Language Models
Long Lian UC Berkeley
longlian@berkeley.com edu
Boyi Li UC Berkeley
boyili@berkeley.. edu
Adam Yala UC Berkeley, UCSP
yala@berkeley. edu
Trevor Darrell UC Berkeley
bresordarrelll@eerkkee. edu
Reviewed on OpenReview:
Abstract
Recent advancements text-to-image diffusion models have yiclded impressive results in generating realistic and diverse images. However, these models still struggle with complex prompts, such those that involve numeracy and spatial reasoning. This work proposes to enhance prompt understanding capabilities in diffusion models. Our method leverages pretrained large language model (LLM) for grounded generation in novel two-stage process. In the first stage, the LLM generates a scene layout that comprises captioned bounding boxes from a given prompt describing the desired image. In the second stage, novel controller guides an off- tttthh-sshef diffusion model for layout-grounded image generation. Both stages utilize existing pretrained models without additional model parameter optimization. Our method significantly outperforms the base diffusion model and several strong baselines in accurately generating images according to prompts that require various capabilities, doubling the generation accuracy across four tasks on average. Furthermore, our method enables instruction-based multi-round scene specification and can handle prompts in languages not supported by the underlying diffusion model. We anticipate that our mothod will unleash users' creativity by accurately following more complex prompts. Our code, demo, and benchmark are available https://lim-rrouddddddfffff tiw.bb..
1 Introduction
The field of text-to-image generation has witnessed significant advancements, particularly with the emergence of diffusion models. These models have showcased remarkable capabilities in generating realistie and diverse images in response to textual prompts. However, despite the impressive results, diffusion models often struggle to accurately follow complex prompts that require specific capabilities to understand. Fig. 1 shows that Stable Diffusion (Rombach et al. 2022), even the latest SDXL (Podell et al. 2023), often could not generate a certain number of objects or undersland negation in the prompi. It. also struggles with spatial reasoning or associating attributes correctly with objects.
One potential solution to address this issue of course to gather a comprehensive multi-modal dataset comprising intricate captions and train text-to-image diffusion model for enhanced prompt understanding.

---

Published in Transactions on Machine Learning Research (02/2024)
Stable Diffusion XL (b) LMD (Ours)
Stable Diffusion XL (b) LMD (Ours)
Attribute Binding
Negation
man in red next to another woman blue
wooden table without bananas
Spatial Relationships
Generative Numeracy
A watercolor painting cats on the grass
3 apples arranged In L-shape on a wooden table
8
Figure 1: (a) Text-lo-image diffuusin models such SDXL (Podell al. 2023 often struggles accurately
follow prompts that involve negation, numeracy, attribute binding, spatial relationships. (b) Our method LMD achieves enhanced prompt understanding capabilities and accurately follows these types of prompts.
realistic photo of gray cat and an orange dog on the grass
A realistic photo of grassy outdoor scene
a gray cat
en orange dog
LLM Layout Generator
Layout-grounded Stable Diffusion
grass
Stage
Stage 2
Our proposed LMD enhances prompt understanding in text-to-image diffusion models through a novel two-stage generation process: 1) An LLM layout generator takes a prompt from the user and outputs an image layout the form of captioned bounding boxes. 2) A stable diffusion model guided by our layout-grounded controller generates the final image. Both stages utilize frozen pretrained models, which makes our method applicable to off-the-shell LLMs and other diffusson models without grounding in their training objectives.
Figure 2:
Nonetheless, this approach presents notable drawbacks. It requires considerable time and resources to curate diverse and high-quality multi-modal dataset, not mention the challenges associated with training finc-tuning diffusion model on such extensive data.
In contrast, we propose a novel training-free method that equips the diffusion model with an LLM that provides grounding for enhanced prompt understanding, Our method LLM-grounded Diffusion (LMD) consists of a two-stage generation process as shown in Fig. 2
In the first stage of our method, we adapt an LLM to be a text-grounded layout generator through in-context. learning. Given prompt describing the desired image, the LLM generates scene layouts in the form of captioned bounding boxes, with background caption and negative prompt for what to avoid in generation.
In the second stage, we introduce novel controller that guides an existing diffusion model without grounding in its training objective (e.g., Stable Diffusion) to follow the layout grounding generated in the first stage. In contrast to previous and concurrent works on region control (c.g., Bar- Tal et al. (2023 Chen et al. (2023): Xie et al. (2023)) that apply semandic control to certain spatial regions, our approach allows precise control over object instances designated regions.
Notably, both stages utilize frozen pretrained models off-the-shelf, making method applicable to LLMs and diffusion models trained independently without any LLM diffusion model parameter optimization.
In addition to enhanced prompt understanding, our method also naturally enables instruction-based scene specification with multiple rounds user requests (Fig. 3) and image generation from prompts in languages not supported by the base diffusion model (Fig. I.1 without additional training.
Shown Fig. LMD provides a unified solution several caveats in prompt understanding at once and enables accurate and high-quality image generation from complex prompts. We demonstrate that diffusion
2

---

Published in Transactions on Machine Learning Research (02/2024)
Instruction-based Scene Specification
LMD (Ours)
LMD (Ours)
A realistic photo of a scene with red car on the left.
Replace the car on the right with a yellow car
Initial user request
8
Followup request 2
LMD (Ours)
LMD (Ours)
Remove the yellow car. Instead, place flowers at that place
Add a blue car to the right
&
Followup request
8
Followup request 3
Figure 3: LMD naturally enables instruction-based multi-round scene specification and is able to adapt subsequent rounds of generation according to users' followup instructions and clarifications.
model grounded with L.L--genrratd layouts outperforms its base diffusion model and several recent basclines, doubling the average generation accuracy across four tasks. Our primary contributions include:
1. We propose training-free two-stage generation pipeline that introduces LLMs to improve the prompt understanding ability of text-to-image diffusion models.
2. We introduce layout-grounded Stable Diffusion, a novel controller that steers an off-tho-shelf diffusion model to generate images grounded on instance-level box layouts from the LLM.
3. LMD enables instruction-bast based scene specification and allows broader language support in the prompts.
4. We propose a benchmark to assess the prompt understanding ability of a text-to-image model and demonstrate the superior performance of LMD over recent baselines.
We expect LMD to empower users with more precise control of text-to-image diffusson models. Our codc. demo, and benchmark are publicly available
2 Related Work
Text-to-image diffusion models. High-quality image generation from lextual descriptions with diffusion models has been popular recently (Ramesh al. [202 Saharia al.. 2022 Rombach et al. 2022 Podell et al.) 2023 Despite the improssive visual quality, these models still tend to exhibit unsatisfactory performance when it comes to complex prompts that involve skills such as binding attributes to objects and spatial reasoning (Ramesh et al. 2022).
LLMs for visual grounding. Many multi-modal models benefit from integrating LLMs for grounding vision models. BLIP-2 (Li et al. 2023a bootstraps vision-language pre-training from a frozen image encoder and an LLM. Flamingo (Alayrac al. 2022) tackles tasks such as few-shot visual question-answering and captioning tasks. Gupta al. (2021) uses Transformer (Vaswani et al. 2017) for layout prediction but focuses generating layouts for limited closed set of object classes in the annotated training set and thus is not able to generate layouts for objects not in the training set. Wu et al. (2023) and Koh et al. (2023) also involve LLMs in conditional image generation. However, these methods still rely on CLIP text embeddings to convey the information to the diffusion model. Therefore, they often exhibit insufficient control compared to our method, which explicitly asks the LLM to reason about the spatial composition of different objects and poses direct spatial control. Concurrent to our work, LayoutGPT (Feng et al. (023) proposes prompting an LLM for layout generation in a CSS structure. While LayoutGPT depends on a dataset annotated with boxes and captions to ritveeve relevant in-context examples for the LLM, our method remonstrates that the ability for
3

---

Published in Transactions on Machine Learning Research (02/2024)
realistic photo of grassy outdoor saage
A realistic photo of a gray car and an orange dog on the
Your tadi is 30 getente the bounding boxes the otgects monooted the coption, along with fockground peoepe dosentiing scone.
an orange
Captoo: reelistic photo gry
gray
dog
Embed Prompt into Template
LLM
Objecte pray
orangg
Completion
dog',
200j).
Parsing
(Iesomtext ccmmlee)]
('rras", [R,
17211]
Template:: You bourding
Caption Avealiatic ploto a sray and crange gnos
Negative prompe:
Backyroand promptt phato
outdoue
grase
Ohjaze:
In stage 1, LMD generates an image layout from a user prompt. LMD embeds the user prompt into a template with instructions and in-context examples. An LLM is then queried for completion. Finally, the LLM completion is parsed to obtain a set of captioned bounding boxes, background caption, and an optional negative prompt.
Figure 4:
generating high-quality layouts is already present pretrained LLM weights and can be prompted with a fixed set of in-context examples without external annotations.
Spatially-condtiionedd image generation methods. These methods create images based given priors such as poses, segmentation maps, strokes, and layouts. Prior to the popularity of diffusion models,
SPADE Park et al. 2019 BlobGAN [pstein et al., 2022 and Layout2lm (Zhao et al. 2019) synthesize photorealistic images from a given layout. Xu et al. (2017); Johnson et al. (2018); Herzig et al.
(0020)
generate images with scene graphs. ControlNet (Zhang & Agrawala, 2023), SpaText (Avrahami et al. 2023
LifoutDiffuse (Cheng al. 2023), LayoutDiffusion,, (Zheng
2023), GLIGEN (Li et al.) 2023b and
ReCo (Yang et al.] 2023) propose training-based based adaptation the diffusion models for spatially-condition
image generation, with Li ct al. (203bb) and Yang et al. (2023) supporting open-vocabulary labels for layout boxes. However, these methods rely annotaled external dalasels such as COCO (Lin et al.) 2014] to supply images with annotations such as boxes and captions. Furthermore, training- baining-based adaptation makes the model incompatible to add-ons such as LoRA weights (Hu et al., 2021) and renders it difficult to train a new LoRA model from a training set without box annotations. In contrast, we propose a training-free free generation controller that steers existing lext-to-image diffusion models that are not specifically trained for layout-grounded image generation and does not require external datasets. Furthermore, our method can also integrate with training-based methods for further improvements.
Very recently. Bar-Tal et al. (023) Chen et al. 2023 Xie et al. 2023 allow training-free region control in image generation and share a similar task formulation to our layout-to-image stage. However, these works ground the image generation on the region semantics and pose little control over the number of object instances inside each semantic region, whereas our method focuses on grounding generation on inslances.
Similar to our instruction-basee scene specification, Brooks et al. (0023) recently proposed instruction- image editing. Wu et al. (2023) and Gupta & Kembhavi (0023) also allow using external image editing models
in an LLM-driven dialog. Different from these methods, we aim to edit the scene layout rather than the
image pizels, which casily allows support for a greater set of instructions such as swapping/ swapping/moving moving objects.
3 LLM-grounded Diffusion
In this section, we introduce method A..--rouudded Diffusion (LMD). LMD focuses on the text-to-image generation setting, which involves generating image Xe given text prompt Our method generates an image in two stages: text-grounded layout generation (Section 3.1 b and layout-grounded image generation (Section 3.2) The layout-to-image stage of our method LMD builds upon the latent diffusion framework (Rombach ct al. 2022 for which we refer readers to Appendix A for preliminaries.
3.1
LLM-based Layout Generation
To generate the layout of an image, our method embeds the input text prompt y into a template and queries an LLM for completion (Fig.
Layout representation. LMD's layout representation comprises two components: 1) a captioned bounding box for each foreground object, with coordinates specified in the (s, y, width, height) format, and 2) simple
4

---

Published in Transactions on Machine Learning Research (02/2024)
and concise caption describing the image background along with an optional negative prompt indicating what should not appear in a generated image. The negative prompt is an empty string when the layout does not. impose restrictions on what should not appear.
Instructions. Our text instructions to the LLM consist of two parts: 1. Task specification:
Your lask is to generale the bounding bozes for the objects mentioned in the caption, along with background prompt describing the scene.
2. Supporting dctails:
The images are of size 512x512... Each bounding box should in the format of o.... If needed, you can make reasonable guesses.
In-context learning. Similar to Brooks et al. (2023) we provide the LLM with mamually curated examples after the task description. Through these examples, we clarify the layout representation and provide preferences lo disperse ambiguity. An example is shown as follows:
Caption: A watercolor painting of a wooden table in the living room with an apple on it Objects: (('a wooden table', [65, 243, 944, 206)), ('un opple', (206, 306, 81, 69))]
Background prompt: watercolor painting of living room Negative prompt:
To ensure precise layout control, we adhere to two key principles in our example design: 1) Each object instance is represented by single bounding box. For instance, if the prompt mentions four apples, we include four boxes with "an apple" in each caption. 2) We leave no foreground objects specified in the boxes to the background caption to ensure all foreground objects are controlled by layout-grounded image generator (Section 3.2. These principles allow for accurate and instance-controlled layout generation.
LLM completion. After providing the in-context examples, we query the LLM for completion:
Coption: fireput prompt. from the user] Objects: (start of LLM completion)
The resulting layout from the LLM completion then parsed and used for the subsequent image generation process. We refer readers to the Appendix K for our complete prompt.
3.2
Stable Diffusion
In this stage, we introduce a controller to ground the image generation on the LLM-generated layout. While previous training-free region control methods (Bar-Tal et al. 2023 Chen et al.) 2023 Xie et al., 2023) apply semantic guidance through regional denoising or attention manipulation, these methods lack the ability to control the number of objects within semantic region. This limitation arises as the different instances are often indistinguishable in either the latent space or the attention map, hindering instance-level control.
In contrast, LMD enables instance-level grounding by first generating masked latents for each individual bounding box and then composing the masked latents as priors to guide the overall image generation. This allows for precise placement. and attribute binding for each object instance.
Per-box masked latents. While diffusion models lack inherent instance-level distinction in their latent space attention maps for fine-gran fine-grained control, we observe that they are often able to generate images with one specified instance. Hence, we process one foreground box at a time for instance-level grounding.
As depicted in Fig, 5 a), for each foreground object i, we first generate an image with a single instance by denoising from Zpp' to ( where 2!") refers the latents of object i at denoising timestep t In this denoising process, use "fbackground prompt/ with fboz caption/" (c.g., "a realistic image of indoor scene with gray cat")) as the texi prompt for denoising. The initial noise latent, is shared for all boxes to ensure globally coherent viewpoint, style, and lighting (i.e., 2f = zT, V i).
We refer readers Appendix A for definitions the terms, such as latents, that introduced the latent diffusion framework.
5

---

Published in Transactions on Machine Learning Research (02/2024)
Box for "a gray cat"
Encouraged/dlsccorraged from attending to "with gray cat"
Attention control:
Per-box Cross- attention Map A
Diuusion
Cross-a attention map of the cat token
Denoise Latentt
Denoise Latents
(ear)
loar)
(eor
Per-box prompt:
Optional SAM Refinement
A realistic photo with a gray cat
Masked Latents "a gray
Masking
Foreground Mask
(a) Step I: per-box masked latent generation
Masked Latents Each Box
Masked Latents Each Box
Q(cur)
S(dog)
Compose
Compose
Decode pixels from latents
Per-box Cross- attention Maps
S:300 Stftu
Denoise
Dencice
(compp
(comp)
A
A (dog)
Attention transfer
Generated Image
(b) Step 2: overall image generation with masked latents as priors
Figure 5: In stage 2, introduce novel layout-grounded controller that guides stable diffusion to generate images based on the layout obtained from the previous stage. Our layout-grounded image generation process consists of two steps: (a) generating masked latents for each box specified in the layout, with attention control ensuring that the object is placed in the designated box; and (b) composing the masked latents as priors to guide the image generation to adhere to the specified layout.
To ensure the object aligns with the bounding box, we manipulate the cross-attention maps 60 of the voise-prediction network Each map describes the affinity from pixels to text tokens:
(4) Sof (max(q."k,)
(1)
where qu and k, lincarly transformed image feature at spatial location and text fcature at token index in the prompt, respectively.
Following Chen et al. (2023); Xie et al. 2023), we strengthen the cross-attention from pixels inside the box to tokens associated with the box caption while attemnating the cross-attention from pixels outside the box. To achieve this, we define a simple energy function:
((A i, v) = Topk,, (A.. bb9) wTopk,, (1-b(1)) where is element-wise multiplication
(2)
((9) is a reclangular binary mask of the box with the region in the box set to 1, Topk, takes the average of top-k values across the spatial dimension u, and w = 4.0. The energy function minimized by updating the latent before each denoising step:
z
- 2
V
Σ
E(A (1) v),
(3)
VEV
(4) where y is the guidance strength; the set. Vg contains the token indices for the box caption in the prompt for box (e.g., while generating the masked latents for a box with caption "a gray cat". V; indicates the indices of tokens that correspond to the box caption in the per-box denoising text prompt "/background prompt) with a groy cat") Denoise(.) denotes one denoising step in the latent diffusion framework.
2-11 - (en
(4)
TThe cross-attention layer index omitted simplicity. We sum energy values selected layers during optimization.

---

Published in Transactions on Machine Learning Research (02/2024)
(a) A top down on cnjty wooden sable apple the left
Swap the apple pear
make apple grean
the right
fruits larger
pear
apple the center
the rod again
Figure 6: LMD and LMD+ support instruction-basee scene specification, empowering the users to add/move remove objects, modify object attributes, and clarify the prompt in multiple rounds of dialog. (a): the initial prompt for the scene; (b)-(i): eight subsequent instructions that sequentially modify the scene. By separating the generation of each foreground object as well as the background, LMD ensures consistent image generation when the same seed used for image generation throughout the dialog.
After generation, we obtain the cross-attention map that corresponds to the box caption, which serves as a saliency mask for the object. We optionally use SAM (Kirillov et al. 2023) to refine the quality of the mask. This can be done by querying either with the pixel location that has the highest saliency or with the layout box. The functionality of SAM can also be replaced by a simple thresholding, as experimented in Section 4.3 With the refined mask for exactly one foreground instance, denoted m(), we perform element-wist multiplication between the mask and the latent at each denoising step to create sequence of masked instance latents (2) yT 0=
(5) Masked latents as priors for instance-level control. The masked instance latents TT are then leveraged to provide instance-level hints to the diffusion model for the overall image generation. As illustrated in Fig. b)),, during each denoising time step in the early denoising process, we place each masked foreground (comp),
2
m (4)
latents 2 onto the composed latents z,
(comp) LatentCoopose) (comp) 2
Vi initialized from 2r for foreground generation for consistency, and (cnmp) 2()) m (1) simply puts the masked foreground latents ( onto the corresponding
where ZT
(comp) is
LatentCompose(zz
location on z
(comp)
Since diffusion models tend to generate the object placement in the initial denoising steps and then object details in later steps (Bar-Tal et al. 2023), only compose the latents from timestep T to r where E [0. balances instance control and image coherency. By primarily intervening during the steps for object placement, our method merely provides instance-level layout. hints rather than forcing each masked region of the resulting generation to look the same as the per-box generation.
To make our guidance more robust, we further transfer the cross-attention maps from per-box generation the corresponding regions in the composed generation by adapting the energy function:
E [coop) (A (comp) A (6) v. = E(A (comp), v, + Σ
A (comp) - AA(0)
(7)
uv
WEV!
where À = 2.0 and the energy value of each box is summed up for optimization. V'' denotes the indices of tokens that correspond to the box caption in the text prompt for the overall denoising process, similar to the definition of V; in Eq. (3)_
In this way, our controller conditions the diffusion model to generate one instance at each masked location, with the final generation natural and coherent in terms of foreground-backkgrumn composition.
Finally, we decode latents 2 (comp) to pixels x via the diffusion image decoder. We refer readers to Appendix B for the overall pseudo-code for layout grounding.
Integration with training-basse based methods. Our training-free controller also be applied along with training- based methods such as GLIGEN (Li et al. 2023b) to leverage instance-amnotated external datasets when available. Since GLIGEN trains adapter layers taking box inputs, the integration with GLIGEN, denoted as LMD+, involves adopting its adapter weights and passing the layout guidance to the adapter
IIn the notation of this work, the denoising process starts from time step step
7

---

Published in Transactions on Machine Learning Research (02/2024)
Stable Diffission XL
LMD
LMD (Ours)
Stable Diffusion XL
LMD Generated Layoul
LMD (Ours)
Generated Layout
citto illustration of four fruits a 2x2 grid: an apple, orange, mango, strawberry
A realstic of white deer and bear an factory
septy fertary
scene
A
A watercolor painting of a scene on moon with a horse on top riding an astronaut
reafistic image plastic balls in a room
oe lavel
Figure 7: LMD outperforms its base text-to- text-to-image diffusion model Podell et al. (2023 in accurately following the prompts that require spatial and language reasoning. Best viewed when zoomed in.
layers. Note that LMD+ uses adapters along with the instance-level guidance introduced above, which greatly surpasses only using GLIGEN adapters, as shown in Table 2 We achieve further enhanced instance and attribute control without additional training through this integration.
3.3 Additional Capabilities of LMD
Our L.MM-rounded generation pipeline allows for two additional capabilities without additional training.
Instruction-based scene specification. Leveraging an LLM that supports multi-round dialog (e-g., GPT-3.5/44) LMD empowers the users to specify the desired image with multiple instructions following an initial prompt (Fig- 3h. Specifically, after the initial image generation, user can simply give clarifications or additional requests to the LLM. With the updated layout from the LLM, we can leverage LMD again to generate images with the updated layout. Updating the layout rather than the raw image gives LMD several advantages, as demonstrated in Fig. 6 1) Our generation remains consistent after multiple rounds of requests instcad of gradually drifting away from the intial image. 2) LMD can handle requests that involve spatial reasoning, which are the limitations of previous instruction-based image editing method Brooks et al. (2023). In contrast, we demonstrate that VisualChatGIT Wu et al. (023 which equips ChatGPT with tools such
as Brooks et al.] (2023) is not able to follow the instructions in Fig Fig.
G. especially for spatial instructions over
multiple iterations of dialog. We refer interested readers to Appendix
G for the comparison. This capability
applies to both LMD and LMD+. We also show additional use cases in Fig. C.1 in Appendix C Our LMD
can handle requests for open-ended scene adjustments, offer suggestions for the current scene, understand user requests within the dialog context, and allow the users to try out different detailed adjustments while preserving the overall image style and layout, facilitating fine-grained content creation.
Supporting more languages. By giving in-content example of non-English user prompt and an English layout output the LLM layout generator accepts non-Euglish user prompls and outputs layouts with English captions. This allows generation from prompts in languages not supported by the underlying diffusion model without additional training (Fig. I.1 We refer readers to Appendix [ for additional details.
4
Evaluation
4.1 Qualitative Comparison
Setup. We qualitatively compare our approach with Stable Diffusion (SD, Rombach et al. (2022) Podell et al. (023)) SD family is also chosen as our underlying base model for layout-grounded image generation given its strong capabilities and widespread adoption in text-to-image generation research. Thanks to the training-free nature our work, our method is applicable to various diffusson models without additional training. Therefore, for Fig. 77 and we use the largest Stable Diffusion model SDXL as the base model of LMD and compare against SDXL bascline (sce Appendix H for details). For all other settings, we
We simply translate the example prompt input our last example to the desired language, while keeping its layout English.
8

---

Published in Transactions on Machine Learning Research (02/2024)
VisualCharGPP (Baseline)
GILL (Baseline)
Correct?
VisualChatPPT (Baseline)
GILL
Correct?
(Baseline)
Attribute Binding
Negation
a man red standing next to another woman in blue
wooden fable without bananas
Spatial Reiationships
Numeracy A watercolor painting cats on the grass
3 apples arranged in an L-shape
wooden
8
Figure 8: We qualitatively compare with VisualChatGPT (Wu et al. 2023 and GILL Koh et al. 2023) that also leverage LLMs in the image generation pipelines. Both baselines lack the ability to accurately follow the prompts for three out of four tasks that our method can solve in Fig. and F.1
Accuracy
Tasks
SD
LMD
LMD+
Negation
28%
100%
(3.6x)
100% (3.6x)
Generative Numeracy
39%
62%
(1.6x)
86% (2.2x)
Attribute Binding
52%
65%
(1.3x)
69% (1.3x)
Spatial Relationships
28%
79% (2.8x)
67% (2.4x)
Average
37%
77% (2.1x)
81% (2.2x)
Table 1: With guidance from the LLM-based layout generator and our novel layout-grounded controller, our LMD significantly outperforms the Stable Diffusion model (SD) that we use under the hood four tasks benchmarking prompt-following abilities. LMD denotes our method directly applied on SD. LMD denotes additionally integrating pretrained GLIGEN (Li al. 2023b adapters into our controller.
use Stable Diffusion v1.5 as the base model unless stated othorwise. We use gpt-4 (OpenAl 2023 for layout generation for all qualitative comparisons. Results. Fig. [ and 7 we observe that our two-stage text-to-image generation approach greatly enhances prompt following ability compared to our base model by generating images that align with the layouts from the LLM.
Comparing with other LLM-based image generators. VisualChatGPT Wu et al. 2023 and GILL (Koh et al. 2023) also leverage LLMs as a part of the image generation pipelines. Both works leverage SD as the underlying image generation model. VisualChatGPP treats SD as a module that can be used by the LLM and passes text caption to it, and GILL outputs a embedding in place of the text embedding for SD. Since both methods utilize LLMs only provide conditions to SD the form of text embeddings, these methods still inherit the problems of insufficient control of text embeddings from the base SD model. In contrast, method asks the LLM to explicitly reason about the spatial relationships and applies direct spatial control on underlying diffusion model, thereby bypassing the bottleneck of the text embedding representation that docs not accurately convey spatial information. As shown in Fig. S neither method accurately follows lext prompls of several categories that our mot is able to correctly generate in Fig. and Fig. F.I in Appendix [F) Furthermore, although the involvement of LLM in VisualChatGPTT and GILL also potentially allows multi-round instruction-based scene specification (Section 3.3). we empirically obscrve that the generated images quickly deviate from the scene of "a wooden table " starting from the second iteration in Fig. G.1 in Appendix G with the final generation being incomprehensible.

---

Published in Transactions on Machine Learning Research (02/2024)
Accuracy
Stage 1/Stage 2
Negation
Numeracy
Attribute
Spatial
Average
Training-fie freee methods:
IMD/MultiDifnssio (Bar- Tal et al.
2023
100%
30%
42%
36%
52.0%
LMD/Backwar BBckward Guidance
Chen et
2023
100%
42%
36%
61%
59.8%
.MD/BoxDiff (ie et al. 2023
100%
32%
55%
62%
62.3%
LMD/LMD (Ours)
100%
62%
65%
79%
76.5%
(+ 14.2)
Training-base based methods:
LMD/GLIGEN Li et 2023b
100%
57%
57%
45%
64.8%
LMD AMD/LMD+ (Ours)
100%
86%
69%
67%
80.5%
(+ 15.7)
LMD/ LMD/LMD+ (Ours, GPT-4)
100%
84%
79%
82%
86.3%
(+ 21.5)
Evaluating generated layouts only (upper bound for image generation):
LMD/-
100%
97%
100%
99%
99.0%
Table 2: Ablations on layout-to-image methods as stage with our LLM layout generator as stage Our proposed layout-grounded controller performs the best among them. Our controller could also be applied on top of training- based GLIGEN (Li et al., 2023b), denoted as LMD+, for additional improvements. Finally, the LLM-ggnnrated layouts almost always align with the prompt, highlighting that the bottleneck is the layout-grounded image generation. The scores for negation task are high because we pass the negative prompts generated by the LLM to the underlying diffusion model, which does not depend on the stage 2 implementation.
4.2
Quantitative evaluation
4.2.1 Proposed benchmark
We propose a text-to-image evaluation benchmark that includes four tasks: negation, generative numeracy, attribute binding, and spatial reasoning. Negation and generative numeracy involve generating specific number of objects not generating speeific objects. Attribute binding involves assigning the right attribute to the right object with multiple objects in the prompt. Spatial rcasoning involves understanding words that describe the relative locations of objects. For each lask, programmattcaally compose 100 prompts and query each model for text-to-image generation, with 400 prompts in total. 1pt-3.5-turbe. (Brown et al., 2020) is used in LMD for the benchmarks. We also implemented LMD+, a LMD variant that integrate pretrained GLIGEN et al. 2023b) adapters into our controller without further training. We refer readers Appendix J for details.
open-vocabulary object detector, OWL-VIT (Minderer al. 2022), to obtain bounding boxes for the objects of interest. We then check whether each generated image satisfies the requirements in the prompt. The accuracy of each task is computed by calculating the proportion of the image generations that match their corresponding prompts over all generations.
Detection-based evaluation. We
Results. As presented in Table D our model shows significant improvements in generation accuracy, ranging from 1.3x 3.6x compared to SD across four tasks and doubling the accuracy on average. Notably, LMD achieves image generation accuracy that more than twice of the SD accuracy for the spatial relationships and the negation task. This highlights the utility of the grounding image generation the LLM layout generator. Furthermore, when additionally integrating GLIGEN to pipeline to leverage in-domain instance-amottaed data, our method, denoted as LMD+, achieves additional improvements.
4.3 Ablation Study
Layout-to-image stage. Comparing with other layout-to-image methods. As shown in Table 2 compared with training-free free layout-to-image generation methods that perform semantic-level grounding, our proposed layout-grounded controller provides much better instance-level grounding. This is justified by the
10

---

Published in Transactions on Machine Learning Research (02/2024)
Image Accuracy
Image Accuracy
Method
Average of tasks
Method
Average of 4 tasks
SD v1.5 (Default)
37%
LMD (w/o SAM)
72.8%
LMD (on SDv1.5) (Ours, default)
77% (2.1x)
LMD (with SAM)
76.5%
SD v2.1
38%
LMD+ (w/o SAM)
82.8%
LMD (on SDv2.1) (Ours)
77% (2.0x)
LMD+ (with SAM)
80.5%
Table 3: LMD achieves comparable gains when adapted to Stable Diffusion v2.1 without any hyperparameter tuning or model training. This shows a promising signal that the gains from our method could carry along with the enhancement of diffusion models. The performance of our method could potentially be improved further with additional hyperparameter tuning.
Table 4:
Ablations using SAM vs using simple attention thresholding in stage 2. While removing SAM leads to slight degradation in LMD, removing SAM leads to even better performance in LMD+.
fact that our training-free controller even surassess training- based method GLIGEN [Li et al. 2023b) in the generative numeracy lask, despite not trained with any instance-level annotation. Furthermore, our controller also sigficantly surpasses training- based method GLIGEN (Li et al. 2023b) in attribute binding and spatial reasoning task. When integrated with GLIGEN to leverage instance-annotated datasets, our integration, denoted as LMD+, allows for further improvements without the need for additional training. Switching the base diffusion model without hyperparameter tuning. As shown in Table 3 thanks to our training- free nature, LMD maintains the gains to the base model (around 2x performance boost) when switch the base diffusion model from SDv1.5 to SDv2.1 without tuming any hyperparameters, including À and w that are introduced by our method [1] This showcases the potential of inlegrating LMD with future diffusion models. Using SAM us a simple attention threshold to obtain the per-bow mask. Instead of using SAM to obtain the mask for each box, we also explored an approach that does not require an additional segmentation module. Alternatively, we sort the pixels in each box according to their attention value with respect to the box caption and pick the top 75% pixels in each box with the highest, attention the mask for the box. As shown in Table 4 the impact of SAM is different for LMD/LMD++ In LMD, since the attention- based guidance is less spatially accurate with respect to the layout boxes, SAM helps to obtain the right mask that covers the object. Therefore, removing SAM leads to a slight degradation in LMD. In LMD+, since the guidance is more spatially accurate, SAM is no longer necessary most of the time. Instead, SAM sometimes picks a region that includes the background, causing confusion and reduced performance. Therefore, removing SAM slightly improves the results in LMD+. We make SAM an optional choice (as described in Fig. 2 but still recommend it for LMD and enable it by default. We refer readers to Appendix D for additional ablations the values of the hyperparameters.
Text-to-layout stage. Ablating in-conteat ezamples. In addition to using the seven fixed in-context examples provided in Table K.2 by default, we also vary the number of in-context examples given to the LLM (i.., "shots"). We show in Table 5 that while GPT-3.5 benefits from more in-context examples, GPT-4 is able lo successfully generate all the layouts even when given only one in-context example. Note that we also observe GPTT4 to still be able to generate layouts without ony in-contert ezamples (i.e., given only the text instructions). However, since no examples are offered as references in this zero-shot setting, the format of LLM outputs are observed to differ in different runs, making it hard to parse with program. Since it is much easier to convey the format through an example than through language instructions, we recommend having at least one example. Our observation shows that LLMs already learn the ability to generate object bozes during pretraining and do not need us to convey through many in-context examples. Varying the model types and the sizes of the LLMs. We also ablate the LLMs used for -oy-uu generation, including using self-hosted LLMS with public weights (Mahan et al.) 2023 Touvron et al., 2023 Mukherjee et al. 2023 Jiang et al.) 2024). The results show that the capability generate high-quality layouts are not limited to proprictary LLMs, and larger LLMs offer much better layout generation capabilities. We refer the readers to Appendix D and Appendix E for more ablations and investigations.
Thee hyperparameters inherited from the latent diffusion framework, such as the number of denoising steps, are also unchanged.
11

---

Published in Transactions on Machine Learning Research (02/2024)
Layout Accuracy (4 tasks)
Color
Shape
Texture
Spatial
# shots
gpt-3. t-trbo
gpt-4
SDv1
0.3765
0.3576
0.4156
0.1246
Shot
89.8%
100.0%
LMD (on SDv1)
0.5495
0.5462
0.5241
0.2570
Shots
96.3%
100.0%
SDv2
0.5065
0.4221
0.4922
0.1342
Shots
99.0%
100.0%
LMD (on SDv2)
0.5736
0.5334
0.5227
0.2704
Table Ablations on the number of in- context examples (*shots") given to the LLM. While GPT-3.5 benefits from more in- context examples, GPT-4 already excels in layout generation even with only one example.
Table 6: Our method
surpasses the base
diffusion models SDv1 and SDv2 T21-
CompBench (Huang et al., 2023) on all four tasks without additional training.
(n) Stable Diffusion XL
(b) Our Layout
Our Generation
(d) Round Layout
Round Generation
watercolar painting of two apples on a wooden table, neither is red and both are green
A Aasrrddr painting
wateroo or painting
8
Figure 9: A failure case occurs when our method, shown in (c), gencrates objects in unintentional viewpoints
and sizes due the ambiguity in the generated layout. The LLM-generated layout (b) suitable for close-up top-down view of small table, but the layout-to-image model assumes side view and thus fails to generate feasible image. Nevertheless, our method still provides more interpretability through the intermediate layout (b) compared to baseline SDXL (a). With an additional request for the side view and correct object sizes, the LLM adjusted the layout in (d) and the final generation (e) is aligned with the text prompt.
4.4 T21-CompBench
In addition our proposod benchmark with detection- evaluation, we evaluate our method on T2I- CompBench (Huang et al. 2023) that additionally uses visual question answering (VQA) models for generation evaluation. The color, shape, and texture tasks employ BLIP (Li et al. 2022 in VQA setting, while the spatial task uses UniDet (Zhou et al. 2022) for evaluation. As shown in Table G our method LMD, when applied on either SDv1 or SDv2, improves the performance on all four tasks. Additional ablations are in Table D4
4.5 Evaluator-based Assessment
Setting. We also assess the prompt following ability of our method and vanilla SD, the base diffusion model that our method uses under the hood. We randomly selected 10 text prompts from our proposed benchmark and generated pair of images per text prompt, one with our LMD+ and one with the base model SD 0 We then invited 11 evaluators to compare each image pair and answer two questions:
Question 1: Which image aligns better with the text prompt?
Question 2: Which image has a more natural and coherent foreground- foreground-baackgrommm background composition?
In addition to an option for preferring each image, "similar" option also provided for each pair.
Results. We average the scores across 110 responses. The results show that our method LMD+ got 88.18% (vs 10.90% for SD) for the first question and 35.45% (vs 31.81% for SD) for the second question. This indicates that our method generates images that accurately align with the prompt compared to the baseline SD without degradation of naturalness or coherency.
6The hase model of LMD+ technically GLIGEN, which itself built on However, setting where only is used imput, conditioning scheme GLIGEN simply becomes degenerate. Therefore, compare with SD directly instead.
12

---

Published in Transactions on Machine Learning Research (02/2024)
5 Discussions
Since we use models off-the-shelf, the LLM may generate layouts that are ambiguous to the diffusion model. For example, the layout in Fig. ( is feasible for a top-down close-up image, but the diffusion model generates an image viewing from the side. This makes the apples not on the table in Fig. Mc). Prompting or fine-tuning the LLM to be more explicit about its assumptions in the layouts (e.g,, viewpoints) may alleviate this problem. The intermediate layout our two-stage generation allows for more interpretability compared to our base model stable diffusion. After diagnosing the point of failure, we give additional request for the side view and correct object sizes to the LLM. The LLM adjusted the subsequent layout generation, which allows generating images that align with the input prompt in round 2, as shown in Fig, (f,,e). Our method also inherits biases from the base diffusion model (Luccioni et al. 2023). Moreover, although our method can handle objects not mentioned in the in-context examples (e.g., the bear and the deer in Fig. 7 the LLM may still generate better layouts for objects mentioned in the in-context examples by referencing layout examples. Our method could also be distilled into one-stage lext-lo-image diffusion model to improve its prompt understanding abilities without leveraging LLMs at inference time for the ease of deployment.
6 Summary
In this paper, we enhance the prompt understanding capabilities of text-to-image diffusion models. We present novel training-free two-stage generation process that incorporates LLM-based lext-grounded layout generation and grounde image generation. Our method also enables instruction saseeto---bseedd scene specification and generation from prompts in languages unsupported by the base diffusion model. Our method outperforms strong baselines in accurately following the prompts in text-to-image generation.
Acknowledgements. The authors would like to thank Aleksander Holynski for the helpful discussions.
13

---

Published in Transactions on Machine Learning Research (02/2024)
References
Jean-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 26-16666,66 2022.
Omri Avrahamii, Thomas Hayes, Oran Gafni, Sonal Gupla, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18370-18380, 2023.
Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusson: Fusing diffusion paths for controlled image generation. arXiv preprint XXt::23020081110 2, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Eiros. Instructpix2pix: Learning follow image editing instructions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogration, pp. 18392-18402, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877 1901, 2020.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.0 03373, 2023.
Jiaxin Chong, Xiao Liang, Xingjian Shi, Tong He, Tianjun Xiao, and Mu Li. Layoutdiffuse: Adapting foundational diffusion models for layout-to aayout-to-image ro-image generation. arXiv preprint 3rXiv:2302.08906 2023.
Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, and Alexei A Efros. Blobgan: Spatially disentangled scene representations. In Computer Vision- ECCV 2022: 17th European Conference, Tel Avie, Israel, October 23-27, 2022, Proceedings, Part XV, pp. 616-635. Springer, 2022.
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305. 2,335393 2023.
Kamal Gupta, Justin Lazarow, A Alessandro Achille, Larry S Davis, Vijay Mahadevan, and Abhinav Shrivastava. Layouttransformer: Layout generation and completion with self-attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1004-1014, 2021.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognilion, pp- 14953 14962, 2023.
Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. Learning canonical representations for scene graph image generation. In Computer Vision ECCV 2020: 16th Europeon Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16, pp.
Springer, 2020.
Jonathan Ho and Tim Salimans. Classifier free diffusion guidance. arXiv preprint arXio:2207.15500 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffuusion probabilistic models. Advances in Newral Information Processing Systems, 03:6840-68511, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXio preprint arXiv:2106.0900 2021.
Kaiyi Huang, Kaiyue Sun, Euze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. arXiv preprint ar arxi::207..00 063500 2023.
14

---

Published in Transactions on Machine Learning Research (02/2024)
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXie:2401.00000 2024.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1219 1228, 2018.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Yen-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304006443, 2023.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. arXiv preprint Xxxx22305.12166 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 11888-12900, PMLR, 2022.
Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language modols. arXiv preprint XXi:23311115091 2023a.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jac Lee. Gligen: Open-set grounded text-to-image generation. arXiv preprint arXiu:2301.07003 2023b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- -CCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740 755. Springer, 2014.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. + Fast solver for guided sampling of diffusion probabilistic models. ai preprinl arXiv:2211.01099 2022.
Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias: Analyzing societal representations in diffusion models. arXiv preprint arXiv:2303. 2023.
Dakota Mahan, Ryan Carlow, Louis Castricato, Nathan Cooper, and Christian Laforte. Stable beluga models, 2023. URL [ttts///wwggggaa........ooiii//e
/o/
stabilttyal/Stabeeeellaa
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection with vision transformers. arXiv preprint orXiv:2205.062. 06223, 2022.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces gpt-4, 2023.
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087704 2023.
Taesung Park, Ming-Yo Yu Liu, Ting- Chun Wang, and Jun-Yan Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern 2019.
recognition, pp.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXie preprint arXit:2307.0195. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishknnn Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn International conference on machine learning, pp. 8748 8763. PMLR, 2021.
15

---

Published in Transactions on Machine Learning Research (02/2024)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint or Xiv::2204 20::2244001255 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthosis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 10684 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medicol Image Computing and Computer- Assisted Cnerrvention-MMCCC. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241. Springer, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35: 36479 36494, 2022.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ar preprint arXiv:2010.02500 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Sounya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anee Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, 1gor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and finc-tuned chat modols, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprinl ar arXiv:2303. 200:20003047,,, 2023.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shon. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. arXiv preprint or w:Xiw:2307.10811 2023.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEF. conference computer vision and pattern recognition, pp. -419, 2017.
Zhengyuan Yang, Jianfeng Wang, Zhe Can, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 14246-14255 2023.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to- text-to-imaae diffusion models, arXie preprint Xiv: 05543, 2023.
Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In Proceedings of the IFFF/CVF Conference on Computer Vision and Pattern Recogrittion, pp. 8584 8893,8933 2019.
16

---

Published in Transactions on Machine Learning Research (02/2024)
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 22490 22499, 2023.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühh. Simple multi-dataset detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 7571-7580, 2022.
17

---

Published in Transactions on Machine Learning Research (02/2024)
A Preliminary introduction to latent diffusion models
The layout-to-image stage (i.c., the image generation stage) of our method LMD builds on off-the-shelf lext-to-image Stable Diffusion models, which is based on the latent diffusion framework (Rombach al.
2022 We present a preliminary introduction to the latent diffusion framework in this section and define the key terms used in our work. We encourage the readers to check Rombach et al. (002 for a detailed explanation of the latent diffusion framework.
Latent diffusion models (Rombach et al. 2022) are powerful generative models that learn the data distribution of complex, high-resolution image datasets. Before training a latent diffusion model, Rombach et al. (2022) first trains an image encoder that converts an image x into vector z in the nigh-dimensional latent space and decoder that converts z back to vector in the image space that is similar to in appearance. By training and sampling diffusion model in the latent space, latent diffusion lowers the cost of training and sampling from high-resolution diffusson models and is widely used in text-to-image generation, with Stable Diffusion as popular model based on the latent diffusion framework. Our method improves the prompt understanding of Stable Diffusion without adapting the weights.
During training, the latent, diffusion framework first maps each training image, denoted as xo, into latent Za with the image encoder that is frozen during the diffusion training stage:
Encode(Xo)
(8)
A timestep t is sampled uniformly from (I,...,T}, where T is a hyperparameter.
Noise E is then sampled from a Gaussian distribution parameterized by timestep and added to the latent zo obtain noisy latent 2₁- A neural network with parameter 0 learns to prodict the added noise for the forward process by minimizing the training objective:
L = ||| t)12
(9)
The neural network described above often uses a variant. of U-Net (Ronneberger et al., 2015) architecture that has attention layers (Vaswani et al. 2017), and thus also referred to as the diffusion U-Net.
At inference time, there are many sampling methods that allow the synthesis of samples from a diffusion model trained in the fashion described above. The general intuition is to go through reverse process (also called demoising process) in which the diffusion model Eş iteratively predicts noise vector Eø t) from and subtracts it to transform into sample 2z-1 that has less noise and is closer to the distribution of the training set, with t initialized as T and zr I). The denoised sample resembles the clean data in the latent space.
One can use DDPM (Ho et al. 2020 to perform sampling from a noise prediction model eg. DDPM predicts the noise for each of the denoising steps and then obtains z from z, using this formula:
4-11
2t
Ote co (2),1) + ofEt
(10)
Vau
I1 as
where 4 N(OO I), a and or are parameterized by variance schedule {B E (0, 1))T, that controls the size of the denoising step.
Denoising diffusion implicit models (DDIM, Song et al. (0200) are a generalization to DDPM which allows sampling with fewer iterations. DDIM applics the following update rule:
2
04
21
C1,E0 (1ea(Zes!)) + oiEt
(11)
Vai
Note that DDIM shares the same training procedure with DDPM, which means we can choose to perform DDIM DDPM for trained diffusion model. When of is set to O, which is the case for our setting, the
18

---

Published in Transactions on Machine Learning Research (02/2024)
denoising becomes deterministic given zy. The results shown in our work are obtained with DDIM with 0, = 0, with other faster sampling methods such as Tu et al. (2022) also applicable to our method.
Since there are many sampling methods given a trained diffusion model that are applicable in the latent diffusion framework, we denote the denoising process, such as the one in Eq. (10 and Eq. 11 as
- Denoise(z:)
(12)
After getting the denoised sample 2o, then decode the image with an image decoder:
X (ecode
(13)
Text-conditional generation through cross-attention. The above formulation describes the uncondi- tional generation process of latent diffusion models. Models such as Stable Diffusion take text as input and perform conditional generation. The difference between conditional and unconditional generation process involves processing the input text into text features, passing the feature tokens to diffusion U-Net, and performing classifier-free guidance (Ho & Salimans 2022). which is described as follows.
Rather than only taking the noisy input x, and timestep 4, the conditional diffusion U-Net 6a(z,, ,,, To (y)) takes in an additional text condition y processed by a text encoder TO(-). The text encoder is a CLIP (Radford et al. 2021) text encoder in Stable Diffusion. After y is tokenized by the tokenizer into discrete tokens, it is processed by a Transformer (Vaswani et al. 2017) to text features ( pixdext where is the number of text tokens in y after tokenization and deext is the dimension of features.
The text features Ta (y)) are then processed by the cross-atlention layers in the diffusion U-Net so that the output of the U-Net can also change depending on the text. For simplicity, we only consider one tross-attention head in this preliminary introduction and refer the readers to Rombach et al. (2022) and Vaswani et al. (2017) for details with the multi-head cross-attention used the U-Net in the latent diffusion framework.
Specifically, each cross-attention layer linearly maps the text features ry into key and value vectors k, V E R R'xdatta where dastn is the attention dimension. Each cross-attention layer also takes in the flattened 2D feature from the previous layer in the U-Net and linearly maps the feature into query vector q Rm dattn where is the dimension of the previous flattened 2D image feature.
Then, a tross-attention map A is computed from the query q, key k, and value v vectors, which describes the affinity from the image feature to the text token feature:
Age = tmfx (q k,)
(14)
where qa and k, are linearly transformed image feature at spatial location and text feature token index in the prompt, respectively.
The attention map is then used for computing a weighted combination of the values v:
O₂ Σ AueVe
(15)
o Rmmxdat is then linearly transformed become the output of the cross-attention layer. The residual connections and layer norms are omitted in this introduction for simplicity.
Samples are generated by classifier-free guidance to ensure aligument. with text prompt y. At training time, with a small probability, the input condition Ty(y) is randomly replaced with a learnable null token To- At inference time, classifier-free guidance uses the following term Ex(x,, t, (y))) in place of the predicted noise ca (xx, t) in the update rule for unconditional generation:
Ex(x1, To(y)) = wea(x,, L, (y()) (1 (),
(16)
where w is the strength of classifier-free guidance, set to 7.5 by default in Stable Diffusion.
19

---

Published in Transactions on Machine Learning Research (02/2024)
Algorithm Layout-grounded image generation.
Input: set of captioned bounding boxes {(b(9) )
Background caption y(bg).
Output: Image Xy-
1: br f- LaaeeGGGssiaa((..11
Per-box masked latent generation:
for each captioned box (b(*), y()) do
(i)
4: Zip + 2T
y + PromptForBox
(bb))
for + T to 1 do
()) A AttnControl (1) y(0 b())
2
2 + Denoise(z) (1) y
8:
9:
end for
A + TemporalAverage(A""" (1)
10:
m (6) SAMRefine(A(1) (8) (Optional: This could be replaced with an attention thresholding
11:
instead.)
12:
2(()
ze
(2)
m
(i)
13: end for
14: Composed image generation: (comp) zt
15: zy
16: y x (C ((.
17: for t T to 1 do
18:
if t rT then
(comp)
+ LatentCoomose((? (comp)) 2 m ()) Vi
19:
z Z:
(comp)
AttnTransfer (comp) (comp)
(A4))) )
20:
21:
end if
22:
2
(comp)
Denoise( (2
(comp) (comp)
23: end for
(comp)
24: Xo + Decode Z
20

---

7. ComposedPrompt composes the prompt for overall generation. We offer two options for the overall prompt: using the original input prompt or composing the prompt as "fbackground prompt) with fdor caption 1], fbor caption 2) ...". The former one allows capturing the object as well as forground- background interactions that are not captured in the layout. The latter allows captions in languages Published in Transactions on Machine Learning Research (02/2024)
B Pseudo-code for
image generation
We present the pseudo-code for our layout-grounding stage (stage 2) in Algorithm functionality of the functions used in the pseudo-code:
We explain the
SampleGaussian samples i.i.d standard Gaussian as the initial noise for the latent tensor.
PromptForoox simply sets "[background promptj with fbox caption (e.g., "a realistic image of on indoor scene with a gray cat") as the denoising prompt.
3. AttnControl performs backward guidance to minimize the energy function Eq. 22 described in Section 3 to encourage the attention to the area within the box and discourage the attention area outside the box. The cross-attention maps A!" are also returned in order allow obtaining a mask for each box.
4. Denoise denotes one denoising step by the diffusion model.
5. TemporalAverage averages the cross-attention map across the timestep dimension.
6. SAMRefine refines the attention map by internally decoding the latent and refining with SAM. If SAM is not enabled, we perform an attention thresholding instead.
unsupported by the diffusion model and stays robust when the caption is misleading (e.g., "neither of the apples is red"). We use the latter by default but also allow the former for fine-grained adjustments.
8. LatentCompose spatially composes each of the latents (0) with respect the corresponding mask (f) replacing the content of the destination latent the masked locations. As for the order of composition, we compose the masked latents with the largest area after masking first.
9. AttnTransfer performs backward guidance to minimize the energy function Eq. 7 in Section 3 to encourage the altention in overall generation within the box to be similar to the attention in per-box generation addition to attention control.
C Additional features and use cases from instruction-based scene specification
As shown in Section 3.3 LMD, equipped with instruction-based scene specification, allows the user to apply follow-up instruction requests in addition to the initial prompt.
Furthermore, we demonstrate two additional use cases supported by instruction- baaed scene specification in Fig. C.T without additional training.
In Fig. C.1 a), instruction-based scene specification allows the users to try out different adjustments on the same generation while preserving the overall image style and layout, facilitating fine tineggrined content creation.
The LLM equipped in LMD can also respond to open-ended requests and present suggestions for improving the scene. Moreover, different from instruction-based image editing methods that only take one instruction without context, our instruction- more natural dialog with users. For example, Fig. (.), our method can respond instructions with phrases such "What are some objects that you can add to make lively?", "undo the last edit", and "adding small pond in
scene specification parses the instruction in its context, allowing for
21

---

Published in Transactions on Machine Learning Research (02/2024)
Instruction-based Scene Specification: Fine-grained Adjustments
LMD (Ours)
LMD (Ours)
Change the white car to red car
realistic photo of road scene with two cars. The car on the left is white. The car on the right is black.
Followup Request
LMD (Ours)
Initial request
Change the black car to blue car
8
Followup Request
(a) LMD allows the users to try out different detailed adjustments while preserving the overall image style and layout, enabling fine-grained content creation.
Instruction-based Scene Specification:
Open-ended Requests and Multi-round Context LMD (Ours)
small bird pecking near the rock. squirrel foraging close to the rock.
What are some objects that you can add to make it lively?
A small pond or water stream nearby, reflecting the sky.
realistic photo of a rock on grass.
8
Initial user request
Followup Request 2
LLM Response
LMD (Ours)
LMD (Ours)
Great. Please undo the last edit and make the
Make it lively by adding more relevant objects
scene lively by adding a small pond instead.
Followup Request
8
Followup Request
(b) The LLM used by L.MD can perform open-ended scene adjustments, give suggestions, and understand user requests based on the contexts over multiple rounds of user dialog.
Figure C.1: Additional features and use cases enabled by instruction-based scene specification.
D Additional ablation studies
D.1 Text- lo-yout stage
Varying the LLM types. All LLMs in Table D.1 generate layouts that almost perfectly follow the requirements in the prompts, indicating the bottleneck to be the layout-to-image stage. gpt-4 shows improved results in layout and the subsequent image generation, compared to gpt-3.5-turbe 5-turbo. The capability to generate high-quality layouts are not limited to proprictary LLMs, with Llama2- based StableBeluga2 (Mahan al.) 2023) and Mixtral-8a -tt--xx-7---Itttt--eee Instruct-vo. (Jiang al. 2024) also able to perform lo-youo generation the stage We believe that fine-tuning these models will lead to better performance in terms of text-to-layout. generation.
ct al. 2023 Touvron et al. 2023 Mukherjee
22

---

Published in Transactions on Machine Learning Research (02/2024)
Layout (Image) Accuracy
Layout Accuracy
Stage 1 Model
Average of tasks
Stage Model
Average of 4 tasks
StableBeluga2
96.5%
(67.0%)
Mixtral--k/BB-Inttrrcttt1111 gpt-3.5-turbo
98.3%
(77.5%)
StableBeluga-78 59.3%
99.0%
(80.5%)
StableBeluga-1
84.0%
gpt-4
100.0%
(86.3%)
StableBeluga2 (70B)
96.5%
Table D.1: Ablations different LLMs in stage
Table D.2: Ablations the LLM model size on StableBeluga Models (Mahan ct al. 2023) based on Llama-2 (Tovvron et al., 2023) for layout generation (stage 1 only). Larger LLMs offer more accurate layout generation compared to smaller LLMs.
1. Although proprictary models such as GPT-3.55 and GPT-4 perform the best, the ability to generate high- quality layouts is also present in open-source models
Mahan et al. (2023); Jiang et al. [2024]; Touvron et al.
(2023). The image accuracy benchmarked using
LMD+ stage 2.
Image Accuracy (Average of tasks)
Image Accuracy (Average of 4 tasks)
Method
w=1
w = 2
w
w = 8
Method
x=
X x=
== 2
À == 3
LMD
72.3%
75.8%
76.5%
72.5% 78.3%
LMD
70.8%
75.0%
76.5%
77.3%
75.0%
LMD+
79.8%
80.0%
80.5%
LMD+
79.3%
79.5%
80.5%
81.8%
78.8%
(a) Ablations on hyperparameter w.
(b) Ablations on hyperparameter A.
Table D.3: Ablations on hyperparameter w and X. Our method is relatively stable in terms of hyperparameter values w and A. While we did not perform hyperparameter search, our default hyperparameter allows optimal performance for both LMD and LMD+. For the hyperparameter X, we found that setting À = 3 leads to better performance compared to our default hyperparameter setting with A 2, which indicates that the performance of our method can be further improved through hyperparameter tuning. Underlined numbers indicate performance with our default hyperparameter selection (w = 4, A = 2). Bold numbers indicate the best performance among all the hyperparameters ablated.
Varying the LLM sizes. We also tested the ability of layout generation LLMs of different model sizes. As shown in Table D.2 larger LLMs offer much better layout generation capabilities.
D.2 Layout-to-image stage
Varying w. is the weight for balancing the loss term on the foreground and the term on the back- ground (Eq. (2)). While we set 4 4 by default, ablate this design choice. As shown by the experimental results in Table D.3a our method is relatively stable in terms of hyperparameter selection. Moreover, even though we did not perform hyperparameter search prior to determining our default hyperparameter value, our default hyperparameter w = 4 already leads to the optimal performance among the hyperparameter values that we searched in this ablation for both LMD and LMD+.
Varying X. À is the weight for the attention transfer term in Eq. 77 As shown Table D.3b setting À= 3 leads better performance compared to our default hyperparaneter setting with 2, which indicates that the performance of our method can be further improved through hyperparameter tuning.
we found that
Ablation results on T2I- Cen-ChhBBench (Huang et al., 2023). In addition to comparing our method with the baseline method Stable Diffusion in Table 6 we further combine our text-to-layout stage (stage 1) with other layout-to- imaee methods as stage in this ablation, similar to Table 2 The results are in Table D.4, with the results for the SD bascline from Huang al. (2023). Our method surpasses not only the base diffusion model SD but also several variants our method that combine our stage with previous layout-to-image methods as stage 2, which shows the effectiveness of our layout-grounded controller.
23

---

Published in Transactions on Machine Learning Research (02/2024)
Stage 1/Stage 2
Color
Shape
Texture
Spatial
SD
0.3765
0.3576
0.4156
0.1246
MMDD/MuliiDiffsiioon
0.4631
0.4497
0.4007
0.1604
LMD/
LLDD/Backwerr /Backward Guidance
0.4877
0.5069
0.4643
0.2361
LMD/ LMD/Boxdiff /Boxdiff
0.4579
0.4967
0.4720
0.1965
LMD LMD/LMD /LMD (Ours)
0.5495
0.5462
0.5241
0.2570
Table D.4: Our method surpasses the base diffusion model SD as well several variants of our method that combines our slage 1 with previous layout-lo-image methods as stage on T21-CompBench (Huang al.)
2023
forest
malistic scena
The only in-context example with prompt "A panda in forest without flowers"
Generated layout for prompt "A apple"
apple
pandaa
realistic outdoor
malistic scene
The only in-context example with prompt "A realistic scene of three skiers standing in line on the snow near palm tree"
Generated layout for prompt "A realistic scene of three bears"
A skier A skier
A skier
palm tree
bear
bear
Figure E.1: The generated layouts are not necessarily similar to the in-context examples in terms the spatial distribution of boxcs. We present the LLM with only one in-contert enample and query with prompt that similar to the example. Top: While the query and the example shares similar structure (only one object), the LLM generates box for "an apple" that is very different from "a panda" in terms of the size and position. Bottom: The LLM does not simply copy boxes for the three skiers in the in-context example to generale the boxes for three bears.
E Are the generated layouts distributed similarly to the in-context examples?
Since our LLM takes a few in-context examples in our text-to-layout stage, it is possible that the LLM prefers to generate samples that are similar to the in-context examples in terms of spatial distribution. To test whether this is the case, present the LLM with only one in-context example and query it with a prompt that is similar to the example. The results shown in Fig. E.1 Even though each of the query prompts shares similar form to the corresponding in-conlext example, the LLM still generates layouts that are tailored to the objects in the query prompt (e.g., the apple and the bears) rather than copying or mimicking the layout boxes from the in-context examples. This qualitative analysis shows that even with the in-context examples as references, the LLM often generates natural layouts according to the prompts, relieving the users from heavy prompt engineering prevent overly similar layouts between the generation and the examples.
F Additional visualizations
We also present Fig. F.1 which includes qualitative comparison with Stable Diffusion v1.5 (abbreviated SDv1) and shares the prompts with Fig.
24

---

Published in Transactions on Machine Learning Research (02/2024)
(a) Stable Diffusion
(b) LMD (Ours)
(a) Stable Diffusion
(b) LMD (Ours)
Attribute Binding
Negation
man in red standing next to another woman
a wooden table without bananas
in blue
Spatial Relationships
Generative Numeracy
A watercolor painting cats on the grass
3 apples arranged In an L-shape on a wooden table
8
Figure F.1: We also generate images with the same text prompts as Fig, 1 with SDv1.5 and LMD on SDv1.5. We obscrve similar results which show that while Stable Diffusion Rombach ct al. (2022)
(a) oflen struggles to accurately follow several lypes of complex prompls, our method LMD (b) achieves enhanced prompt understanding capabilities and accurately follows these types of prompts.
G Benchmarking VisualChatGPT and GILL for multi-round instruction-based scene specification
Gis (Wu et al. 2023 and GILL (Koh et al. 2023) involve LLM in their image generation pipelines and thus could potentially take instructions from multiple rounds of dialog for image generation. Therefore, in addition to the qualitative benchmark in Fig. g also benchmark both methods for multi-round scene specification. As shown in Fig. G.1 the generated images quickly degrade starting from the second iteration, showing that neither method is able to take instructions from multiple rounds of dialog for image generation. In contrast, our method able to handle several rounds of sequential requests on image generation without generation degradation, shown in Fig. 6
H Details for SDXL integration
Thanks to the training-free nature of work, our method is applicable to various diffusion models without additional training. Therefore, we also apply our method on SDXL 1.0 (Podell et al. 2023). the latest stable diffusion model which has 3x larger U-Net module compared to previous stable diffusion models (Rombachl et al. 2022).
It is forward to apply the LMD pipeline directly to SDXL UNet, which has a very similar procedure to applying the LMD pipeline to SD v1/v2. This approach only requires marginal modifications of the LMD pipeline: different from SD v1/v2 that use only one text encoder for encoding the prompts, SDXL involves two text encoders for text feature generation, and the attention control proposed in LMD needs to be applied to the cross-attention with both text encoders taken into account. The rest. follows from the standard LMD pipeline.
Inspired by methods such as Ramesh et al. 2022 that generate low resolution images and then upsample the generation to the target resolution, an alternative approach to perform denoising with the standard LMD with standard SDv1/v2 resolution (i.c., 512 x 512) and then perform upsampling with SDXL refiner for few steps to the intended resolution (e-g., 1024 x 1024). Since most of the generation still happens in the standard resolution latent space, with the SDXL only involved limited number of steps for high-resolution latents, this approach is more efficient compared to the former approach. We compare the generation for the same scene with SDXL baseline and both approaches in Fig, H.] Both approaches present much better prompt following ability compared to SDXL baseline. We observe similar generation quality on both approaches. Therefore, we use the latter approach by default.
For Fig. D and Fig, 7 we use SDXL 1.0 as the base model of LMD and compare against SDXL as a strong baselie.. For all other settings, including the qualitative evaluation setting, we use Stable Diffusion
25

---

Published in Transactions on Machine Learning Research (02/2024)
VisualChatGPT (Baseline)
Swac the apple
(b) GILL (Baseline)
Svep the apple the pour
Swep the apple and the por
The sage shel
Ceraste an ingguat
Oerenze vuge dos viswo awaddertaee
Eoroeethoper
forowetheagee
Meweel
Plage (rn.
Macarrpless al
the
Mow.anagdrre to
the aage to snie
Movthheapt to center
Hai Iocer secur
Movethoperr
Heweee
Hupp lon,, on senggnnnnn htwp:///llaeeer...eeeeeeeo
Aun-ape cw
then peras anttd usi sigeraaapper ondddd mage seeris
Mmggdtie opeorernehyyy 7he 4ech617
Moxe www.app0aa
Thank
Phit Lp
Maketherpforrdl
Hekebot huto
Ho for
then acttidl in wuggeaits bath lagee thainage
Wo Bot allaler
The parvenat in deisppiiee comtmentt
www.e/ppppp......oo.. redagile
vade thainap foreee
(conversation continues on the righf)
(conversation continues on the right)
Figure G.1: VisualChatGPT Wu al. (2023 and GILL Koh et al. (2023) generally cannot handle more than one round of image generation requests, with the generated image degraded starting from the second request. In contrast, our method is able to handle several rounds of sequential requests image generation without generation degradation, shown in Fig. 6
v1.5 (denoted as SDv1) unless stated otherwise. For fair comparison with Wu et al. 2023 and Koh et al. (2023) that only use Stable Diffusion v1.5, we also generate images for the same set of prompts of Fig. with Stable Diffusion v1.5 in Fig. F.1
Generating images from languages not supported by the underlying diffusion model
As shown in Fig. by asking the LLM to always output layouts in English even if the prompt non- English (e.g., Korean or Chinese as Fig. L.1 and providing an in-context example of non non-English input and English layout, LMD is able to generate images from prompts in languages not supported by the underlying
26

---

Published in Transactions on Machine Learning Research (02/2024)
(a) SDXL Bascline
(b) Applying LMD directly SDXL
(c) Applying LMD SD in low-res and refine with SDXL in high-res
Figure H.1: LMD can be easily applied on the latest stable diffusion model SDXL (Podell et al. 2023). We compare generated images from text prompt "A realistic photo of gray cat and an orange dog on the grass". (a) directly generates the image from the text prompt. SDXL does not accurately generate the image from the prompt, showing that simply scaling the diffusion model does not necessorily lead to improved prompt following ability. (b) Thanks to our method being training-free our method can be directly applied on SDXL without additional training. (c) An alternative way to integrate our mothod with SDXL to use our method generate low-resolution images with SD and then refine the image in high-resolution in SDXL. Since most denoising is completed in low-resolution latents, this approach is more efficient.
Supporting More Prompt Languages
Stable Diffusion
Supporting More Prompt Languages
Stable Diffusion
AI ø
EAARREKEE F
ERI žAA2 Exx flo 0 RI= 2
DR GEOG 2HH E PEEN
(Translation: A watercolor painting of an indoor scene with a plate of fruits placed on a table.)
(Translation: A watercolor painting of indoor scene with a plate of fruits placed on a table.)
LMD (Ours)
LMD (Ours)
8
The translation is not part of the input to our method.
'The translation is not part of the input to our method.
Figure Ll: By asking the LLM to always output layouts in English, LMD is naturally able to generate images from prompts in languages not supported by the underlying diffusion model.
diffusion model. We simply translate the prompt input of the last in-context example to non- non-English, while keeping the output in this example in English. No adaptation needed on the diffusion model since the underlying diffusion model still takes in an English layout as input.
J Details for text-to-image benchmarks
We pick 10 common object types from the COCO dataset Lin al. (0114 for generation!" 7
For negation and generative numeracy task, each prompt requires the model to generate layout of scene with some number of a certain object or without certain object. Then we count the number of objects and consider the layout to be correct if the number of the object of that particular type matches the one in the prompt, with the number ranging from to 5.
BBackpack, book, bottle, bowl, car, cat, chair, cup, dog, and laptop.
27

---

Published in Transactions on Machine Learning Research (02/2024)
The objective for each prompt in the attribute binding task is to generate object of color and another object of another color, for which the evaluation is similar to other tasks.
For the spatial relationship task, we generate an object at certain location and another object at an opposite location (left/right and top/bottom). We then check the spatial coordinates of the boxes to ensure the layout exactly matches the prompt. In each task, we generate 100 text prompts, with 400 text prompts total.
Prompts. For the negation benchmark, use the prompt A realistic photo of scene without jobject namej.
For generative numeracy, use the prompt A realistic photo of scene with [numbber] (object namej.
For attribute assigmment, we use the prompt A realistic photo of a scene with fmodifier 1] jobject name 1J and [modifier 2/ fobject name 2/, where the two modifiers are randomly chosen from a list of colors (red, orange, yellow, green, blue, purple, pink, brown, black, white, and gray).
For the spatial relationship benchmark, we use the prompt A realistic photo of scene with fobject name 1] the [location] and fmodisier 2] fobject name2] on the jopposile localion), where the location is chosen from left, right, top, and bottom.
Implementation details. For LMD, we use Stable Diffusion v1.5 by default. For LMD+, we use GLIGEN
[Li et. al. 2023b) model without additional training or adaptation. We selected the GLIGEN (Li et al. model trained based on Stable Diffusion v1.4, which is the latest at the time of writing, We use 7= 5, 2.0, repeated 5 times for each denoising timestep and linearly decreases for every five denoising steps until the repetition is reduced 1, and we do not perform guidance after 30 steps. & in the Topk(.) in Eq. (2) is set to 20% of the area of the mask for each mask. The background part (second term) of Eq. ( is weighted by = 4.0. We run the denoising process with 50 steps by default. We only perform latent compose in the first half of the denoising process (first 25 steps). The qualitative visualizations/ intaalizationsquuauttttttr quantitative comparisons are generated by LMD+/LMD, respectively, by default unless stated otherwisc.
2023b
r = 0.4, guidance scale 7.5. The energy minimization
Our LLM prompt
Our LLM prompt is listed in Table K.1 Our in-context examples are listed in Table K.2
28

---

Published in Transactions on Machine Learning Research (02/2024)
You intelligent bounding box generator. will provide you with caption for photo, image, painting-
task is generate the bounding boxes the objects mentioned in the caption, along with a background prompt describing the scene. The images are of size 512x512. The top-left corner has coordinate [8, 81. The botton-right corner has coordinnate [512, 512]. The bounding boxes should not overlap or go beyond the image boundaries. Each bounding box should be in the format of (object name, [top-left x coordinate, top-left coordinate, box width, box height]) and should not include nore than one object. Do not put objects that are already provided in the bounding boxes into the background prompt. Do not include non-existing or excluded objects in the background prompt. Use "A realistic scene" as the background prompt if no background is given in the prompt. If needed, can make reasonable guesses. Please refer to the example below for the desired format.
[In IIn-context Examples]
Caption: [User Prompt] Objects:
Table K.1: Our full prompt to the LLM for layout generation. LLM starts completion from "Objects:".
Caption: realistic image of landscape blue truck, with red air balloon and bird in the Objects: [('a green car', [21, 281, 211, 159]), ('a blue truck', [269, 283, 209, 160]), ('a red air balloon', [66, 8, 145, 135]), ('a bird', [296, 42, 143, 100])] Background prompt: realistic landscape scene Negative prompt:
depicting green parking the left of
Caption: realistic top-down view of wooden table with two apples on Objects: [('a wooden table', [20, 148, 472, 2161), ('an apple', , [150, 226, 100, 100]), ('an apple', [280, 226, 100, 19a])]
Background prompt: A realistic tow-down view Negative prompt:
11 Caption: realistic scene of three skiers standing in line on the snow near a palm tree 12 Objects: [('a skier', [5, 152, 139, 168]), ('a skier', [278, 192, 121, 158]), ('a skier', [148, 173, 124, 155]), ('a palm tree', [404, 185, 103, 251])]
Background prompt: A realistic outdoor scene with snow Negative prompt:
Caption: oil painting of a pink dolphin jumping on the left of steam boat the Objects: [('a boat', [232, 225, 257, 1491), jumping pink dolphin', [21, 249, 189, 123]>]
Background prompt: oil painting the sea
10 Negative prompt:
20
21 Caption: cute cat and an angry dog without birds
22 Objects: I('a cute cat', [51, 67, 271, 3241), ('an angry dog', [302, 119, 211, 2281)]
23 Background prompt: A realistic scene
24 Negative prompt: birds
25
26 Caption: Two pandas in a forest without flowers
27 Objects: [('a panda', [30, 171, 212, 226]), ('a panda', [264, 173, 222, 221])]
28 Background prompt: A forest
Negative prompt: flowers
Caption: oil painting of living
wi without chairs with painting mounted the the cabinet Objects: [('a painting [88, 85, 335, 203]), ('a cabinet' [57, 368, 404, 201)), ('a flower vase [166, 222, 92, 108]), ('a flower vase'. [328, 222, 92, 108])] living
wall, cabinet below the painting, and two flower
33 Background prompt: oil painting
34 Negative prompt: chairs
Table K.2: Our in-context examples. We use fixed in-context examples for layout generation.
29


Reviews:
[{"review_id": "H6rNiSxOWJ", "reviewer": "Reviewer_Yihf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n- Two-Stage Training-Free Method: This approach is a key strength, enabling the model to leverage pretrained components without the need for additional training. This enhances efficiency and reduces the computational resources required for deployment.\n- Instruction-Based Editing: The paper introduces a method that can interpret complex instructions for image generation. This flexibility allows for more precise and creative control over the output, catering to a wide range of applications.\n- Utilization of Powerful LLM Models: By integrating large language models, the paper harnesses their advanced natural language understanding capabilities. This integration significantly improves the model's ability to interpret and respond to text prompts, leading to more accurate and contextually relevant image generation.\n\nWeakness\n\nA major concern regarding this paper arises from the methodology illustrated in Figure 5, which may be perceived as relatively straightforward by researchers familiar with diffusion models. This simplicity risks the method being classified as somewhat trivial. Except for this concern, I believe other issues are not critically detrimental, as they also represent the inherent challenges intertwined with the paper's strengths. Below are some of such weaknesses.\n\n- Dependence on Pretrained Models: Relying heavily on large language models for generating scene layouts might limit the model's adaptability to contexts or nuances not well-represented in the LLM's training data. \n- Computational Resources: While the method is training-free, the use of powerful LLMs and diffusion models might still require significant computational resources, potentially limiting its accessibility for users with less powerful hardware.\n- Generalization and Robustness: The paper's approach might face challenges in generalizing across diverse and complex scenarios, especially in cases where the text prompts are ambiguous or highly creative.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a novel two-stage generation process to enhance the prompt understanding capabilities of text-to-image diffusion models. This method, called LLM-grounded Diffusion (LMD), leverages a pretrained large language model (LLM) for generating scene layouts based on text prompts. These layouts guide an existing diffusion model to produce accurate images. Key contributions include:\n\n- Improved prompt understanding in text-to-image models using a training-free, two-stage generation process involving LLMs.\n- Introduction of a novel controller to steer diffusion models based on LLM-generated bounding box layouts.\n- Capability for instruction-based scene specification and support for multiple languages.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I found the concept of instruction-based specification, as discussed in this paper, to be particularly fascinating and a standout feature. I think it would be beneficial to focus more on this aspect, especially by highlighting its practical applications and the real-world impact it can have.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z7w9lMS1sm", "reviewer": "Reviewer_UKCc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n\n1. Extensive empirical evaluation with several baselines across benchmarks.\n2. State of the art performance across all reported metrics.\n\n**Weaknesses**\n\n1. Unclear technical novelty compared to existing training-free, layout-based guidance methods for text-to-image diffusion models.\n2. Unclear claims.\n3. Presentation could be made more precise.\n\nI will expand on these points in the requested changes, and I am looking forward to discussing with the authors to clarify my misunderstandings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Text-to-image diffusion models can fail to correctly capture the spatial relationships, absence, semantic attributes, or number of objects in a prompt. This submission presents a training-free method to guide the generation of text-to-image diffusion models with layouts generated by a large language model. \n\nThe effectiveness of the proposed method is evaluated quantitatively on two benchmarks: a novel one that includes negation, numeracy, attribute binding, and spatial relationships; and the existing T2I-CompBench. Samples are also compared qualitatively by means of a reader study. The proposed method achieves good performance across all benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Technical novelty**\n\nI appreciate the authors for citing and comparing with several recent methods that have been proposed to overcome these limitations of text-to-image diffusion models. In particular, when referring to the works by Bar-Tal et al., Chen et al., and Xie et al., the authors mention the presented submission differentiates itself by focusing on *instances* rather than *semantics*. I am not sure I fully understand this claim, could the authors expand on this? \n\nAs presented in Sec. 3.2, the submission falls within the framework proposed by Chen et al. and Xie et al. to guide diffusion by cross-attention optimization. Then, I do not understand what the difference between *semantics* and *instances* is in this context. Do the authors mean that the \"masked latents\" (as presented in Sec 3.2) are constructed per-instance? Is this the main differentiating factor from existing methods, i.e. the *Per-box masked latent generation* loop in Algorithm 1?\n\nIt could be helpful to precisely state the technical novelty of the proposed method rather than briefly presenting it at a high-level.\n\n---\n\n**What is the price for improved performance**\n\nThe proposed method achieves state-of-the-art performance across all tasks. What is the trade-off here compared to existing methods? Is the proposed method more computationally demanding because of the per-instance approach?\n\n---\n\n**Quantitative benchmarks**\n\nI was wondering whether the authors could motivate the choice of comparing with MultiDiffusion, Backward Guidance, and BoxDiff on the novel benchmark only and not also on T2I.\n\n---\n\n**Unclear claims**\n\nThere are a few claims throughout the paper that I am not sure I understand completely:\n\n* Sec. 3, introductory paragraph: \"given text prompt y, *potentially* by denoising\". What do the authors mean by \"potentially\" here? Are there other ways their proposed LMD method could generate images? If so, could the authors expand on this?\n\n* Sec. 3.2: \"and $V_i$ contains ...\". I am not sure I understand this sentence, and in particular what tokens are included in $V_i$. Could the authors clarify this?\n\n* Sec. 3.3: \"Since we edit the layout rather than the raw pixels ... as demonstrated in Fig. 6\". It is unclear to me what the difference between editing the layout and editing the \"raw pixels\" is. Furthermore, the sentence reads as Fig. 6 demonstrates certain editing instructions cannot be achieved by Brooks et al. However, Fig. 6 presents a case where LMD can follow editing instructions. I am not sure I understand what instructions LMD can follow that Brooks et al. cannot. Could the authors make this claim more precise?\n\n* Sec. 4.3, Layout to image stage: \"compared with training-free semantic control methods\". Could the authors clearly state what methods they are referring to?\n\n* Sec. 4.3: \"Switching the base diffusion model without hyperparam tuning\". What do the authors mean by \"hyperparam tuning\"? I was under the impression that LMD is training-free. Could the authors clarify in what capacity hyperparameter tuning is a part of their proposed method?\n\n* Sec. 6: \"Our method outperforms *strong* baselines\". I am not sure whether *strong* is appropriate here. In Table 2, all competing methods do not achieve more than around 60% in the proposed tasks.\n\n---\n\n**Presentation**\n\n1. Extensive use of jargon: throughout the paper, several terms are used without definition (e.g., latents, decoding, tokens, text features, hyperparam, noise-prediction network, cross-attention). These terms should be defined precisely to improve clarity.\n2. Equations are not fully defined: the Denoise function in Eq. 5 and the Compose function in Eq. 7 are not defined. I think the Compose function is defined in the Appendix but there it appears with a different name. A more extensive introduction on how Stable Diffusion achieves text-to-image diffusion should be included to better frame the role of cross-attention in Eq. 1.\n3. Figures are cited in the text out of order.\n4. Not all Appendices are cited in the text.\n5. Page 10, which contains 5 tables, significantly thwarts readability. Could the authors place tables where they are referred to in the text? Or move some tables to the appendix?\n6. Typos in Algorithm 1:\n\nShould the output be $x_0$ rather than $z^{comp}$?\n\nLine 8: what does the sentence \"with cross-attention map extracted to $A'^{(i)}_t$ mean?\n\nLines (18-19) and (20-21): could these pair of lines not be broken and put on one line? Also, I think $A^{(i)}$ is missing a $i=1, \\dots, N$? Or am I missing something about the algorithm.\n\n7. Tables 9 and 10 are cited in the Appendix but are missing from the text.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pj5INWHlA5", "reviewer": "Reviewer_nXop", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work presents substantial progress on instruction-following capabilities in image generation. The proposed method is relatively straightforward to implement as it relies on pretrained models. The paper is both well-written and provides a rich series of analyses and comparisons that reinforce the contribution.\n\nA few parts of the paper would benefit from additional analysis or explanation. I list the more prominent cases here followed by a list of fairly minor comments.\n\nSince no training is involved, the approach leans heavily on a few elements that were not or nearly not ablated. This includes constants such as $\\omega = 4.0$ in Eq. 3 and $\\lambda=2.0$ in Eq. 8, as well as the choice of few-shot examples in the model prompt. The work would benefit from showing how changing these, especially the former 2, impacts performance. The prompt, in particular, might benefit from a discussion focusing on the limitations of encoding images as scenes involving bounding boxes. For instance, does the prompt introduce a bias towards elements that are spatially distributed similar to the examples (or where the examples intentionally chosen for diversity)? Does using fewer examples in the prompt worsen performance? One might also imagine this working relatively poorly for prompts that involve hard-to-segment scenes, such as one actor standing in front of another, a large scene with many elements (e.g., dozens of people on a beach), or unusually interconnected/abstract elements (e.g., an Escher painting).\n\nAnother issue is the evaluation in Sec. 4.2.3. While collecting human feedback is valuable, 11 responses is too low a number to establish any meaningful trends. It is also not clear how 11 responses were used for 10 prompts -- was one prompt rated twice?\n\nOther issues:\n\n- In a few places, the work emphasizes its merit relative to other work in ways that would benefit from adjustment. Citing the lack of a need for training as a strength, for instance. One could imagine using images generated using this work as data augmentation in order to substantially improve a SD model, not unlike DALLE-3, in which case training would result in a model that could match its abilities without requiring an LLM. In another, it mentions related work that focuses on a \"limited closed set\", but does not acknowledge that a few-shot prompt of examples (as used in this work) confers biases of its own towards a limited (if not closed) set of demonstrations.\n- Perhaps I missed it, but I could not find how the model is prompted when modifying scenes. Presumably it is shown both the few-shot prompt and the prior turn of the conversation?\n- Alg. 1 on P17 has some unnecessary line breaks (e.g. on L18/19).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Current image generation models struggle to depict scenes that match rigid requirements, such as containing the right quantity or spatial ordering of elements. This work proposes an LLM-powered training-free method for doing so that leverages the LLM's ability to encode textual restrictions from the prompt. The LLM is first few-shot prompted to generate a scene layout using coordinate-based bounding boxes. Then, the diffusion model (here, variants of StableDiffusion) is carefully controlled to generate scenes that respect these layouts while generating otherwise equally natural looking images. The resulting images are both qualitatively and quantitatively found to be better aligned with instructions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In line with the weaknesses noted in the previous paragraph, please add/change:\n\n- The study in Sec. 4.2.3, either expanding the prompt set manually rated substantially (~50 examples would be good) or removing the section entirely.\n- Ablations for the $\\omega$ and $\\lambda$ terms to highlight the model's sensitivity to those choices.\n- A discussion and/or ablations surrounding the role of the (examples in the) few-shot prompt in the model's performance.\n- The discussion surrounding the benefit relative to related work, slightly. E.g., by noting the benefit of approaches that do use training as not requiring an additional LLM in the loop.\n- Minor changes/additions pointed out above (e.g., fix Alg. 1).", "broader_impact_concerns": "I did not find such issues. The broader impacts of this work are roughly in line with other work on image generation and work involving LLMs.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper address an important issue of diffusion based text-to-image generative models, where the generative image cannot well reflect the prompts. It introduce a two stage method by integrating LLM to generate object layouts. After generating the object layouts, a new algorithm is proposed to integrate multiple objects into the generation process.

All reviewers find the paper novel and interesting. There were some concerns regarding the presentation and experimental results. The authors provided detailed rebuttals and some extra experimental results to support the rebuttal. All reviewers appreciate the rebuttal and recommend acceptance.

In summary, this paper proposes a new training free method to address the inconsistency of generative images and prompts and achieve state of the art results. The impact could be signifiant. However, I highly recommend the authors to revise the current submission according to the reviews and add additional results into the revision as needed.
Decision: Accept (poster)