LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
Authors: Weixi Feng
Wanrong Zhu
Tsu-Jui Fu
Varun Jampani
Arjun Reddy Akula
Xuehai He
S Basu
Xin Eric Wang
William Yang Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language Models
Compositional Image Generation
3D scene synthesis
TL;DR: We propose a method to construct structured prompt for LLMs to understand and generate visual layouts in various domains.

Abstract: Attaining a high degree of user controllability in visual generation often requires intricate, fine-grained inputs like layouts. However, such inputs impose a substantial burden on users when compared to simple text inputs. To address the issue, we study how Large Language Models (LLMs) can serve as visual planners by generating layouts from text conditions, and thus collaborate with visual generative models. We propose LayoutGPT, a method to compose in-context visual demonstrations in style sheet language to enhance visual planning skills of LLMs. We show that LayoutGPT can generate plausible layouts in multiple domains, ranging from 2D images to 3D indoor scenes. LayoutGPT also shows superior performance in converting challenging language concepts like numerical and spatial relations to layout arrangements for faithful text-to-image generation. When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40\% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness. Lastly, LayoutGPT achieves comparable performance to supervised methods in 3D indoor scene synthesis, demonstrating its effectiveness and potential in multiple visual domains.

LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
Weixi Feng!*
Wanrong Zhu
Tsu-jui Fu'
Varun Jampani
Arjun Akula?
Xuehai He
Sugato Basu² Xin Eric Wang William Yang Wang*
University of California, Santa Barbara 2Google
University of California, Santa Cruz
com/ we wi-eiiiiin BeeggLayyoutaoe
Numerical Reasoning] There elephants standing beside pool of water.
Spatial Reasoning] carrol some knife cutting board.
StaboleDiffusion Attend and
LayoutGPT GUIGEN
StableDifussio
Attend and
LayoutGPT
(x2.1) 1)
Excite
(v2 1)
Excite
GLIGEN
[3D Living Room] Room Type: Living Room;
[3D Bedroom) Room Type: Bedroom;
Room Size: 7.7m 3.6m
Room Size: 3.0m 4.8m
ATISS
LayoutGPT
LayoutGPT
1
Furnitures
Out-of Bounda-B
Furnitures Overlapped
Figure I:
Generated layouts from LayoutGPT in 2D images and 3D indoor scenes. LayoutGPT can serve as a visual planner to reflect challenging numerical and spatial concepts in visual spaces.
Abstract
Attaining high degree of controllability in visual generation often requires intricate, fine-gra omn-igrined inputs like layouts. However, such inputs impose a substan- tial burden on users when compared to simple text inputs. To address the issue, we study how Large Language Models (LLMs) can serve visual planners by generating layouts from text conditions, and thus collaborate with visual gener- ative models. We propose LayoutGPT, method to compose in-context visual demonstrations style sheet language to enhance the visual planning skills of LLMs. Layou:GPT can generate plausible layouts in multiple domains, ranging from 2D images to 3D indoor scenes. LayoutGPT also shows superior performance in converting challenging language concepts like numerical and spatial relations layout arrangements for faithful text-to-image generation. When combined with downstream image generation model, LayoutGPT outperforms rext-to-image models/systems by 20-40% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness. Lastly, Layout- GPT achieves comparable performance to supervised methods in 3D indoor scene synthesis, demonstrating its effectiveness and potential in multiple visual domains.
* "equal contribution, correspondence to (weixifeng, wanrongzhu | (cc uusb. edu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Can Large Language Models (LLMS) comprehend visual concepts and generate plausible arrangments in visual spaces? Recently, LLMs have shown significant advancement in various reasoning skills
[50, 49] that remain challenging to visual generative models. For instance, text-to-image generation
(T21) models suffer from generating objects with specified counts, positions, and attributes [10, 24]. 3D scene synthesis models face challenges in preserving furniture within pre-defined room sizes [30]. Addressing these issues necessitates the development of compositional skills that effectively arrange components in coherent manner, accurately reflecting object specifications and interactions.
Visual layout an essential symbolic representation that has been widely studied as it reflects the compositions of a visual space [33, 53, 45, 34). For instance, layout generation models [21, 25, 17, 53, 23] can be combined with region-controlles image generation methods [56, 27] improve image compositionality [52]. But unlike LLMs, these models are restricted to discrete categories or have limited reasoning skills for complicated text conditions. Recently, LLMs like ChatGPT [37], are adopted as a centralized module of frameworks or systems where multiple foundational computer vision models are integrated. Through defined action items or API calls, LLMs can interact with visual generative models to extend the systems' capability into image generation tasks. [51].
Despite the advancement, existing approaches that involve the collaboration between LLMs and image generation models are either limited to executing the latter through program generation or using LLMs for language data augmentation for image editing [3]. Current LL--baseed systems fail to improve the compositional faithfulness of generated image by simply using T2I models through API calls. While one could additionally integrate models that synthesize images with the guidance of layouts [56, 27J, keypoints [27], or sketches [20, 57J, users still have to create fine-graines inputs on their own, leading to extra efforts and degraded efficiency compared to pure language instructions.
To address these challenges, we introduce LayoutGPT, a training-fres free approach that injects visual commonsense into LLMs and enables them to generate desirable layouts based on text conditions. Despite being trained without any image data, LLMs can learn visual commonsense through in-context demonstrations and then apply the knowledge to infer visual planning for novel samples. Specifically, we observe that representing image layouts is highly compatible with how style sheet language formats images on a webpage. Therefore, as LLMs are trained with program data, constructing layouts as structured programs may enhance LLMs' ability to "imagine" object locations from merely language tokens. Our programs not only enable stable and consistent output structures but also strengthen LLMs' understanding of the visual concepts behind each individual attribute value. When combined with a region-controlled image generation model [27], LayoutGPT outperforms existing methods by 20-40% and achieves comparable performance human users in generating plausible image layouts and obtaining images with the correct object counts or spatial relations.
In addition, we extend LayoutGPT from 2D layout planning to 3D indoor scene synthesis. With a slight expansion of the style attributes, LayoutGPT can understand challenging 3D concepts such as depth, furniture sizes, and practical and coherent furniture arrangements for different types of rooms. We show that LayoutGPT performs comparably to a state-of-the-ar art (SOTA) supervised method. Our experimental results suggest that LLMs have the potential to handle more complicated visual inputs. Our contribution can be summarized as the following points:
We propose LayoutGPT, a program-guided method to adopt LLMs for layout-based visual planning in multiple domains. LayoutGPT addresses the inherent multimodal reasoning skills of LLMs and can improve end-user efficiency.
We propose Numerical and Spatial Reasoning (NSR-IK) benchmark that includes prompts characterizing counting and positional relations for text-to-image generation.
Experimental results show that LayouIPPT effectively improves counting and spatial rela- tions faithfulness in 2D image generation and achieves strong performance in 3D indoor scene synthesis. Our experiments suggest that the reasoning power of L.LMS can be leveraged for visual generation and handling more complicated visual representations.
2

---

2
Related Work
Image Layout Generation Layout generation has been an important task for automatic graphical design for various scenarios, including indoor scenes [40, 46], document layouts [59, 60, 15]. and graphical user interface [8]. Previous work has proposed various types of models that need to be trained from scratch before generating layouts. LayoutGAN [25] is a GAN-based framework to generate both class and geometric labels of wireframe boxes for fixed number of scene elements. LayoutVAE [21] generates image layouts conditioned on an input object label set. Transformer- based methods are proposed to enhance flexibility in the layout generation process. For instance, LayoutTransformer [17] adopts self-attention to learn contextual relations between elements and achieve layout completion based on a partial layout input. BLT [23] proposes hierarchical sampling policy so that any coordinate values can be modified at the sampling stage to enable flexible and controlled generation. However, existing methods restricted to class labels and fail to reason over numerical and spatial concepts in text conditions. In contrast, I LayoutGPT can convert challenging textual concepts 2D layouts and generate free-form, detailed descriptions for each region.
Compositional Image Generation Recent studies have shown that text-to-image generation (T2I) models suffer from compositional issues such missing objects, incorrect spatial relations, and incorrect attributes [24, 2]. StructureDiffusion [10] proposes to adjust text embeddings by utilizing prior knowledge from linguistic structures. Aitend-and-Exciee [4] optimizes attention regions that objects attend on separate regions. Another line work strives to introduce extra conditions as inputs. For example, ReCo [56], GLIGEN [27], and Layout- Guiioanee [6] can generate images based on bounding box inputs and regional captions. [52] combines a layout generator and a method to achieve accurate generation results. While focus layout generation, also employ layout-to-image models to generate final images and show the effectiveness of LayoutGPT.
Indoor Scene Synthesis Indoor scene synthesis aims at generating reasonable furniture layouts in a 3D space that satisfies room functionality. Early work adopting autoregressive models requires supervision of 2D bounding boxes and other visual maps [40]. Later, SceneFormer [47] proposes to apply a set of transformers to add furniture to scenes. While previous work adopts separate models predict different object attributes, ATISS [38] demonstrates that single transformer model can generate more realistic arrangments while being more efficient. In this work, we investigate leveraging LLMs to achieve scene synthesis without any fine-tuning.
LLMs for Vision Language inputs have been an essential part of many vision language tasks [43, 11, 28, 14]. With the strong generalization ability of contemporary LLMs, recent work attempts to adapt the power of L.MSs on multimodal tasks [3]. For instance, multimodal chain-of-thouggh
[58] trained a model to incorporate visual inputs rationales for question answering. [22] proposes to learn translation parameters to map embeddings between visual and language domains such that an LLM can ground on both modalities. VisProg [18] and ViperGPT [44] use LLMs to design modular pseudocode instructions or executable Python programs to achieve visual reasoning. LLMScore [32] leverages LLMs to evaluate text-to- text-to-image models. Visual ChatGPT [51] proposes a prompt manager that supports the execution of various image generation models. In this work, we directly involve LLMs in the generation process by leveraging LLMs to design visual layouts through in-context learning and structured representations.
3 Method
3.1 Overview
Given condition C, the goal of layout generation is to predict set of tuples O {oyli 1, 2.......2, n.. where each tuple oy denotes the layout information of a 2D or 3D bounding box of object In image planning, is the input text prompt, 0, consists of category cy, bounding box location t, (zz. Mi) R² and bounding box size s; (w,, h2) R i.e. oj (cj. sy).) Similarly, in 3D scene synthesis, C specifies the room type and room size, о, consists of category cj. location t, E RR3, size s, RR3 and orientation r, R, i.e. o, (cy, ty, rz). While ej can modeled as discrete value, our method directly predicts the category text.
3

---

Instruction:
Description;
Toxt-conditoneed Image Generation
Available object categories (foe scene synthes15); T
Loyoutto-mmage
Input Condition C
Layout Generation: trser of Luidir that his clock Sonse Syntheds "Room Type: Sedroom, foom 3.5m
first Le contect exeeplart/
allmmoo.. saying
Syrtheas
Ioyout:
Rrom Tree Bettore
Roondaari ligth Siipu with
closk]
height: 81px:
Mbdbee led
witth 93pk
Structuu
length: 23300
with 757ps
harofy defined propprtiis
in-ocntent
exemplar cang (0
15pc
Normalize
hoght 113pc
exeeplanty
daylh Sópr
inference conditionny
ontrisiton: 50 degrees
Indoor Scene Synthesis
humppc tower of toa
tas ciect in
Layout
LUN Outputs/ towe!
LLM
naget, 148ps
Stee
Parse into lsyouts
Inpid prompt
tayoute CSS format
Figure 2: The overview process of our I LayoutGPT framework performing 2D layout planning for text-conditioned image generation or 3D layout planning for scene synthesis.
3.2 LayoutGPT Prompt Construction
As is shown in Fig. 2, LayoutGPT prompts consist of three main components: task instructions, and in-context exemplars in CSS structures with normalization.
CSS Structures In autoregressive layout generation, oj is usually modeled as a plain sequence of values, i.e. (C1, I1, 81, W1, h1, C2 .2) [17, 23]. However, such a sequence can be challenging for LLMs to understand due to underspecified meaning of each value. Therefore, we seek a structured format that specifies the physical meaning of each value for LLMs to interpret spatial knowledge. We realize that image layouts are highly similar to how CSS (short for Cascading Style Sheets) formats the layout of a wehpage and defines various properties of the mmg tag in HTMI. For instance, Zj, Uj corresponds to the standard properties left and top, while Wy. hy corresponds to w dth and height in CSS. As LLMs like GPT-3.5/4 are trained with code snippets, formatting image/scene layouts in CSS structures potentially enhances the LLMs' interpretation of the spatial meaning behind each value. Therefore, as is shown in Fig. 2, we place category name ej as the selector and map other attribute values into the declaration section following standard CSS styles.
Task Instructions & Normalization Similar to previous work in improving the prompting ability of LLMs [48, 42. 37], we prepend task instructions to the prompt to specify the task goal, define the standard format, unit for values, etc. Besides, as the common length unit of CSS is pixels (px), we normalize each property value based on a fixed scalar and rescale the value to maximum of 256px. As will be shown in later sections (Sec. 4.4 5.4), all three components play important roles in injecting visual commonsense into LLMs and improving generation accuracy.
3.3 In-Context Exemplars Selection
Following previous work [1, 54], we select supporting demonstration exemplars for in-context learning based on retrieval results. Given a test condition C, and support set of demonstrations D {(C,, ok)]k- 1, 1.2..... we define a function f(C,, between two conditions. For 2D next-conditioned image layout generation, we adopt the CLIP [39] model extract text features of C, (usually a caption) and the image feature of C₂ and measure the cosine similarity between them. For the 3D seene synthesis task where each room has length rl and width rw, we measure distance with [(Cx, C,) = ||r rl, 112 +ruw - rw; 12. We select supporting demonstrations with the top-k least distance measures and construct them as exemplars following the CSS structure in Fig. 2. These supporting examples are provided GPT-3.5/4 in reverse order, with the most similar example presented last.
that measures the distances

---

Table I: Dataset statistics and examples of the NSR-1K benchmark for image layout planning and text-to-image (T2I) generation with an emphasis on numerical and spatial reasoning.
Task
Type
Example Prompt
# Train
Val
Test
Single Category
"There are girafles photo."
14890
T21 Numerical
Two Categories
"Three potted plants with one vase in the picture."
7402
197
Reasoning
Comparison
"A picture three with few fire hydrants,
more than that fire hydrunts."
of cars
7402
100
Natural
"A fenced in pasture four horses standing around eating grass."
9004
351
T21 Spatial
Two Categories
"A dog the right of bench."
360
199
Reasoning
Natural
"A black laying top of bed new pillows." "
3.4 Image and Scene Generation
For (ext-conditioned image synthesis, utilize a layout-to-image generation model to generate images based on the generated layouts. As for each object layout in 3D scene synthesis, we retrieve 3D object based on the predicted category, location, orientation, and size following [38]. We directly render the scene with the retrieved 3D objects. See Sec. 4 & Sec. 5 for more details.
4 LayoutGPT for Text-Conditioned Image Synthesis
In this section, provide extensive evaluation of LayoutGPT for 2D text-to-image (T21) synthesis and compare with SOTA T2I models/systems. An ablation study is conducted to demonstrate the effect of individual components from LayoutGPT. We also showcase qualitative results and application scenarios of our method.
4.1 Experiment Setup
Datasets & Benchmarks To evaluate the generations in terms of specified counts and spatial locations, buman-written
we propose NSR- 1K, benchmark that includes template- basee and human written (natural) prompts from MSCOCO [29]. Table summarizes our dataset statistics with examples. For template-bar based prompts, we apply a of filters to obtain images with only 1-2 types of object and then create prompts based on object categories and bounding box information. As for natural prompts, we extract COCO captions with keywords to suit the task of numerical reasoning (e.g. "four") or spatial reasoning (e.g. "on top of"") and ensure that all objects from the bounding box annotations are mentioned in the caption avoid hallucination. Each prompt from NSR-IK is guaranteed to have corresponding ground truth image and layout annotations. Detailed benchmark construction processes are described Appendix B.I.
Evaluation Metrics To evaluate generated layouts, we report precision, recall, and accuracy based on generated bounding box counts and spatial positions [9, 16]. For spatial reasoning. each prompt falls into one of the four types of relations ([eef, right, top, below)) and we use the bounding box center for evaluation following PaintSkills [7]. To evaluate generated images, first obtain bounding boxes from GLIP [26] detection results and then compute average accuracy based on the bounding box counts or spatial relations. We also report CLIP cosine similarity between text prompts and generated images for reference. Detailed metric descriptions are listed in Appendix B.2.
Baselines As consider both layout evaluation and image evaluation, we compare LayoutGPT with end-to-end T2I models (Stable Diffusion [41], Attend-an aond--nd-Exiiie [4])² and two-stage systems that generate layouts first and then apply GLIGEN [27] the myyott-o-image model. We also evaluate ground truth layouts and tuman-drawn layouts as the theoretical upper bounds. The human-drawn layouts collected through crowdsourcing, in which specifically ask human annotators to draw layouts given text prompts. We slightly modify Layout'ransforma [17] as a baseline for supervised conditional layout generation. Detailed descriptions of baseline setups and human annotating discussed in the Appendix A and E.
2 Attend and and-anddd-eeee Excite uses Stable Diffusion (SD) as the generative backbone. For both end-to-end T21 models, report results SD v1.4 and SD v2.11
5

---

Table 2: Comparison of LayoutGPT with baseline methods in terms of counting and spatial correctness. Line 5-11 generates layout and adopts GLIGEN [27] for layout-guided image generation. "Human" (line 11) denotes layouts collected from human users given text prompts. Text bold shows the best results of LayoutGPT.
Numerical Rensoning
Spatial Reasoning
Layout Eval.
Image Eval.
Layout Eval.
Image Eval.
Precision
Recall
Accumecy
Aoc. (GI IP)
Sim.
Accumey
Text Image
Diffusion
0.256
0.252
Stable Diffusion (v2.1)
42.44
0.256
17.81
0.256
Altend-and-- Mieend-and-Excite (SD [14
38.96
0.258
24.38
0.263
Attend and Excite (SD v2.1)
45.74
0.254
26.86
0.264
Teu Layout Image
Layoorrrnnfoommr
LayouGaPT (GPT-335)
94.81
96.49
86.33
51:20
0.258
82.54
$2.86
012664
LayouGPT (Codox)
90.19
88.29
72.02
46.64
0.254
74.63
45.58
0.262
LayouGPT (GPT:3.5, chat)
0..61
LayouGPP (GPT-4)
78.36
86.29
78.43
35.64
0.261
91.73
60.64
0.268
10
GT layouts
100,00
100.00
100,00
53.23
0.256
100.00
62.54
0,261
0.258
Stable Diffusion
I PP
LavonGPTHOLIGGEN Human +GIIGEN
Stable Diffusion
L LavuttPPTHLLGEN HGLIGE
Three clocks train station under a concrete arch (Numetical)
Two aady bears stuffed snowman wearing hats (Numerical)
A yellow surfboard sits next to a bicycle on a brick sidewalk (Spatial)
cat is sitting on basket under bench (Spatial)
Qualitative comparison between Stable Diffusion, LayouGGPT, and human annotations regarding numerical (top row) and spatial reasoning (bottom row) skills.
Figure 3:
4.2 Evaluation Results
Quantitative Results As shown in Table 2, among the variants LayoutGPT (96-#9), GPT-3.5 achieves the best performance in numerical reasoning while GPT-4 performs the best in generating correct spatial positions. LayoutGPT outperforms LayoutTransformi (#5) by large margins, proving the strong moosl reasoning skills of LLMs. As for image-level evaluation. LayoutGPT surpasses end-to-end T2I models (#1 by 20-40% in GLIP-based accuracy and relatively 1-6% in CLIP similarity. Therefore, using layouts and faithful generation outcomes. In addition, LayoutGPT achieves similar layout accuracy as human users (numerical # 4 111 (86.33% v.s. 92.56%), spatial 49 #11 (91.73% 91.17%)), which implies its potential to spare users from drawing layouts manually. The discrepancy between layout accuracy and GI-IP-baed accuracy suggests that the bottleneck mainly stems from layout-guided image generation and GL.P grounding results.
intermediate representation indeed leads to more reliable
In addition, Lay binds attributes each object's bounding box with 100% accuracy HRS [2] color prompts. We further evaluate the attribute correctness rate (accuracy) on the final generated images when combining LayoutGPT with GLIGEN/ReCo. As shown in Table 3, our system largely improves the color correctness over Stable Diffusion with multiple objects.
Qualitative results We show the qualitative results of LayoutGPT and baselines in Fig. 3. LayoutGPT can understand visual commonsense such as the clock sizes train station (top left) or complex spatial relations between multiple objects (bottom right), while SD fails to generate correct numbers or positions. Besides, LayouGGPT demonstrates a similar layout design to human users (bottom left). Fig. 11 in the Appendix visualizes the results of attribute binding using LayoutGPT and ReCo [56].

---

Table 3: Color binding accuracy evaluated on prompts from HRS--Bench [2]. We follow the benchmark and use a classifier to identify the color of generated objects.
Attribute binding Accuracy (%)
Models
Prompts w/ 2 objects
Prompts w/ 3 objects
Prompts w/ 4 objects
Overall
SD1.4 Attend-and- Excite
18.57
10.10
11.36
12.84
31.43
19.19
20.45
22.96
LayoutGPT GLIGEN
19.19
14.77
LayoutGPT* ReCo [56]
37.37
34.09
36.96
+GLIGEN
T
+GLIGEN
LayouGiPT
+GLIGEN
Dense layout Planning
bathroom featuring sink and toilet
Different kinds of doughmuts display
A big brown cow walking down sidewalk near home
Text-based Inpainting
Grey and black cat sitting under bench, gray
cat silling on a basket under bench. a black and white cat with its tril curked in the basket
A cat lying on top of suitcase. black
with tail and its head down, looking the viewer wooden banch with two
lying its eyes closed
brown basket with handle the side wooden beneh curved backrest
purple suitcase with of blue, yellow, and white flower
Figure 4: Dense layout planning: LayoutGPT can generate rich objects categories in complex scenes for MSCOCO 2017 Panoptic prompts [29]. Text-based inpainting: LayoutGPT can generate free-form regional descriptions that are not mentioned in the global prompt.
4.3 Application Scenarios
By utilizing LLMS as layout generators, LayoutGPT can be applied to diverse set of scenarios for accurate and creative image generation.
Dense Layout Planning: In Fig. 4 (top), we apply random in-context examples from COCO17 panoptic annotations with 6- 15 bounding boxes per image. LayoutGPT can be applied scenarios that imply numerous objects (e.g. different kinds of donuts) or various categories (e.g. bathroom or street view). Though only few objects are mentioned in the prompts, LayoutGPT predicts layouts for the whole scene and imagines common objects that are usually visible in each scene.
In addition, the inherent language generation ability of LLMS enables method to generate fine-grained regional descriptions from coarse global prompts (Fig. 4 bottom). LayoutGPT can enrich the description of each object with details that are mentioned in the prompt, producing suitable outputs for models like ReCo [56].
Text-based Inpainting:
Counterfactual Scenarios: We test LayoutGPT counterfactual prompts provided by GPT-4 [35]. The in-context examples are randomly drawn from MSCOCO 2017[29], which greatly differs from the counterfactual prompts. As shown in Fig. 5, LayoutGPT manages to generate reasonable layouts on these challenging prompts and handles the relationship between objects well.
4.4 Ablation Study
Component Analysis Table 4 presents the component analysis of our CSS-style prompt spatial reasoning prompts. Comparisons between line 1-3 entails that the task instructions (#2) and CSS format (# effectively improve layout accuracy. Format in-context exemplars in CSS structures

---

Table 4: Ablation study of LayoutGPT (GPT-3.5) on spatial reasoning prompts. "w/ Instr.": with prepended task instructions. "w/ CSS": format in-context demonstrations in CSS style. "w/ Norm.": normalizing attribute values to integers by fixed size.
w/ Instr.
w/
w/
Layout-to-Image Model
Layout Eval
Image Eval
CSS
Norm.
Acc.
Acc. (GLIP)
CLIP Sim
55.12
34.35
0.259
2
78.23
47.92
0.263
3
80.82
51.38
0.264
44.10
26.43
0.257
/
GLIGEN [7]
81.84
52.08
0.264
73.36
44.88
0.262
7
76.61
47.56
0.263
82.54
52.86
0.264
82.54
31.02
0.258
Layout-Guidand [6]
10
layouts
100.00
33.92
0.257
a.rosserr parchod stop giant bppoiticc
of crato orating to curry
theestly oatrouus Loding fox benterdee sexing
nubber loaing on
Figure 5: Qualitative examples of LayoutGPT's performance on counterfactual prompts.
show a more significant effect on accuracy. Pairwise comparisons of line 5-7 support the argument that the CSS style is the most essential component. While solely applying normalization degrades accuracy in line 4, line 5 shows that it slightly improves the performance when combined with other components.
Model-Agnostic Property We show that LayoutGPT is agnostic to layout-guided image generation models in line 9-10 in Table 4. We feed the same generated layouts from LayoutGPT to Layout- Guidance [6] and compute image-level metries. Compared to using ground truth layouts (#10), LayoutGPT 9) shows minor gap in GLiP-based accuracy and comparable CLIP similarity score. The discrepancy in GLIP-based accuracy similar to that in Table 2, implying that the layouts generated by our method are agnostic to the downstream model.
5 LayoutGPT for Indoor Scene Synthesis
5.1 Task Setup
Datasets & Benchmarks For indoor scene synthesis, we use an updated version of the 3D-FRONT
dataset [12, 13] following ATISS [38]. After applying the same pre-processing operations, we end up with 4273 bedroom scenes and 841 scenes for the living room. only use rectangular floor plans of the test set for evaluation since LayouGPPT is not compatible with irregular ones. Hence, we end up with 3397/453/423 for train/val/test split of bedroom scenes and 690/98/53 for train/val/test split of living room scenes.
Evaluation Metrics We follow prior work [38] to report KI. divergence between the furniture category distributions of predicted and ground truth scenes. We also render scene images from four camera angles for each scene and report FID scores [19]. addition, we report out-of-bound rates, i.e. the percentage of scenes with furniture exceeding the floor plan boundary.
5.2 Evaluation Results
Quantitative Results The evaluation results are recorded in Table We provide a random baseline for comparison denoted as "Random Scenes", in which the scene is randomly sampled from the in-context exemplars for each inference run.
Notice that while the scenes in "Random Scenes" are sampled from the training set, the out-of-boundary rate larger than 0 since the 3D-FRONT dataset contains small portion of scenes with bound furniture.
8

---

Table 5: Comparison of LayoutGPT with ATISS on indoor scene synthesis. "Random Scenes" means
randomly sampling one training scene from the in-context demonstrations for each inference room sample. denotes results reproduced by us)
Bedrooms
Living Rooms
Models
Out bounds (L)
Div. ())
FID (L)
of bounds (L)
KL Div. (
FID (J)
Random Scenes
11.16
0.0142
23.76
9.43
0.1239
79.61
A11SS-[17]
49.88
0.0113
30.02
83,02
0.1054
85,40
LayourGPT (GPT:3.5)
43,26
0.1995
28.37
73.58
0.1405
76.34
LayourGPT (GPT-3.5, chat)
57.21
0.0846
29.66
81.13
0.2077
89.40
LayourGPT (GPT-4)
51.06
0.1417
29.88
64.15
0.1613
78.60
Master Bedroom
Master Bedroom
Living room
Living & Dining room
tmibar
Cvelagpal
Ou DoueL
LayoutGPT
Figure 6: Visualization of LayoutGPT across different types of rooms with different floor plan sizes.
For both bedrooms and living rooms planning, LayoutGPT attains lower out-of-bound rates than ATISS (bedrooms: 43.26% vs. 49.88 88%8 living rooms: 64. 16% vs. 83.02%), which verifies Layout- GPT's spatial reasoning ability in 3D environments. In addition, Layou has lower FID compared to ATISS (bedrooms: 28.37 vs. 30.02; living rooms: 76.34 vs. 85.40), which indicates that the planned scene has higher quality. Noted here that the living room split contains much more objects on average (11 for living rooms vs. bedrooms) and a low-resource split with only 690 training scenes. Therefore, while living rooms are challenging for both methods, LayoutGPT shows more significant improvement over ATISS as supervised methods tend to overfit in early epochs.
Meanwhile, ATISS performs better in terms of KL divergence, which means that the overall furniture distribution predicted by ATISS is closer to the test split. We observe that LayoutGPT tends avoid furnitures that are extremely rarely seen in each scene (e.g. coffee tables for bedrooms) as these objects appear less frequently in the in-context demonstrations. The limited in-context demonstration size also restricts LayoutGPT to have a universal observation of the furniture distributions.
Qualitative Results As shown in Fig. LayoutGPT manages to understand common 3D concepts, such as "the pendant lamp should be suspended from the ceiling" and "nightstands should be placed by the headboard of the bed" (bottom row). When given floor plan size for both living and dining rooms, LayoutGPT can also generate complicated 3D planning with dining tables and chairs on one side, and sofa, coffee table, and a TV stand on the other side (bottom right).
5.3 Application Scenarios
LayoutGPT can follow text captions to arrange furniture in the scene (see Fig. 7). When the captions enumerate complete list of furniture, LayoutGPT strictly follows the captions to generate the furniture and achieve KL Div. value close to zero.
Text- Synthesis:
Partial Scene Completion: Thanks to the autoregressive decoding mechanism, LayoutGPT can complete scene with partial arrangments such that the additional furniture remains coherent with the existing ones. Through in-context demonstrations, LayouGGPT learns critical (visual) commonsense such as visual symmetric (e.g. nightstands in Fig. (a)). positional relations (e.g. stool the end of the bed in Fig. 8 (b)), and room functions (e.g. desks and chairs in the dining area in Fig, 8 (d)).

---

living somm with threz coffee lables, a stand. mulli sest sula, dining table, four dizing chairs, console table
"A living swom with mulli seat wola, ooffee table, a stand dining table, four diming chains, ceiling and pendurt lamp." Room max 413px, withh
"A botroom with a douele bed, two wardrobes and pendant laup. Room Size max leapth 262ps, width 207pp
"A hodrcom with a double bed, wandrobe two tables, cabinet and pendant
pendant Room Size: length 2000 400ps
length 214px, 306gs
Figure 7: Generation of 3D scenes based on text captions that enumerate the furniture.
Partial
Completed
Partial
Completed
Partial
Completed
Partial
Completed
Bodroom, 3.8m 3.3m
Bedroom, 3.5m 4,2m
Living Dining zoom, 5.
Living Dining room,
3.9m
(b)
(c)
(d)
Figure 8: LayoutGPT can successfully complete a partial scene for different rooms. We provide three starting objects for bedrooms and seven objects for living rooms.
5.4 Ablation Study
Similar to Sec, 4.4, we study the effect of task instructions, CSS structure, and normalization on indoor scene synthesis (see Table 6). In contrast to our conclusion for 2D planning in Sec. 4.4, com- parisons between line 1-4 show that normalization (#4) 4) is the most critical component for sup- pressing the out-of-bound rate while the CSS structure is also effective. We observe that LLMs occasionally copy attribute val- ues directly from in-context emplars even though the room sizes are different. Therefore, normalizing all exemplars to the same scale can reduce the out-of-bound rate. CSS style facili- tates LLMSs to understand the physical meaning behind each attribute value and hence leads to almost the best result when combined with normalization (# 7).
Table 6: Ablation studies on LayoutGPT on the bedroom split for 3D indoor scene synthesis.
w/ Instr.
w/ CSS
w/ Norm.
Out of Bound |
KL Div. |
FID J
55.32
0.1070
56.83
2
54.85
0.1153
58.85
3
51.77
0.0776
55.62
4
46.57
0.1276
58.24
5
V
51.30
0.0741
57.64
V
46.81
0.0913
58.61
7
43.74
0.0848
57.70
8
v
43.26
0.0995
$6.66
6 Conclusion
In this work, we address new direction of generative model collaborations. Specifically, are interested in how Large Language Models (LLMs) can collaborate with visual generative models. To this end, we propose LayoutGPT, an approach that turns LLM into visual planner through in-context learning and CSS style prompts. LayoutGPT can generate plausible visual arrangements in both image space and 3D indoor scenes. LayoutGPT can effectively improve image compositions by generating accurate layouts and achieves comparable performance in indoor scene synthesis compared to supervised methods. Besides, LayouGPTT can improve user efficiency in image generation and serve essential part of unified system for all types of multimodal tasks.
10

---

Acknowledgments
The work was funded by an unrestricted gift from Google and a gift from the Robert N. Noyce Trust to the University of California via the Noyce Initiative. We would like to thank Google and the Robert N. Noyce Trust for their generous sponsorship. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the sponsors' official policy, expressed or inferred.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. ArXiv, abs/2204. 14198, 2022. 4
[2] Eslam Mohamed Bakr, Pengzhan Sun, Xiaogian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. HRS-Bench: Holistic, Reliable and Scalable Benehmark for Text-to- Image Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision
(ICCV), pages 20041-20053, October 2023. 6, 6, 19
[3] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. CVPR2023, 2022. 2
[4] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-exxinn
Attention-basee semantic guidance for text-to-image diffusion models. ACM Transactions
Graphics (TOG), 42(4))1-10, 2023. 3, 5, 6, 21
[5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Mísra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. ArXiv, abs/2107.03374 2021. 15
[6] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.033373 2023. 3,
[7] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3043-3054 2023. 5, 16
[8] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium user interface software and technology, pages 845-8544 2017.
[9] Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W.Taylor. Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction. In International Conference Computer Vision (ICCV), 2019. 5
[10] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-fire structured diffusion guidance for compositional text-to-image synthesis. ICLR, 2023. 2,
[11] Stella Frank, Emanuele Bugliarello, and Desmond Elliott. Vision-and-language or vision-for- language? on cross-modal influence in multimodal transformers. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9847-9857, 2021. 3
11

---

[12] Huan Fu, Bowen Cai, Lin Gao, Ling- Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2021. 8
[13] Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d furniture shape with texture. International Journal of Computer Vision, 129:3313-3337, 2021. 8
[14] Tsu-Jui Fu*, Linjie Li*, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. An Empirical Study of End-to-End Video- Ladeo-Language Transformers with Masked Visual Modeling. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[15] Tsu-Jui Fu, William Yang Wang, Daniel McDuff, and Yale Song. DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents. In Association for the Advancement of Artificial Intelligence (AAAI), 2022. 3
[16] Tsu-Jui Fu, Xin Eric Wang. Scott Grafton, Miguel Eckstein, and William Yang Wang. SSCR: Iterative Language-Basd Based Image Editing via Self-Supervised Counterfactual Reasoning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. 5
[17] Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry S Davis, Vijay Mahadevan, and Abhinav Shrivastava. Layouttransformer: Layout generation and completion with self-attention. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1004-
1014, 2021. 2, 3, 4, 5, 6
[18] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. CVPR 2023, 2022. 3
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 8
[20] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Cre- ative and controllable image synthesis with composable conditions. In International Conference on Machine Learning, 2023. 2
[21] Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochastic scene layout generation from a label set. In Proceedings of the IEEE/CVF Interna- lional Conference on Computer Vision, pages 9895-9904, 2019. 2,3
[22] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. In International Conference on Machine Learning, 2023.
[23] Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, and Irfan Essa. Blt: bidirectional layout transformer for controllable layout generation. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pages 474 490. Springer, 2022. 2, 3, 4
[24] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 2023. 2,
[25] Jianan Li, Tingfa Xu, Jianming Zhang, Aaron Hertzmann, and Jimei Yang. LayouGAN: Generating graphic layouts with wireframe discriminator. In International Conference on Learning Representationss 2019. 3.
[26] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975, 2022. 5, 17
[27] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2, 3, 5, 6, 8, 21
[28] Zhuowan Li, Cihang Xie, Benjamin Van Durme, and Alan Yuille. Localization vs. semantics: How can language benefit visual representation learning? arXiv preprint 22Xiv:2212.0028/ 2022. 3
12

---

[29] Tsung- Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollär, and C LLwwrnnce Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 6-12, 2014, Proceedings, Part 13, pages 740-755. Springer, 2014, 7,
[30] Jingyu Liu, Wenhan Xiong, lan Jones, Yixin Nie, Anchit Gupta, and Barlas Oğuz. Clip-layout: Style-consistent indoor scene synthesis with semantic furniture embedding. arXiv preprint arXiv:2303.08 033665 2023. 2
[31] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models, XrXiv preprint arXiv:2304.00 200402230,, 2023. 3
[32] Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. LLMScore: Unveiling the power of large language models in text-to-image synthesis evaluation. In Thirty-
seventh Conference on Neural Information Processing Systems (NeurIPS), 2023. 3
[33] Andrew Luo, Zhoutong Zhang, Jiajun Wu, and Joshua Tenenbaum. End-to-end optimization of scene layout. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3754-3763, 2020. 2
[34] Wan-Duo Kurt Ma, JP Lewis, W Bastiaan Kleijn, and Thomas Leung. Directed diffusion: Direct control of object placement through attention guidance. arXiv preprint arXiv:2302. 2023.
2
[35]
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023. 7, 15
[36]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.0215.55 2022. 15
[37] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022. 2, 4
[38] Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Information Processing Systems, 34: 1011-12026, 2021. 3, 5, 8, 20
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021. 4
[40] Daniel Ritchie, Kai Wang, and Yu-an Lin. Fast and flexible indoor scene synthesis via deep convolutional generative models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6182-6190, 2019. 3
[41] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674-10685, 2021. 5, 6, 21
[42] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai. Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
[43] Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language, pages 70-80, 2015. 3
13

---

[44] Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
2023. 3
[45] Bo Wang, Tao Wu, Minfeng Zhu, and Peng Du Interactive image synthesis with panoptic layout generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7783-7792, 2022. 2
[46] Kai Wang. Yu-An Lin, Ben Weissmann, Manolis Savva, Angel X Chang, and Daniel Ritchie. Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks.
ACM Transactions Graphics (TOG), 38(4):1-15, 2019. 3
[47] Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor scene generation with transformers. In 202/ International Conference 3D Vision (3DV), pages
106-115. IEEE, 2021. 3
[48] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. 4
[49] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. Survey Certification. 2
[50] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[51] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.046711 2023. 2,3
[52] Qiucheng Wu, Yujian Liu, Handong Zhao, Trung Bui, Zhe Lin, Yang Zhang, and Shiyu Chang. Harnessing the spatial-temn temporal attention of diffusion models for high-fidelity text-to-image synthesis. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7766-7776, 2023. 2,3
[53] Cheng-Fu Yang. WannCyuan Fan, Fu-En Yang, and Yu-Chiang Frank Wang. Layouttranforrmrr: Scene layout generation with conceptual and spatial diversity. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3732-3741, 2021. 2
[54] Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In AAAI Conference on Artificial Intelligence, 2021. 4
[55] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303. 11311, 2023. 3
[56) Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14246-14255, 2023. 2, 2, 6,7
[57] Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023. 2
[58] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multi- modal chain-of of-thoughtt reasoning language models. arXiv preprint arXiv:2302.00923, 2023.
3
[59] Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau. Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics (TOG), 38(4): 2019.
3
[60] Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document layout analysis. 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1015-1022. IEEE, 2019.
14

---

A Implementation Details
In this section, we provide detailed description of our prompt construction and instantiate instruc- tions examples.
Task instructions As is shown in Table 7, the specific task instructions start with verbalized de- scriptions of the task and are followed by the formal definition of the CSS style. As for the indoor scene synthesis, we additionally provide list of available furniture and the normalized frequency distribution for fair comparisons with the supervised method. Yet we discover that the provided frequency distribution has little effect the generation results, based on the trivial change in the KL divergence. In some cases, it is important to make L.LMs sample from a defined distribution instead of learning the distribution from in-context exemplars, which we leave for future work.
Table 7: The prepending instructions provided to GPT-3.5/4 during our LayoutGPT's 2D and 3D layout planning process. The instructions listed here are for the setting with CSS structure and with normalization.
Task
Instruction for GPT-3.554
2D Layout
Instruction:
Plamning
Given sentence prompt that will be used to generate an image, plan the layout of the image. The generated layout should follow the CSS style, where each line starts with the object description and is followed by its absolute position. Formally, each line should be like "object (width: ?px height: ?px: left: ?px: top: 7px: }". The image is 64px wide and 64px high. Therefore. all properties of the positions should not exceed 64px. including the addition of left and width and the uddition of top and height.
3D Layout Planning
Instruction:
Synthesize the 3D layout of an indoor scene from the bottom-up view. The generated 3D layout should follow CSS style, where each line starts with the furniture category and is followed by the 3D size, orientation, and absolute position. Formally, each line should follow the template: FURNITURE (length: ?px: width: 7px: height: 7px: left: ?px: top: ?px: depth: ?px: orientation: ?degrees:) All values are pixels but the orientation angle is degrees.
Available furniture: armchair, bookshelf, cabinet, ceiling chair, siildreen_abbiee cof- steee desk, double_bed, dressing_ dressing_ ttalle floor_lamp, kids_bed, nightstand, pendant Jamp, shelf, single_bed, bed, sofa, stool. table, tv tv_stand, wardrobe
Overall furniture frequencies: (armchair: 0.0045; bookshelf: 0.0076; cabinet: 0.0221; lamm:ngglamp: 0.062; chair: 0.024; children_cabinee cabinet: 0.0075; coffee_ _aflle 0.0013; desk: 0.0172; double bed: 0.1682; dressing_ehant chair: 0.0063; dressing table: 0.0213; floor_lamp: 0.0093; kids_bed 0.0079; nightstand: 0,2648; pendant_lani 0,5;, shelf: 0.0086; single_bed: bed: 0.021 sofa: 0.0018; stool: 0.012; table:
0.0201; tv rv_stand: 0.0308; wardrobe: 0557)
Base LLMs We use four variants of GPT models, (1) Codex [5] (code-davinei-022) an LLM that is fine-tuned with large-scale code datasets and can translate natural language into functioning code snippets: (2) GPT-3.5 [36] (text- wwweetttavviie davinci-003) which trained generate text or code from human instructions; (3) GPT-3.5-chat (gpt-3 3. 10-5--turie turbo) and (4) GPT-4 [35] (gpt -4), which are both optimized for conversational tasks. For the last two models, we first feed the in-context exemplars as multiple turns of dialogues between the user and the model to fit into the API design. However, we generally observe that GPT-3.5-chat and GPT-4 are not as strong as GPT-3.5 in learning from the in-context demonstrations, especially when the dialogue format follows a certain structure instead of free-form descriptions.
Hyperparameters For all LLMs, we fix the sampling temperature to 0.7 and apply no penalty to the next token prediction. For image layouts evaluation in Table 2, we fix the number of exemplars to 16 for numerical reasoning, and 8 for spatial reasoning, based on the best results of preliminary experiment. However, we do not observe significant gaps in evaluation results when using different amounts of exemplars (see Sec. B.4). For each prompt, we generate five different layouts/images using baselines or Lay and thus result in 3810 images for numerical reasoning and 1415 images for spatial reasoning in all reported evaluation results. As for indoor scene synthesis, we fix the number of exemplars to 8 for bedrooms and 4 for living rooms to reach the maximum allowed input tokens. We set the maximum output token as 512 for bedrooms and 1024 for living rooms as bedrooms have ~5 objects per room while living rooms have ~11 objects per room. We generate one layout for each rectangular floor plan for evaluation.
15

---

most two B LayoutGPT for 2D Layout Planning
B.1 NSR-IK Benchmark Construction
We rely the MSCOCO annotations to create NSR-IK with ground-truti layout annotations. Note that each image in COCO is paired with a set of captions and a set of bounding box annotations.
Numerical Reasoning We primarily focus the competence of T21 models to count accurately, i.c., generate the correct number of objects as indicated in the input text prompt. The prompts for this evaluation encompass object counts ranging from 1 to 5. To design the template-based T2I prompts, we initially sample possible object combinations within an image based on the bounding box annotations. We only use the bounding box annotation of image when there types of objects within the image. As result, the template-bar -aste prompts consist of three distinet types: (1) Single Category, wherein the prompt references only one category of objects in varying numbers; (2) Two Categories, wherein the prompt references two categories of distinct objects in varying numbers; and (3) Comparison, wherein the prompt references two categories of distinct objects but specifies the number of only one type of object, while the number of the other type is indicated indirectly through comparison terms including "fewer than", "equal number of", and "more than", As for natural prompts, we select COCO captions containing one of the numerical keywords from "one" to "five" and filter out those with bounding box categories that are not mentioned avoid hallucination.
Spatial Reasoning We challenge LLMs with prompts that describe the positional relations of two more objects. Our spatial reasoning prompts consist of template-based prompts and natural prompts from COCO. To construct remplate-basct prompts, we first extract images with only two ground-truth bounding boxes that belong to two different categories. Following the definitions from PaintSkill [7], we ensure the spatial relation of the two boxes belong to (left, right, above, below). Specifically, given two objects A, B, their bounding box centers (æ,, YA), (I yB) and the Euclidean distance d between two centers, we define their spatial relation Rel( (A, B) as:
above A below
if Ma " > sin(T/4)
if UB "A & sin( -T/4) d
ReI(A, B)
B on the left of A if ER=IA < cos (3T\3444)
B on the right of A
if EB-RA cos(TT44)
The definition basically dissects a circle centered at A equally into four sectors that each represent spatial relation. While the definition may not stand for all camera viewpoints, it allows us to mainly focus on the front view of the scene. Then, we utilize the category labels and the pre-defined relations to form prompt, as is shown in Table 1. As for the natural COCO prompts, we select prompts that contain one of the key phrases (the left/ right of, on top of, under/below) and ensure that the bounding box annotations align with our definition.
B.2 Evaluation Metrics
We denote the set of n object categories in the ground truth annotation as CGT C1, C2. Cu. where Fey Tey s.. ... Tc, represent the number of objects for each category. Additionally, we denote the set of m object categories mentioned GPT-3.5/44 layout prediction as Cpred = Ei,C2,... ón. where x'4 H' ..... z'e represent the number of objects for each category accordingly. If category Gg is not mentioned in Cpred, then z' is assigned a value of 0, and vice versa.
Σ Eminta,-.
1+0+2
Categories
cat
bed
pillow
precicion
Σ
1+0+3
75%
Ground Truth X,
2.
2
Prediction
1 0 3
Σ Xm
1+042
recall
60%
2+1+2
Figure closeup example of how we compute the layout automatic evaluation metrics for numerical reasoning.
16

---

Table 8: Closeup of various in-context example formats with ablated CSS structure and normalization for 2D layout planning.
CSS Structure
Normalization
In-context Example Format Demo
a teddy bear to the right of a book Layout:
Pronip::
teddy bear: 0.50, 0.71, 0.50, 0.15
book: 0.50, 0.61, 0.00, 0,26
Prompt: a teddy bear to the right of a book
Layout:
teddy bear [width: 0,50; height: 0.71; left: 0.50; top: 0.15; I
book (width: 0.50; height: 0.61; left: 0.00; top: 0.26;
Prompt: a teddy bear to the right of a book
Layout:
teddy bear: 32, 31,
book: 31. 38, 0, 16
Prompt: a teddy bear to the right of a book
Layout:
teddy bear (width: 32px; height: 45px; left: 31px; top: 9px:
book (width: 31px; height: 38px; left: Opx; top: 16px;)
The numerical reasoning ability of GPT-3.5/4 on layout planning assessed using the following metrics: (1) precision: calculated as Σi- Sm
min(2r oe is an indication of the percentage predicted
objects that exist in the groundtruth; (2) recall: calculated as 2n min( Plye) -i13 Peg
indicates the percent-
age of ground-truth objects that are covered in the prediction; (3) accuracy: In the "comparison" subtask, an accuracy score of is achieved when the predicted relation, whether it is inequality equality, between the two objects is accurately determined. For all other numerical subtasks, accuracy equals to if the predicted categories and object numbers precisely match the ground truth. In other cases, the accuracy is O. Fig. 9 shows an example of how we compute the precision and recall. The accuracy for this single example is 0 since the predicted object distribution does not match the ground truth in every category.
For spatial reasoning, we evaluate spatial accuracy based the LLM A-M-generated layouts and GLIP- based layouts. We adopt [26] finetuned on COCO to detect involved objects from the generated images and obtain the bounding boxes. For both types of layouts, based on the above definition and compute the percentage of predicted layouts with the correct spatial relation. For all evaluation benchmarks, similarity between the generated image feature and the corresponding prompt feature.
categorize the spatial relation
measure the CLIP similarity, which is the cosine
B.3
GPT-3.5/4 Prompting
In Sec. 4.4, investigate the impact of three components in the structured prompts: (1) Instruction, which examines whether detailed instructions explaining the task setup and the format of the sup- porting examples included in the prompt. (2) Structure, which evaluates the impact of different formatting settings on the presentation of the bounding box aspects of height, width, top, and left. The "w/ CSS" setting formats the aspects in CSS, while the "w/o CSS" setting presents the four aspects in a sequence separated by a comma. (3) Normalization, which investigates the effects of rescaling the bounding box aspects in the "w/ Norm." setting, while the "w/o Norm." setting presents the aspects as relative scales to the canvas size in floats that range from (0.
specified canvas size and presenting them as integers in pixels
Table 7 shows the detailed prepending instructions LayoutGPT provided to GPT-3.5/4 models during 2D layout planning. Table compares the formats of supporting examples with ablated structures and normalization settings.
For experiments in Sec. 4.3, to adapt to the nature of the Panoptic task, we add the following additional instruction when prompting LayoutGPT: "The objects layou might be dense, and objects may overlap with each other. Some objects might not huve been mentioned in the prompt, but are very likely to appear in the described scenario." " To generate counterfactual prompts for text-to-image generation, the following text prompt is provided to GPT-4: "Please provide few counterfactual prompts that
17

---

Table 9: The automatic metric scores Lay (GPT-3.5) with different in-context sample selection approaches. All values are in percentage (%).
Exemplar In-Context Exemplars
Numerical Reasoning
Spatial Reasoning
Selection
Layout Accuracy1
GLIP Accuracy
Layout Accuracy!
GLIP
Precision
Recall
Accuracy1
Fixed Random
16
64.83
92.71
87.66
47.10
80.14
47.07
4
88.93
95.02
76.17
50.20
85.30
51.66
Retrieval
93.32
95.63
82.68
50.58
82.54
52.86
16
94.81
96.49
86.33
51.25
82.40
51.09
In- In-context exemplar
In-context exemplar
In- Io-oontext exemplar
LayoutGPT PT
LayoutG
LayoutG PT
dog the water with timsee moich
close of eog neat haod
orange next each other on a buraap surface
an orange an applle and an prion
constracted with oven insidee of
kitcher refr igerator,
scoue, and an even
zafrigata
bird that has many
a close up of bird
The view of loste
close up of an sople acniff
Black and white shato
dog standing on too
laptop
Gosil
dog
Figure 10: Comparison between the most similar in-context exemplar and the generation results of LayoutGPT.
depict rarely seen the spatial relationship between the 80 MSCOCO object categories. An example would be monkey riding on top of a bird"".
B.4 Additional Experiments
Random In-Context Exemplars Empirically, selecting in-context exemplars can be critical for the overall performance of LLMs. Apart from our retrieval-augmented method in Sec. 3, we also experiment with fixed random of in-context exemplars. Specifically, randomly sample examples from the training (support) set D to form a fixed set of in-context demonstrations for all test conditions Cy. Therefore, the fixed random setting results in in-context exemplars that are unrelated to the test condition Cy. The minor gap between lines 1&5 in Table 9 verifies that LayoutGPT is not directly copying from the in-context exemplars in most cases. Fig. 10 further justifies the argument with layout visualization of the most similar in-context exemplars and the LayoutGPT outputs.
Number of In-Context Exemplars We take a closer look at the effects of the number of in-context exemplars in the prompt as shown Table For counting, we observe that the number exemplars is positively correlated with the counting accuracy. We conjecture that LLMs learn to make more accurate predictions for challenging prompts (e.g., comparison) by learning from more few-shot exemplars. As the layout accuracy also accounts for results where CSS parsing fails, we observe that the LLMs generate more consistent CSS-style code by learning from more examples. However, we cannot observe a similar trend in spatial reasoning prompts. We conjecture that LLMs only require as few as four demonstrations to learn the differences between the four types of spatial relations. The small optimal number of in-context exemplars implies that L.LMs already have 2D spatial knowledge and can map textual descriptions to corresponding coordinate values. Yet it is important to find proper representation to elicit such knowledge from LLMs as implied in Sec. 4.4.
18

---

Input LayoutPPTy. fluttering next to car window.
Input: A brown horse standing nexl to white truck.
Input: grey and brown cal sitting under purple umbrella.
Outpet Lay
Output:
Output:
red bird with wings fluttering
brown honse standing still
gray sitting position pupple umbrella
window open
a white truck with four black wheels
(fralled))
Figure 11: Attribute binding examples of LayoutGPT and generated images using ReCo.
Table 10: The layout performance on each numerical reasoning subtask. LayoutGPT (GPT-4).
Results reported on
Prompt Source
Subtask
Precision
Recull
Accuracy
Single Category
85.96
85.96
85.96
Template
Two Categories
85.14
85.04
66.60
Comparison
77.80
Natural Prompts from MSCOCO
72.08
87.1
82.79
Total
78.36
86.29
78.43
Performance on Numerical Subtasks Table 10 presents the performance layout generation in various numerical reasoning subtasks. Regarding template- beselatte-baedd prompts, the LayoutGPT demon- strates superior performance the "Single Category" numerical reasoning task, exhibiting precision, recall, and accuracy values around 86%. However, when it comes to the "Two Category" numerical reasoning task, while precision and recall experience minimal changes, the accuracy drops to 66%. For the "Comparison" subtask, the accuracy hovers around 78%. These outcomes indicate that Lay- outGPT encounters greater challenges when confronted with multi-class planning scenarios, whether the number of objects is explicitly provided indirectly implied through comparative clauses.
For natural prompts extracted from MSCOCO, noteworthy observation is the high recall accom- panied by relatively lower precision. This discrepancy arises due to the ground truth bounding box annotations encompassing only 80 object classes, whereas the natural prompts may mention objects beyond the annotated classes. Consequently, our LayoutGPT may predict object layouts corresponding to classes not present in the ground truth, which, despite lowering precision, aligns with the desired behavior.
Performance on Size Comparison Reasoning Task We evaluate LayoutGPT's reasoning ability regarding object size. We use the standard HRS benchmark [2] which designed for benchmarking compositional text-to-image models. HRS prompts for size reasoning contain comparison terms between randomly sampled common objects. The size relations described in HRS size prompts are often counterfactual and rarely seen (e.g., "a person which is smaller than chair and larger than horse". "a car which is smaller than a banana and chair and bigger than airplane") LayoutGPT achieves an accuracy of 98.0% / 99..1% % 192.1 % when the prompt involves size comparison between 2/3/4 objects. Meanwhile, the best size reasoning performance of nine rext-to-image models reported by the HRS benchmark only 31.1% / 0.2% / 0%. The results further verify that LayouGPT acquires decent reasoning ability on rare scenarios counterfactual prompts.
B.5 Failure Cases
Fig. 12 shows typical failure cases numerical and spatial relations. As previously discussed, we observe in Table 10 that numerical prompts that involves two type of objects ("Two Categories" and "Comparison") are more challenging to LayoutGPT and the image generation model. these subtasks, LayoutGPT tends to predict much smaller bounding boxes to fit all objects within the limited image space. The small boxes further challenge GLIGEN fit the object within the limited region, shown in Fig. 12 (right).
19

---

"A light shines on five clocks showing
"A vintage photo of a dog laying couch"
"Four zebras grazing enclosed area with shade trees"
LeyouRGPT
L-YRUIGAFE-LL AGEE
Laynutcrt
LayoueGPT CLICEN
Numerical failure
Spatial failure
Incompatible layouts
Figure 12: Typical failure cases of Layout and the generation results using GLIGEN.
In-context exemplar
PP
In-context exemplar
LayoutCPT
In-context exemplar
LayoulGPT
scene differences
30
Duplication
Modiffcation based on exemplars
Generation
exemplars
(new objects, significant differences in object sizes or locations)
100
200 sorted samples
300
400
Figure 13: Sorted scene differences between LayoutGPT generated scenes and the most similar in- context exemplars of 423 testing bedroom samples. We partition the distribution into three segments duplication
representing different behaviors of LayoutGPT. Duplication: The generated scene of the exemplar. Modification: LayoutGPT slightly modifies one exemplar as the generated layout. Generation: LayoutGPT generates novel scenes that highly different from the exemplars.
C LayoutGPT for 3D Scene Synthesis
Due to the limitation in datasets, the conditions are room type and room size instead of text descrip- tions. While ATISS [38] utilizes the floor plan image as the input condition, L.LMs with image inputs. Therefore, we convert the floor plan image into the specification of the room size. Therefore, the input conditions similar "Room Type: Bedroom, Room Size: max length 256px, max width 256px".
compatible
C.1 Exemplar Selection
Similar to Sec. B.4, we investigate the effect of using random set of in-context exemplars for indoor scene synthesis. When we apply 8 random bedroom layouts from training set in-context exemplars, the out-of-bound rate increases from 43.26% in Table 5 to 85.58%. The significant differences suggest that LayoutGPT heavily relies on rooms with similar floor plans to maintain objects within the boundary. Yet we verify that the generated layouts from LayoutGPT are not duplicates of the in-context exemplars most cases.
We first define training scene layout as a set of objects 5 = fof.... .ef......o. o and generated scene layout as 5 {of o Note that o, consists of category cj. location t, R size s, e R and orientation r, e R. i.e. oj = (cj, Sj. We define the scene difference D(-I.) between S and as
n
DSS
Σ min (1t) - t.]l
||s) $!111)-
1.
We set t/ s/ to if S does not have a single object that belongs to the same category as c. For each testing sample of the bedroom, compute the scene differences between the generated layout
20

---

Out-of-Boue Bound Fumiture
Overlapped Objects
Inharmonoous placement
Figure 14: Typical failure cases of LayoutGPT.
and all eight in-context exemplars and use the minimum value as the final scene difference. Note that "meters" instead of "pixels".
all parameters used for computation
We plot the scene differences of all 423 testing samples in Fig. 13. We empirically discover that scene difference below 1.0 means S is highly similar to S", which we conclude as duplication from in-context exemplars. A scene difference below 6.0 shows moderate differences object sizes or locations between two scenes, representing modification based S generate S Finally, scene difference larger than 6.0 represents new objects or significant differences object sizes or locations between the exemplar and the generated layouts, i.e. generation. Fig. 13 shows that 34/11 1 1/278 scenes belong to applicatto/moodiiaatioonggneeeee scenes have no out-of- out-of-bound furniture. Therefore, LayoutGPT is performing generation instead of duplicating in-context exemplars in most cases.
Among each category, 30/67/143
C.2 Failure Cases
While LayoutGPT achieves comparable results as ATISS, LayoutGPT cannot avoid typical failure cases as shown in Fig. 14, such out-of-boume furniture and overlapped objects. Fig. 14 (right) shows an incorrect placement of nightstands on the same side of the bed while they are commonly placed on each side of the bed headboard. Future work could focus more sophisticated in-context learning or fine-tuning methods to improve the LLMs' understanding of 3D concepts.
D LayoutGPT for 2D Keypoint Planning
addition to application 2D and layout planning, we investigate the feasibility of leveraging LayoutGPT for 2D keypoint planning to facilitate text-conditioned image generation. In this approach, we utilize LayoutGPT to predict keypoint distributions based on a given text prompt, and subsequently employ GLIGEN [27] for keypoint-to-ine image generation. The keypoint format used aligns with the specifications outlined in MSCOC02017 [29], focusing on 17 keypoints that correspond to the human methodology for selecting supporting examples in the context of 2D layout planning (Section B), we retrieve the k-most similar examples from the training set of MSCOC02017 and utilize these examples to provide keypoint distributions input to GPT-3.5/44. Table 11 presents an illustrative example of the input format employed for keypoint planning with GPT-3.5.
skeleton. Similar
Fig. 15 presents several illustrative examples that compare the images generated by condition- ing on keypoints planned by our LayouGPT with those generated by end-to-end models such as StableDiiflusion-v.. [41] and Attend-and-Fxcht [4]. In this preliminary demonstration, we observe that LayoutGPT exhibits promising potential offering inherent control over specific movements or actions through keypoint planning.
Nevertheless, it is worth noting that keypoints planning presents considerably greater challenges compared to bounding box layout planning, attributable to several evident factors. Firstly, keypoints planning necessitates the prediction of the positions of nodes, which is significantly more complex than the 2D layout planning involving four aspects or the 3D layout planning encompassing seven aspects. Secondly, the distribution of keypoints encompasses a much larger array of spatial relations due to the numerous possible body movements. In contrast, previous 2D layout planning tasks only involve four types of spatial relations. These inherent complexities render keypoint planning heavily reliant on in-context demonstrations. However, the limited availability of annotations pertaining to
21

---

Table 11:
The prompting input provided to GPT-3.5 for LayoutGPP keypoint planning.
Instruction:
Given sentence prompt that will he used to generate an image, plan skeleton keypoints layout of the mentioned objects. The skeleton keypoints include the following 17 nodes: nose. left_ye,
eye, right, eye, left car. right ear. left shoulder, right shoulder, left elbow, right elbow, left wrist, right. wrist, left hip, right hip, left knee. knee, left_ankle right_ankle, The generated keypoints layout should follow the CSS style, where each line starts with the keypoint node name and is followed by its absolute position.
eye,
Formally, each line should be like "node_ name [left: ?px: top: ?px; J". Please follow this format strictly. Do display other variation of formats. Notice that some keypoint nodes may not he visible the canvas. In such cases, simply put "node_name name (left: Opx; top: Opx; for the invisible nodes. The image 6dpx wide and 64px high. Therefore, all properties of the positions should not exceed 64px.
Prompt: a man on a surfboard in a river near a couple of trees and branches
Keypoints:
person#l:
nose [left: 36px; top: 33px;
left eye [left: 36px; lop: 33px; ]
right eye [left: 36px; wpp 33px;
left_ear (left: 37px; top: 33px;
right_eat ear (left: Opx; top: Opx:
left shoulderr [left: 38px; top: 34px: }
fightshoulddr (left: 36px; top: 35px;
left_elbont (left: 35px; top: 34px; ]
right elbow (left: 35px; top: 38px;
left wrist (left: 33px; top: 32px; ]
right_ wrist [left: 33px: top: 39px;
left hip (left: 39px; top: 39px; ]
right_hin hip (left: 37px; top: 40px;
left knee (left: 38px; top: 44px;
right knee (left: 3xp lop: 44px: ]
left ankle (left: 39px; lop: 49px;
right ankle (left: 37px; top: 48px; }
[MORE SUPPORTING EXAMPLES]
Prompt: a man leaning on a surfboard in the water riding wave Keypoints:
body movements in the MSCOCO dataset further exacerbates the challenges associated with reliable keypoint planning. Therefore, we leave the exploration of this potential direction to future research endeavors.
E
Ethical Statement
In addition to the layouts predicted by GPT-3.5/4, we also incorporate human-planned layouts as natural baseline for comparative analysis. To facilitate this, we provide annotators with an interface featuring blank square space where they can draw bounding boxes. Alongside the input text prompt, also present the noun words phrases from the prompt to human annotators, instructing them to draw a bounding box for each corresponding element. We intentionally refrain from imposing additional constraints, enabling annotators freely exercise their imagination and create layouts based on their understanding of reasonable object arrangements. To compensate annotators for their efforts, we offer payment rate of SO.2 US dollars per Human Intelligence Task (HIT). The average completion time of approximately 30 seconds per HIT, which corresponds to an average hourly payment rate of $24.
F Limitations
The current work has several limitations that provide opportunities for future research. Firstly, while this work focuses on 2D and bounding box layouts and makes preliminary attempt at keypoints, there exist various other methods for providing additional spatial knowledge in image/scene
22

---

A chimpanzes holds foothbrush their hand
person standing in some water flying kite
SubleDiflsion x2.1
Attend and Excto
LayoutGPT GUSEN
SubleCiluuion v2.1
Artend-and Excit
LayouIGPT + GUIEN
A close up of monkey driving motoreycle road
woman giasses holdling laptop couch
Sibls v
LayourGPT . @l IENN
StableCifluson v2
Attend-and-Bece
LayoueGiP GUXSIN
Figure 15: Plausible examples of LayouGPT((PPT4) planning keypoints distributions before con-
ducting text-conditioned image generation.
generation, such segmentation masks and depth maps. Future work could explore integrating LLMs with these alternative visual control mechanisms to broaden the scope of visual planning capabilities. Secondly, the current work primarily addresses visual generation tasks and lacks unified framework for handling other visual tasks like classification or understanding. Extending the proposed framework to encompass a wider range of visual tasks would provide more comprchensive and versatile solution. Thirdly, this work is a downstream application that attempts to distill knowledge from LLMs' extensive knowledge bases. Future research could explore more fundamental approaches that directly enhance the visual planning abilities of various visual generation models. By developing specialized models that are explicitly designed for visual planning, it may be possible to achieve more refined and dedicated visual generation outcomes. Overall, while the current work demonstrates the potential of using I.LMs for visual planning, there are avenues for future research to address the aforementioned limitations and further advance the field of visual generation and planning.
G Broader Impact
The utilization of LLMs for conducting visual planning in compositional 2D or 3D generation has significant broader impacts. Firstly, LLMs alleviate the burden human designers by simplifying the complex design process. This not only enhances productivity but also facilitates scalability, as LLMs efficiently handle large-scale planning tasks. Secondly, LLMs exhibit remarkable capabilities in achieving fine-grained visual control. By conditioning on textual inputs, LLMs can easily generate precise and detailed instructions for the desired visual layout, allowing for precise composition and arrangement of elements. Moreover, LLMs bring a wealth of commonsense knowledge into the planning process. With access to vast amounts information, LLMs can incorporate this knowledge to ensure more accurate and contextually coherent visual planning. This integration of commonsense knowledge enhances the fidelity of attribute annotations and contributes more reliable and realistic visual generation outcomes.
It is worth noting that this work represents initial foray into the realm of visual planning using LLMs, indicating the potential for further advancements and applications in this area. As research in this field progresses, we can anticipate the development of more sophisticated and specialized visual planning techniques, expanding the scope of LLMs' contribution to diverse domains, such as architecture, virtual reality, and computer-aided design.
H Additional Qualitative Examples
We present additional visual showcases to demonstrate the capabilities of LayoutGPT in different contexts. Fig. 16 showcases examples related to 2D numerical reasoning, Fig. 17 illustrates examples of 2D spatial reasoning, and Fig. 18 displays examples of 3D scene synthesis. These showcases offer further insights into the effectiveness and versatility of our approach across various domains.
23

---

light shines five clocks showing times in different
SD2.1
Attend Excite
GPT-3.5
+GLIGEN
GPT-3.5,
+GUGGNN
GPT-4
+GUGGN
These three birds walking along the beach looking for food
502
Attend Excite
GPT-3.5
+GUGEN
GPT-3.5, chat.
+GLIGEN
GPT-4
+GLGGEN
photo station with trains on the tracks.
There are three elephants standing beside poal of water.
SD2.1
Attend-n d-r--d------ae Exclte
GPT
GLIGEN
GPT-3.5 chat
GGIGGN
GPT4
GUIGEN
Reshann
alpohont
three airplanes that landed near large
SD2.1
Aitend-n-xxille
GPT-3-5
+Gi IGFN
GPT:3.5, chat
KGFN
GPT4
GII IFF
airotote ain
butterfly sculptures
GPT33.5
A lone bicycle next to bike path near the water.
1
Attend-n-Exclle
GPT-3.5, chat
Alcycle
dog laying bed near laptop
SD2.1
Attendin Excite
GPT-3.5
+GLIGEN
GPT-3.5, chat
GUGGN
GPT-4
+GLIGEN
Innlop
Figure 16: Qualitative examples of variants of LayoutGPT on numerical reasoning prompts.
24

---

black cat laying in the sun under bench
5D2.1
Attend--Exaitee
GP7.35
+GLIGEN
GPT-3.5 chat
GGUGEN
GPT-4
GLIGEN
cup to right a donut
S02.1
Attendin Excte
GPT-3.5
GGIIGNN
GPT-3 5, chat
GUIGNN
GPT-4
+GUGEN
Amoutt
cup
black laying top bed next to pillows.
GPT:3.5
GPT-3.5,
GUIGEN
Several glass beer bottles under park bench.
3D23
Extie
GPT.3.5
GLIIGN
G chat
GGIGEN
GPT4
+GUGEN
chat
to the left of
Attend-o-Excile
GLIIEN
GPT.3.5 chat
GUIGEN
GUGEN
ebo
banana sitting on top of cup on desk.
SD2.1
Attend--ffaitt
GPT.3.5
+GLIGEN
GPT.3.5, chat
GLIGEN
GPT44
GLIGN
A cat sitting on wood floor under white and blue umbrella,
SD2
GPT:3.5
+GLIGEN
GPT.3.5. chat
+GUIGNN
GPT-4
GUICEN
Figure 17: Qualitative examples of variants of LayoutGPT on spatial reasoning prompts.
25

---

GPT-3.5
GPT-33.5-cht
GPT-4
Figure 18: Additional qualitative examples of variants of LayoutGPT bedroom scene synthesis.
26


Reviews:
[{"review_id": "9429WPmpPd", "reviewer": "Reviewer_ykMz", "summary": "This paper proposes LayoutGPT, a method to compose in-context visual demonstrations in style sheet language to enhance the visual planning skills of LLMs. As the first work to use LLMs to generate layouts from text conditions, LayoutGPT can generate plausible layouts for 2D images and 3D indoor scenes, including challenging language concepts like numerical and spatial relations. Those generated layouts can be further used for image generation. When combined with a region-controlled image generation model, LayoutGPT outperforms existing text-to-image generation methods by 20-40% and achieves comparable performance as human users in generating plausible image layouts and obtaining images with the correct object counts or spatial relations.", "questions": "Please refer to the questions in the weakness section.", "limitations": "The authors mention the limitations in the supplementary materials.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This is the first work to explore the ability of LLMs for layout generation. It reveals the spatial reasoning ability of LLMs and might inspire future explorations in this direction.\n2. The training-free approach is easy to adopt for various applications.\n3. The presentation is clear and easy to follow.", "weaknesses": "1. Although this was no such exploration before, the proposed approach is a straightforward application of LLM. It would be better and more inspiring if authors could provide some in-depth analysis of the spatial reasoning abilities of LLM.\n2. It is unclear if layoutGPT is robust to the selection of in-context exemplars or not. What's the size of the reference set? What if the text condition describes a rare scenario that does not appear in the reference set? Experiments in the supplementary material show that the performance is sensitive to the number selected in-context exemplars. \n3. The evaluation is conducted only on NSR-1K for numerical reasoning and spatial reasoning in text-to-image synthesis and ATISS for indoor scene synthesis. Although the authors claim that layoutGPT can be used for accurate attribute binding and text-based inpainting, only visual results (fig.4) are shown and there are no quantitative experiments on such datasets to demonstrate such abilities and applications. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n> * Weakness 1: straightforward application and in-depth analysis of spatial reasoning ability\n\nOne of our main contributions is to combine style-sheet program synthesis with LLMs and in-context learning as the CSS style language inherently share similarity with bbox-based planning for visual generation. Different from previous work [48,52] that uses simple representation (x1,y1,x2,y2), our ablation study show that CSS structure elicits much stronger spatial reasoning skills from LLMs (see Table 3). This finding requires the awareness of shared properties between LLMs’ pretraining data (CSS-involved programs) and image/3D layout representations. While “w/ instruction” can achieve considerable spatial accuracy (lines 2&6 in Table 3), CSS structure is more important by comparing lines 6-8, which has been analyzed in Sec. 4.4. Therefore, we respectfully disagree that LayoutGPT is a straightforward application of LLMs.\n\nIn addition, we substantiate the claim based on Figure 3 in the attached PDF response. Fig.3 (top) shows that CSS enables LayoutGPT to not only avoid unreasonable overlap but accurately generate spatial relations involving multiple objects. We hypothesize that LLMs understand the integer values in the 2D space much better because CSS explicitly declares the property names for these values.  Fig.3 (bottom) shows the spatial reasoning ability across different versions of GPTs beyond quantitative differences. It’s surprising that GPT-4 can generate the shape and position of “a straw” precisely given that *no straw box examples are provided in the in-context demonstrations*. We hypothesize that either GPT-4 is pre-trained on a more comprehensive collection of layout data or the image branch training benefits layout generation as language tokens.\n\n\n> * Weakness 2: Size of ICL and robustness to exemplar selection\n\nFor evaluation results in the main paper, we used eight (k=8) in-context demonstrations for image layouts and eight/four (k=8/4) for 3D bedroom/livingroom synthesis. \n\nIn our supplementary Table 3, we show that even with random ICL exemplars or fewer exemplars (k<8), LayoutGPT can achieve comparable layout accuracy and GLIP accuracy for both numerical and spatial prompts. LayoutGPT tends to generate extra objects with random exemplars due to its hallucination nature, yet this does not harm the overall accuracy of counting and spatial reasoning. Besides, “train platform” and “straw” in additional rebuttal PDF material Fig. 3 never appear in the exemplars and are not part of the COCO annotations. However, LayoutGPT still manages to elicit the ability of LLMs to generate their 2D boxes. Therefore, we believe that our method is robust to rare scenarios and various sizes of exemplars.\n\n\n> * Weakness 2: LayoutGPT’s performance on rare scenarios\n\nPlease refer to the general response. \n\n> * Weakness 3: Attribute binding & text-based inpainting.\n\nPlease refer to the general response for attribute binding clarification and evaluation.\n\nText-based inpainting is a creative application scenario and has no standard benchmark/evaluation yet. It shows the flexibility and potential of LayoutGPT in “imagining” intricate object-level descriptions for efficient and creative image generation. \n\n> * Weakness 3: More quantitative evaluation of LayoutGPT’s reasoning ability\n\nPlease refer to the general response regarding LayoutGPT’s counterfactual prompts. We also report additional quantitative results on LayoutGPT’s size reasoning performance.\n"}, {"review_id": "Z0kUDUyF2x", "reviewer": "Reviewer_1i69", "summary": "This paper introduces LayoutGPT, a training-free approach that injects visual commonsense into LLMs and enables generating plausible 2D images and 3D scenes conditioned on layouts based on text conditions. Specifically, the authors experiment with four variants of GPT models: Codex, GPT-3.5, GPT-3.5-chat and GPT-4 and showcase that LLMs can produce meaningful 2D/3D layouts using a CSS (Cascading Style Sheets) format, where every object is modelled as labelled bounding box, parametrized with three random variables indicating its category, size and location. For the case for 3D indoor scene synthesis, they parse the layouts into 3D scenes by simply replacing the bounding boxes with 3D objects from a library of assets. For the case of 2D image synthesis, they rely on GLIGEN that is a layout-to-image model to convert the generated layout to a 2D scene. For the case of 2D image synthesis, the authors compare their model to Stable Diffusion and Attend-and-Excite and evaluate the generated layouts based on the precision, recall and accuracy of the generated bounding boxes. To measure whether the generated image matches the provided text description, the authors report the CLIP/GLIP cosine similarity between text prompts and the generated images. For the task of 3D indoor scene synthesis, the authors compare their model to ATISS and measure the generation quality by reporting the KL-divergence between the object category distributions in the ground-truth and the generated scenes. For both tasks, LayoutGPT outperforms most baselines on most metrics and from the qualitative results, it seems that LayoutGPT faithfully generate layouts that match the input conditioning.\n\nOverall, I think this is a nice work that introduces an elegant way for using LLMs for 2D and 3D layout synthesis. I think that the idea of representing scene layouts in the CSS format is very intuitive and greatly simplifies the task of layout synthesis. My main concern, as also discussed in the Weaknesses section, is related to whether the proposed model can robustly generate (i) layouts with more complex text conditioning for the case of the image synthesis, (ii) 3D scenes conditioned on detailed scene descriptions clearly describing the how many and what objects should be placed in the scene.\n", "questions": "1. I am wondering whether the LLMs can produce diverse layouts conditioned on the same text conditioning. I believe that this is an interesting experiment that the authors should include in their evaluation. In addition, another valuable analysis would be to compare existing GPT models w.r.t. their capabilities for generating diverse layouts conditioned on the same input prompt. Being able to generate diverse layouts is a very important trait of existing models, hence I think the authors should provide additional experiments that demonstrate whether this is possible or not.\n\n2. Looking at the Experimental Evaluation and in particular the Image Synthesis results, I am wondering how robust is the proposed model if the input text prompt contains larger descriptions. Looking at all the results both in the main paper and in the supplement, I think that the authors show layout generations with at most 5 objects (see Fig 3., top row, right column). Have the authors tried to condition the layout generation with more detailed text prompts with more objects? How good would their model work? \n\n3. In Table 1, the authors mention that their proposed NSR-1K benchmark contains several text descriptions that have comparisons e.g. \"A picture of three cars with a few fire hydrants, the number of cars\nis more than that of fire hydrants\". I checked both the main paper and the supplementary but I was not able to find any conditioning like this. I think it would be great if the authors can provide some examples that show that their model works well with these more challenging text conditioning.\n\n4. Can the authors clarify why LayoutGPT cannot work with floor plans of various shapes. In L219-220 of  the main paper they state that it is not compatible with irregular floor plans but I am not sure why this is really an issue?\n\n5. For the quantitative evaluation in Section 5.1 the authors should also mention the image resolution of the rendered images, used to compute the FID scores. Moreover, they mention that to compute the FID score they render scene images from four camera angles (L224-225), are these angles random per scene? I think it is important that the authors clarify this for reproducibility purposes.\n\n6. Some paper references that are missing that I think the authors should include in their final version of their paper are listed below:\n* Variational Transformer Networks for Layout Generation, CVPR 2021\n* BLT: Bidirectional Layout Transformer for Controllable Layout Generation\n* LayoutDM: Discrete Diffusion Model for Controllable Layout Generation, CVPR 2023 (this is a concurrent work but still it might be good to add it in the reference list)", "limitations": "The authors discuss the limitations of their work and show several failure cases in their supplementary material. In addition, they also discuss potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. To the best of my knowledge, the idea of using LLM for 2D and 3D layout synthesis is novel and the authors clearly demonstrate that LLMs can produce complex 2D images and 3D scenes in a CSS format. Unlike other concurrent works that try to use LLMs for similar tasks, I think this model is simpler and more generic, hence it can be applied on both 2D scene synthesis and 3D scene synthesis.\n\n2. I particularly liked that the authors provide results on both a 2D and a 3D task. They compare their model with several strong baselines and showcase that the proposed model can consistently produce 2D images/3D that match the input conditioning. Moreover, from the quantitative evaluation, we note that the proposed training-free method achieves state-of-the-art performance w.r.t. most metrics for both tasks.\n\n3. I appreciated the additional ablations as well as the various implementation details provided in the supplementary. In addition, I think that also the proposed NSR-1K benchmark, can potentially be very useful for various tasks. Although the authors don't mention whether they plan on releasing these benchmark, I would like to encourage them to do so as I think such benchmarks can greatly benefit the research community.\n", "weaknesses": "1. For the case of the 3D indoor synthesis task, I am wondering whether the authors tried to condition the scene generation on more detailed text descriptions that go beyond simply specifying the room type and the size of the room. For example, given a description like \"a bedroom with one double bed, two nightstands and 1 wardrobe\", would LayoutGPT be able to generate a layout that matches this description? I assume that to some extend this should work quite well so I am not sure why the authors did not provide these type of results. I think it would be beneficial for the paper to include them for the final version of the paper.\n\n2. For the experimental evaluation of the 2D scene synthesis, the authors report precision, recall and accuracy. Is there a reason why not also report MeanIoU on the bounding box parameters? I am not sure whether I am missing out something but I think this metric is very important as the model generates bounding boxes in practice. In addition, I think that the section B2 in the supplementary that discusses the Evaluation metrics and in particular the accuracy computation is not very clear. I think it is good to polish this section a bit.\n\n3. Although, I appreciate that the authors proposed a new benchmark for their image synthesis experiment, I think they should have also evaluated their model on the 2017 Panoptic version of COCO dataset that has been previously used by several generative models that perform layout generation. From the description in L152-161, it is not clear to me whether the panoptic version of the COCO dataset is included in the proposed benchmark. Can the authors please clarify this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading and insightful comments.\n\n> * Weakness 1: 3D layout planning with detailed description \n\nThanks for your suggestions. Please note that 3D-FRONT does not have ground truth captions for the rooms. Besides, existing work mostly conditions on room types or floor plans instead of using language descriptions. As a reference, we generate template-based captions (e.g. “bedroom with a double bed, two wardrobes, and a pendant lamp.”) and use them as conditions. As shown in the table below, the near-zero KL Div. value indicates that LayoutGPT faithfully follows the descriptions to generate the type and amount of objects. However, captions do not provide additional information to improve out-of-bounds or FID scores. Please see Fig. 2 in additional materials for visualization. \n\n|                                | Bedroom       |         |       | Living room   |         |       |\n|--------------------------------|---------------|---------|-------|---------------|---------|-------|\n|                                | Out of bounds | KL Div. | FID   | Out of bounds | KL Div. | FID   |\n| LayoutGPT (GPT-3.5, w/caption) | 54.27         | 3.21e-7 | 27.68 | 77.36         | 3.54e-5 | 76.87 |\n\n\n> * Weakness 2: MeanIoU for 2D layout evaluation \n\nWe did not use MeanIoU to evaluate layout performance because the nature of the task is generation instead of prediction. There may be inexhaustible possible valid layouts for the same prompt. Besides, the prompt does not describe object sizes or specific locations. Therefore, the MeanIoU between the prediction and the reference image may not adequately evaluate the “correctness” of the prediction in terms of numerical or spatial reasoning.\n\n> * Weakness 2: Metric for 2D layout evaluation\n\nThanks for your suggestion. We will revise the metric section and provide some visualization examples. The accuracy is basically the percentage of test samples that end up with the correct numerical or spatial relations based on the layout (Layout Acc.) or the detected layout (GLIP Acc.).\n\n\n> * Weakness 3: Regarding COCO2017 panoptic subset \n\nOur NSR-1k benchmark is built on COCO2014 with 80 categories. Thank you for your suggestion and we evaluate LayoutGPT on the Panoptic benchmark. Please see our response to your “Question 2” below for detailed setup and experimental results.\n\n\n> * Question 1: Diversity of the layouts planned by LayoutGPT \n\nBeyond our observation of the diversity, we also generate five different layouts for each prompt and compute the standard deviation (std) of bounding box sizes and locations. The normalized bounding box sizes have a std of +-0.151 and the normalized locations have a std of 0.083. Please also note that users can adjust the diversity by setting different temperatures for the LLMs decoding stage, which further guarantees the diversity of layouts.\n\n\n> * Question 2: Prompting LayoutGPT with more objects \n\nWe test LayoutGPT on 500 examples randomly sampled from the validation set of COCO2017 Panoptic with 6~15 annotated bbox (out of the consideration for GPT length limit). We retrieve 8 supportive examples from the train set during in-context learning.\n\nTo adapt to the nature of the Panoptic task, we add the following additional instruction when prompting LayoutGPT:\n“The objects layout might be dense, and objects may overlap with each other. Some objects might not have been mentioned in the prompt, but are very likely to appear in the described scenario.”\n\nWe generated images using GLIGEN and the LayoutGPT's output and achieve 86 FID score compared to 90 FID using LayoutTransformer[17]+GLIGEN\n\nIn Fig.4 of the additional rebuttal PDF material pdf, we show the dense layout predicted by LayoutGPT together with GLIGEN’s visualization. Results show that LayoutGPT can be smoothly applied to more complicated scenarios with more objects (middle: loads of donuts) and with more categories (left: indoor scene; right: outdoor streetview).\n\nIt is worth noting that, even though the prompt may only mention a few objects, LayoutGPT is able to predict the layout of the whole scene (as requested in the instruction above), including the objects that may commonly appear under each scenario (e.g., left: towel by the sink, mirror over the sink, vase on the counter, …). This further demonstrates LayoutGPT’s powerful reasoning ability with commonsense knowledge.\n\n\n> * Question 3: Demonstrative examples for numerical reasoning with comparative terms \n\nThanks for pointing out. We’ll provide more demonstrative examples for numerical reasoning with comparison terms in the next revision due to space constraints. Yet please refer to Fig.4&5 in the additional rebuttal PDF for more than 5 objects per image.\n\n\n> * Question 4: Regarding floor plans with various shapes \n\nNon-rectangular floor plans from 3D-FRONT are represented as a long list of vertices or binary images. At this time, LLMs cannot understand the meaning behind the list or take images as inputs. Therefore, we leave it as future work to improve LLM’s compatibility or multimodal skills.\n\n\n> * Question 5: Image resolution for 3D scene FID \n\nThanks for pointing out the issue. We render 256x256 images for each scene from four fixed camera positions (0,0,2), (0,2,0), (2,0,0), and (2,2,2) (unit: meters). Cameras always point towards the origin (0,0,0). \n\n\n> * Question 6: Add reference \n\nWe will add these references in the next revision for completeness.\n"}, {"review_id": "VtC2ozuNhz", "reviewer": "Reviewer_HVHZ", "summary": "The paper propose LayoutGPT that can generate visual arrangements of objects using the input prompts, providing a way to collaborate with visual generative models for compositional layout based image generation in both 2D and 3D. Experiment results show that such a method can largely improve layout-based generation using reasoning ability of large language models.", "questions": "1. In table 4, when generating images, is the mainly prompt used as the input or are those colored sentences used as inputs?\n2. It seems to me that layoutGPT can do a lot of guessing instead of reasoning. What if you use counter-factual examples or simple examples that rarely appear in real life?", "limitations": "The authors don't include limitation section. One thing is that LLMs can be biased such that generated imagery can also biased.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and easy to follow.\n2. The authors have conducted a comprehensive evaluations/ablations on the method.\n3. The paper provides an interesting way to connect large language models with visual generative models for image generation without any additional training.", "weaknesses": "1. **The paper simply proposes a module using LLMs to conduct visual planning**, i.e., extract / reason object relations and the number of objects when given input text descriptions. For image generation, extracted bounding boxes are simply used as inputs to existing layout-based methods, thus no technical contributions to visual generative models.\n2. **The paper over-claimed more or less.** For example, in Line 196, the author claimed \"LayoutGPT can perform accurate attribute binding\". However, in text-based inpainting section of Figure 4, the purple suitcase doesn't have the specified design \"a blue, yellow and white flower\" and the cat isn't black and white. Similarly, the spatial relationships extracted from the method seem to be off. The first example of text-based inpainting, the cat should be sitting under a bench, thus it seems to me that the method isn't that reliable.\n3. **Lacking some baselines**. For 2D image generation, there are a few works that do layout-based image generation, specifically using bounding boxes. It would be good to compare with these methods. One example I can think of is [1], which also doesn't require additional training.\n4. **Related Works**. It would be quite relevant to include compositional image generation, where you generate images conditioned on multiple specifications or objects.\n\n[1] Chen et al., Training-free layout control with cross-attention guidance (CVPR 2023)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing suggestions for improvements and would like to clarify a few misunderstandings in the response.\n\n> * Weakness 1: Technical contributions of LayoutGPT \n\nWe respectfully disagree that our contribution is limited. We have never claimed that LayoutGPT contributes to visual generative models. Our main contribution is: 1) combining program synthesis with ICL for LLMs to achieve layout generation; 2) NSR-1K benchmark; 3) proving the effectiveness of facilitating visual planning&generation using LLMs. We are also one of the first to show LLMs’ potential in understanding 3D concepts and 3D planning. As far as we know, layout generation alone has been extensively studied in 2D [17, 21, 23, 24] or 3D [31,35,44]. Moreover, LayoutGPT shows strong performance in both domains. \n\n\n> * Weakness 2: The paper over-claimed more or less.\n\nWhile we respectfully disagree, we do notice the confusion on attribute binding. Please refer to the general response for clarification. \n\nThe inaccurate “purple suitcase” and “black and white cat” attributes to the bottleneck from the layout-to-image model. As is claimed above, the downstream layout-to-image model is NOT the main focus of our work. The examples of text-based inpainting aims to shows the flexibility and potential of LayoutGPT in “imagining” intricate object-level language descriptions for creative image generation.  Fig. 1 in additional rebuttal PDF material shows that ReCo [53] can accurately represent each object with output from LayoutGPT. \n\nAs for the mistake in spatial relation in Fig. 4 (bottom-left), these examples are generated by using just an instruction and no in-context examplars. We only append the following text to the instruction to enable text-based inpainting:\n\n*IMPORTANT: apart from generating name and bounding box location for each object, you should also write a detailed description of the object, and add the description to each CSS line. For example, dog {{width: 20px; height: 19px; left: 42px; top: 25px; description: a white dog with many black dots}}*\nHence, the exception in Fig.4 doesn’t represent the overall performance. Please refer to Table 2&3 for systematic evaluation. \n\n\n> * Weakness 3: Lacking baselines like “Training-free layout control with cross-attention guidance (CVPR 2023)”\n\nAs is claimed above, our main focus is layout generation, not layout-guided image generation. Yet, in fact, we have included the indicated work upon submission. In Table 3, line 9-10 shows the results of the listed work (which we refer to as “Layout-Guidance”) conditioning on the layouts predicted by LayoutGPT. The results verify that LayoutGPT is a model-agnostic approach, and can be applied to various layout-to-image generation models (line 209-214).\n\n\n> * Weakness 4: Add related work discussion on compositional image generation \n\nWe will add a separate subsection for compositional image generation for completeness. We appreciate any specific references you would like to provide.\n\n\n> * Question 1: Input for image generation in Figure 4 \n\nFor images in Figure 4, we input the main prompt, all the colored sentences, and corresponding bounding boxes to GLIGEN. GLIGEN will condition on the colored sentence when rendering objects for each bounding box. Similarly for ReCo in Fig.1 in the additional rebuttal PDF materials.\n\n\n> * Question 2: It seems to me that layoutGPT can do a lot of guessing instead of reasoning. Counterfactual / rarely seen prompts \n\nWe respectfully disagree that LayoutGPT do more guessing than reasoning. We conducted multiple experiments to show the reasoning behind. First, in our appendix, line 98-105 and Table 3 shows that LayoutGPT achieves strong performance with random exemplars. LayoutGPT elicits reasoning abilities instead of copying from exemplars. Second, Fig.2&4 shows that LLMs generates novel sizes/locations instead of copying from the exemplars. Lastly, Table 3 in main paper and Fig. 3 (top) in additional rebuttal PDF materials show that all components are essential for correct spatial relations. Please also kindly refer to the general response.\n\n\n>* Limitations: Limitation section\n\nWe would like to clarify an important misunderstanding regarding the limitation section. We have included a limitation section in the supplementary material (Section F, line 210-225) where we have discussed multiple points ranging from layout domains to knowledge distillation. We are happy to include the bias discussion in the next revision. \n"}, {"review_id": "KuuOfPFmgZ", "reviewer": "Reviewer_uKyZ", "summary": "The paper highlights the challenges faced by existing models in generating objects with specified counts, positions, attributes, and sizes, and emphasizes the need for compositional skills that can effectively arrange components coherently, accurately reflecting object specifications and interactions. \nThe authors propose LayoutGPT which aims to improve the utility of visual planning skills of Large Language Models (LLMs) by in-context learning. \nThe method shows promising results in generating plausible layouts in multiple domains including 2D images and 3D indoor scenes.\nExperiments show that LayoutGPT outperforms text-to-image models and achieves comparable performance to human users in designing visual layouts for numerical and spatial reasonings. \nIn addition, LayoutGPT also shows comparable performance to supervised methods in 3D indoor scene synthesis. \nThe paper also proposes a new benchmark called NSR-1K for evaluating generations in terms of specified counts and spatial locations. \n", "questions": "1. The authors mention that the in-context examples are provided in reverse order based on similarity. Does the order have an impact on the effectiveness of the model？\n2. How are the colored sentences in Figure 4 generated?\n3. For 3D layout planning, how do the furniture frequencies in the instruction impact the performance?\n4. Why does the CSS structure have more impact on task performance compared to instruction?\n\n", "limitations": "I do not see any limitations to this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors propose a novel solution for handling the issue that the current visual; generation models lack visual arrangments ability. They adopt LLMs as visual planners to generate layout information of objects in the target 2D images or 3D scenes, with the help of in-context visual demonstrations in style sheet language.\n2. LayoutGPT generates reasonable layouts in multiple domains, including 2D images and 3D indoor scenes.\n3. They build a challenging benchmark that characterizes counting and positional relations for text-to-image generations.\n4. Experiments show the effectiveness of the proposed method.\n5. The paper presents promising results and provides insights into the application of LLMs for visual planning and generation, both in terms of methodology and empirical findings.", "weaknesses": "\n1. The writing of motivation is not clear due to a lot of inexplicable expressions.\n\n    - The claim that existing visual generative models are not equipped with various reasoning skills that exist in LLMs is dubious, given the disparate nature of reasoning skills required for visual generative models compared to LLMs. Good reasoning skills are not equal to good generative effects. Therefore, the T2I models that fail to generate objects with specific counts, positions, and attributes do not indicate lacking reasoning skills. \n\n    - In the sentence, 'But unlike LLMs, ... discrete categories ...', do the authors mean LLMs are already capable of generating layout? if yes, there should have some citations, and there is no need to further introduce the LLMs as a centralized model because there is no logical correlation with the previous text. In addition, why did the authors strengthen the discrete categories, could the LLMs generate continuous categories?\n\n    - Unclear description of the drawbacks of the existing LLMs-centered systems. \n\n    - The motivations behind constructing layouts with structured programs are not sufficiently persuasive, considering that LLMs also are trained on the plain text, and alternative methods exist for representing image layouts. In fact, the strict imposition of a structured format may pose challenges for LLMs.\n\n    - To my knowledge, visual inputs generally refer to images or videos, and the LLMs are only able to handle textual inputs. Moreover, both tasks in this work involve textual information as input, and there is no experimental evidence supporting the claim that LLMs have the potential in handling complex visual inputs.\n\n\n    - The meaning of \"addressing the inherent multimodal reasoning skills of LLMs\" requires clarification. To my understanding, LLMs already possess visual planning capabilities, and the authors' objective is to effectively leverage these skills rather than fundamentally modify or address them.\n\n\n2. Inconsistent notations.\n \n    - $o_j$ vs $\\mathbf{o}_j$ in the section 3.1.\n\n    - $o_j$ represents the layout of an object, so the sequence should be (c_j, x_j, y_j, w_j, h_j).\n\n\n\n3. The utilization of CSS as a structured format for representing layouts is a bit farfetched and unnecessary. Firstly, LLMs are capable of understanding the meaning of each element in a plain sequence (o) if detailed task instruction is provided. Secondly, structured representation is not solely limited to the CSS format; alternative formats such as, 'layout: c_1(x_1, y_1, w_1, h_1), c_2(x_2, y_2, w_2, h_2), ...'.\n\n\n4. The reasons behind the lower performance observed when GLIGEN generates images based on ground truth (GT) layouts during image evaluation, as well as when GLIGEN synthesizes images using layouts provided by humans, remain unexplained in Table 2.\n\n\n5. It is important to note that LayoutGPT is specifically designed for generating layouts, encompassing object categories and corresponding bounding boxes. Therefore, it is unreasonable to conclude that LayoutGPT can accurately perform attribute binding. Correct attribute binding is only achieved when the attributes of objects in the generated images align with the textual descriptions. However, wrong attribute bindings between the generated images and sentences are apparent, such as the closed car window in the top row of Figure 4 and the absence of a handle on the basket in the bottom row of Figure 4.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for pointing out ambiguities in the manuscript, and we’d like to clarify a few misunderstandings below.\n\n\n>* Weakness 1-1: Reasoning ability of current T2I models \n\nWe did not claim that existing visual generative models are not equipped with various reasoning skills. It is a fact that “text-to-image generation (T2I) models suffer from generating objects with specified counts, positions, and attributes” (line 22), which has been discovered in previous work like Stable Diffusion/DALLE/Imagen and etc.\n\n\n\n>* Weakness 1-2: Meaning of discrete categories\n\nWe are one of the first work to use LLMs to generate layouts for both 2D and 3D scenes. The sentence refers to previous layout generation models (not LLMs) that are transformer-based and trained from scratch. These models previous a “class” for each bounding box where the class label is represented as a fixed-length one-hot encoding vector instead of free-form text. Thus, these vectors represents “discrete categories”. LLMs is much more flexible that can generate both class labels or detailed descriptions in free-form text.\n\n\n\n\n>* Weakness 1-3: Drawback of existing LLM-centered systems \n\nLine 37-38 refers to previous work like Visual ChatGPT [48] or MM-React [52] that runs image generation models as APIs and directly feed user prompt to the APIs. We will revise the sentence for clarification.\n\n\n>* Weakness 1-4: Motivation of using structured output format during LLM layout planning \n\nWe’d like to point out that the all GPT models used in this study (Codex, GPT3.5, GPT4) are trained on both plain text and code snippets, which benefits them to adhere to structured output smoothly. Please find more discussion below in response to your “Weakness 3”.\n\n\n>* Weakness 1-5: Regarding LLM and visual input \n\nWe have never claimed that our work enables LLMs to take visual inputs. Our experimental results support the potential because LayoutGPT can understand the spatial concepts (left, right, …) behind coordinate values in 2D or 3D spaces. We also show that LayoutGPT has the potential to handle complicated skeleton sequences with a few examples (Fig. 6 & Table 5 in appendix).  \n\n\n>* Weakness 1-6: The meaning of \"addressing the inherent multimodal reasoning skills of LLMs\" \n\nWe focus on designing a method to elicit the visual planning skills from LLMs. Table 3 proves that LLMs cannot effectively use the inherent reasoning skills without structured representations. Fig. 3 in the additional rebuttal PDF materials also substantiate the claim.\n\n\n>* Weakness 2: Regarding notations \n\nThanks for pointing out, we will unify $o_j$ and $\\textbf{o}_j$ in the revision. \n\n$\\textbf{o}_j$ refers to the object layout, and our definition of $\\textbf{o}_j$ (line 106) includes bbox location ($x_j, y_j$) and bbox size ($w_j, h_j$) through $\\textbf{t}_j$ and $\\textbf{s}_j$. For 3D objects, $\\textbf{o}_j$ also includes orientation $\\textbf{r}_j$ (line 108).\n\n\n>* Weakness 3: LayoutGPT with other structured format \n\nIn Section 4.4 (page 7, line 201-208), we conducted an ablation study to check the effect of CSS structure and compare it with plain text structure. We compare prompts w/ CSS stuctures (e.g., “teddy bear {width: 32px; height: 45px; left: 31px; top: 9px; }”) and w/o CSS structures (e.g., “teddy bear: 32, 45, 31, 9”). (See Table 2 in the appendix for detailed examples). Results in Table 3 show that wrapping layout w/ CSS structures surpass plain text structure, which verifies the effectiveness of our method. \n\nLayoutGPT’s superior performance w/ CSS structure may due to the fact that OpenAI GPT models have read tons of HTML/CSS code during its pretraining process, and thus have acquired numerical & spatial concepts for webpage planning with similar format.\n\n\n\n>* Weakness 4: GLIGEN’s performance on ground-truth layouts \n\nThere are still bottlenecks in existing layout-to-image models. The reason for GLIGEN’s image evaluation scores are two-fold:\n\n(1) GLIGEN has its shortcomings in conducting layout-to-image generation. Even though it is conditioning on groundtruth layout, it may still fail to render images with perfect quality.\n\n(2) The object detection model GLIP also has its shortcomings in detecting objects. This might explain why the generated images have a much lower image-level object accuracy than the layout-level object accuracy.\n\nWe mentioned the above reasons in line 186-188, and will include more discussions in revision.\n\n\n>* Weakness 5: Regarding attribute binding \n\nPlease see general response.\n\n\n>* Question 1: Providing in-context exemplars in reverse order \n\nWe follow previous work [1, 51] for reverse order. The impact of exemplars order is not the main focus of the work.\n\n\n>* Question 2: How are the colored sentences in Figure 4 generated? \n\nWe append the following additional paragraph to the instruction with one simple example and 0 ICL exemplars:\n\n*IMPORTANT: apart from generating name and bounding box location for each object, you should also write a detailed description of the object, and add the description to each CSS line. For example, dog {width: 20px; height: 19px; left: 42px; top: 25px; description: a white dog with many black dots}*\n\n>* Question 3: Influence of furniture frequencies in 3D layout planning \n\nWe observe that instruction without the furniture frequency has similar KL Div., FID and OOB rates. We append the information for completeness yet finding a proper method to input the distribution remains a future study.\n\n\n>* Question 4: Impact of CSS structure vs. Impact of instruction \n\nThe CSS structure explicitly clarifies the spatial meaning of each number with the dense format of “Property name: Property value;”. While instruction also explains the meaning of each value in (x,y,w,h), we hypothesize that it is a weaker implication for LayoutGPT to understand the values. \n\n>* Limitations: Limitation section\n\nWe have included a section on LayoutGPT’s limitations in the supplementary material (Section F, line 210-225)."}]

Author rebuttal:
# General Response

We thank all reviewers for their constructive feedback and comments. We would like to address reviewers’ common concerns in the following general response:


> * Quantitative and qualitative results regarding attribute binding **(Reviewer ykMz & HVHZ & uKyZ)**

We would like to clarify that “accurate attribute binding” in lines 196 & Fig. 4 refers to binding attributes to generated bounding boxes instead of objects in the generated images. We will revise the writing to avoid future confusion. 

Here, we’d like to show more quantitative and qualitative results on attribute binding. LayoutGPT binds attributes to each object’s bounding box with 100\% accuracy on HRS [1] color prompts (e.g. “a green car and a blue chair”). On top of that, we evaluate the attribute correctness rate (accuracy) on the final generated images when LayoutGPT is combined with downstream image generation models. The below indicates that the major bottleneck lies in the layout-guided generation part of the system. Fig.1 in the additional rebuttal PDF material shows that LayoutGPT+ReCo ends up with more faithful object attributes. 


|                  | Attribute binding accuracy using HRS eval metric on generated images |                      |                     |                      |
|------------------|----------------------------------------------------------------------|----------------------|---------------------|----------------------|
|                  | Overall                                                              | Prompts w/ 2 objects | Prompt w/ 3 objects | Prompts w/ 4 objects |
| SD1.4            | 12.84                                                                | 18.57                | 10.10               | 11.36                |
| A&E              | 22.96                                                                | 31.43                | 19.19               | 20.45                |
| LayoutGPT+GLIGEN | 18.68                                                                | 22.86                | 19.19               | 14.77                |
| LayoutGPT+ReCo   | **36.96**                                                                | **40.00**                | **37.37**               | **34.09**                |



> * LayoutGPT’s performance on rare scenarios / counterfactual prompts **(Reviewer ykMZ & HVHZ)**

We provide more discussions on LayoutGPT’s performance on rare scenarios or counterfactual prompts.

* Quantitative Results

We first evaluate LayoutGPT’s reasoning ability regarding object size. We use the standard HRS benchmark [1] which is designed for benchmarking compositional text-to-image models.
HRS prompts for size reasoning contain comparison terms between randomly sampled common objects. The size relations described in HRS size prompts are often counterfactual and rarely seen (e.g., “a person which is smaller than a chair and larger than horse”, “a car which is smaller than a banana and chair and bigger than airplane”). LayoutGPT achieves an accuracy of 98.0% / 93.1% / 92.1% when the prompt involves size comparison between 2/3/4 objects. Meanwhile, the best size reasoning performance of 9 text-to-image models reported by the HRS benchmark has only 31.1% / 0.2% / 0%. The results verify that LayoutGPT acquires decent reasoning ability on rare scenarios / counterfactual prompts.


* Qualitative Results

In addition, we ask GPT-4 to write a few counterfactual prompts with the following instructions: 

*“Please provide a few counterfactual prompts that depict rarely seen the spatial relationship between the 80 MSCOCO object categories. An example would be "a monkey riding on top of a bird"”.*

We test LayoutGPT on these counterfactual prompts with 8-shot in-context learning. The supportive examples for in-context learning are from the MSCOCO2017 train set that depicts everyday scenarios, which is very different from the ChatGPT-generated counterfactual prompts used for testing. 

We show the illustrative demo of LayoutGPT’s prediction in Fig.5 in the additional rebuttal PDF material. LayoutGPT demonstrates competent layout planning ability on these challenging counterfactual prompts and handles the relationship between objects well.


[1] HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models (ICCV’23).


Meta-review: This submission proposes to use large language models (LLMs) with in-context examples to generate layouts with text conditions.  Layouts are represented as css and the method is evaluated on its ability to generate layouts for 2D images (with counting and spatial relations) and 3D indoor scenes (given room type and size).

Overall, reviewers are positive as this is the first work to show that LLMs can be leveraged to generate these layouts with in-context learning without explicit training.  As the work does have limitations, reviewers had additional questions and concerns for the authors which were addressed during the author response period.

The authors are encouraged to incorporate clarifications and reviewer suggestions in their camera ready including:
1. Clarifying details including number of the in-context examples, notation, and other reviewer questions
2. Discussion and experiments on attribute binding.
3. Generation of 3D scenes based on text descriptions that go beyond just the room type and size.
4. Additional examples of what the model can handle well and what it cannot (including rare scenarios and analysis of 2D reasoning ability of the proposed model)
5. Add discussion of compositional image generation to related work
6. Move discussion of limitations to main paper
Decision: Accept (poster)