Title: LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large Language Models

Compositional Image Generation

3D scene synthesis
Abstract: Attaining a high degree of user controllability in visual generation often requires intricate, fine-grained inputs like layouts. However, such inputs impose a substantial burden on users when compared to simple text inputs. To address the issue, we study how Large Language Models (LLMs) can serve as visual planners by generating layouts from text conditions, and thus collaborate with visual generative models. We propose LayoutGPT, a method to compose in-context visual demonstrations in style sheet language to enhance visual planning skills of LLMs. We show that LayoutGPT can generate plausible layouts in multiple domains, ranging from 2D images to 3D indoor scenes. LayoutGPT also shows superior performance in converting challenging language concepts like numerical and spatial relations to layout arrangements for faithful text-to-image generation. When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40\% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness. Lastly, LayoutGPT achieves comparable performance to supervised methods in 3D indoor scene synthesis, demonstrating its effectiveness and potential in multiple visual domains.

LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

Abstract

Attaining a high degree of user controllability in visual generation often requires intricate, fine-grained inputs like layouts. However, such inputs impose a substantial burden on users when compared to simple text inputs. To address the issue, we study how Large Language Models (LLMs) can serve as visual planners by generating layouts from text conditions, and thus collaborate with visual generative models. We propose LayoutGPT, a method to compose in-context visual demonstrations in style sheet language to enhance the visual planning skills of LLMs. LayoutGPT can generate plausible layouts in multiple domains, ranging from 2D images to 3D indoor scenes. LayoutGPT also shows superior performance in converting challenging language concepts like numerical and spatial relations to layout arrangements for faithful text-to-image generation. When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness. Lastly, LayoutGPT achieves comparable performance to supervised methods in 3D indoor scene synthesis, demonstrating its effectiveness and potential in multiple visual domains.

1 Introduction

Can Large Language Models (LLMs) comprehend visual concepts and generate plausible arrangments in visual spaces? Recently, LLMs have shown significant advancement in various reasoning skills [50, 49] that remain challenging to visual generative models. For instance, text-to-image generation (T2I) models suffer from generating objects with specified counts, positions, and attributes [10, 24]. 3D scene synthesis models face challenges in preserving furniture within pre-defined room sizes [30]. Addressing these issues necessitates the development of compositional skills that effectively arrange components in a coherent manner, accurately reflecting object specifications and interactions.

Visual layout is an essential symbolic representation that has been widely studied as it reflects the compositions of a visual space [33, 53, 45, 34]. For instance, layout generation models [21, 25, 17, 53, 23] can be combined with region-controlled image generation methods [56, 27] to improve image compositionality [52]. But unlike LLMs, these models are restricted to discrete categories or have limited reasoning skills for complicated text conditions. Recently, LLMs like ChatGPT [37], are adopted as a centralized module of frameworks or systems where multiple foundational computer vision models are integrated. Through defined action items or API calls, LLMs can interact with visual generative models to extend the systems' capability into image generation tasks. [51].

Despite the advancement, existing approaches that involve the collaboration between LLMs and image generation models are either limited to executing the latter through program generation or using LLMs for language data augmentation for image editing [3]. Current LLM-based systems fail to improve the compositional faithfulness of a generated image by simply using T2I models through API calls. While one could additionally integrate models that synthesize images with the guidance of layouts [56, 27], keypoints [27], or sketches [20, 57], users still have to create fine-grained inputs on their own, leading to extra efforts and degraded efficiency compared to pure language instructions.

To address these challenges, we introduce LayoutGPT , a training-free approach that injects visual commonsense into LLMs and enables them to generate desirable layouts based on text conditions. Despite being trained without any image data, LLMs can learn visual commonsense through in-context demonstrations and then apply the knowledge to infer visual planning for novel samples. Specifically, we observe that representing image layouts is highly compatible with how style sheet language formats images on a webpage. Therefore, as LLMs are trained with program data, constructing layouts as structured programs may enhance LLMs' ability to 'imagine' object locations from merely language tokens. Our programs not only enable stable and consistent output structures but also strengthen LLMs' understanding of the visual concepts behind each individual attribute value. When combined with a region-controlled image generation model [27], LayoutGPT outperforms existing methods by 20-40% and achieves comparable performance as human users in generating plausible image layouts and obtaining images with the correct object counts or spatial relations.

In addition, we extend LayoutGPT from 2D layout planning to 3D indoor scene synthesis. With a slight expansion of the style attributes, LayoutGPT can understand challenging 3D concepts such as depth, furniture sizes, and practical and coherent furniture arrangements for different types of rooms. We show that LayoutGPT performs comparably to a state-of-the-art (SOTA) supervised method. Our experimental results suggest that LLMs have the potential to handle more complicated visual inputs. Our contribution can be summarized as the following points:

We propose LayoutGPT, a program-guided method to adopt LLMs for layout-based visual planning in multiple domains. LayoutGPT addresses the inherent multimodal reasoning skills of LLMs and can improve end-user efficiency.

We propose N umerical and S patial R easoning (NSR-1K) benchmark that includes prompts characterizing counting and positional relations for text-to-image generation.

Experimental results show that LayoutGPT effectively improves counting and spatial relations faithfulness in 2D image generation and achieves strong performance in 3D indoor scene synthesis. Our experiments suggest that the reasoning power of LLMs can be leveraged for visual generation and handling more complicated visual representations.

2 Related Work

Image Layout Generation Layout generation has been an important task for automatic graphical design for various scenarios, including indoor scenes [40, 46], document layouts [59, 60, 15], and graphical user interface [8]. Previous work has proposed various types of models that need to be trained from scratch before generating layouts. LayoutGAN [25] is a GAN-based framework to generate both class and geometric labels of wireframe boxes for a fixed number of scene elements. LayoutVAE [21] generates image layouts conditioned on an input object label set. Transformerbased methods are proposed to enhance flexibility in the layout generation process. For instance, LayoutTransformer [17] adopts self-attention to learn contextual relations between elements and achieve layout completion based on a partial layout input. BLT [23] proposes a hierarchical sampling policy so that any coordinate values can be modified at the sampling stage to enable flexible and controlled generation. However, existing methods are restricted to class labels and fail to reason over numerical and spatial concepts in text conditions. In contrast, LayoutGPT can convert challenging textual concepts to 2D layouts and generate free-form, detailed descriptions for each region.

Compositional Image Generation Recent studies have shown that text-to-image generation (T2I) models suffer from compositional issues such as missing objects, incorrect spatial relations, and incorrect attributes [24, 2]. StructureDiffusion [10] proposes to adjust text embeddings by utilizing prior knowledge from linguistic structures. Attend-and-Excite [4] optimizes attention regions so that objects attend on separate regions. Another line of work strives to introduce extra conditions as inputs. For example, ReCo [56], GLIGEN [27], and Layout-Guidance [6] can generate images based on bounding box inputs and regional captions. [52] combines a layout generator and a region-controlled method to achieve accurate generation results. While we focus on layout generation, we also employ layout-to-image models to generate final images and show the effectiveness of LayoutGPT.

Indoor Scene Synthesis Indoor scene synthesis aims at generating reasonable furniture layouts in a 3D space that satisfies room functionality. Early work adopting autoregressive models requires supervision of 2D bounding boxes and other visual maps [40]. Later, SceneFormer [47] proposes to apply a set of transformers to add furniture to scenes. While previous work adopts separate models to predict different object attributes, ATISS [38] demonstrates that a single transformer model can generate more realistic arrangments while being more efficient. In this work, we investigate leveraging LLMs to achieve scene synthesis without any fine-tuning.

LLMs for Vision Language inputs have been an essential part of many vision language tasks [43, 11, 28, 14]. With the strong generalization ability of contemporary LLMs, recent work attempts to adapt the power of LLMs on multimodal tasks [31, 55]. For instance, multimodal chain-of-thought [58] trained a model to incorporate visual inputs as rationales for question answering. [22] proposes to learn translation parameters to map embeddings between visual and language domains such that an LLM can ground on both modalities. VisProg [18] and ViperGPT [44] use LLMs to design modular pseudocode instructions or executable Python programs to achieve visual reasoning. LLMScore [32] leverages LLMs to evaluate text-to-image models. Visual ChatGPT [51] proposes a prompt manager that supports the execution of various image generation models. In this work, we directly involve LLMs in the generation process by leveraging LLMs to design visual layouts through in-context learning and structured representations.

3 Method

3.1 Overview

Given a condition C , the goal of layout generation is to predict a set of tuples O ' t o j | j ' 1 , 2 , . . . , n u where each tuple o j denotes the layout information of a 2D or 3D bounding box of object j . In image planning, C is the input text prompt, o j consists of a category c j , bounding box location t j ' p x j , y j q P R 2 and bounding box size s j ' p w j , h j q P R 2 , i.e. o j ' p c j , t j , s j q . Similarly, in 3D scene synthesis, C specifies the room type and room size, o j consists of category c j , location t j P R 3 , size s j P R 3 , and orientation r j P R , i.e. o j ' p c j , t j , s j , r j q . While c j can be modeled as a discrete value, our method directly predicts the category text.

a clock on the wall saying it is

2:41 in the afternoon

Task Instruction:

/* Task Description;

Format Definition;

Available object categories

(for scene synthesis);

… */

/* first in-context exemplar*/

Prompt: a clock on the wall saying it is 2:41

in the afternoon.

Layout:

clock {

height: 81px;

width: 93px;

top: 119px;

left: 15px;

}

/* second in-context exemplar*/

…

/* inference condition*/

Prompt: a tower of a building that

has a clock on it.

Layout:

Visualization

LLM

Layout-to-Image

Model

Input Condition

𝓒

𝒋

Image Layout Generation

: 'A tower

of a building that has a clock on it'

: 'Room Type:

Bedroom, Room size: 3.5m x 3.8m'

Retrieve

in-context

exemplars

using

𝑓(# | #)

Parse into

layouts

Input prompt

Output layouts in

CSS format

Indoor Scene Synthesis

Text-conditioned

Image Generation

CSS Structure

Normalize to 'px'

Condition:

Room Type: Bedroom

Room Size: length 256px, width 278px

Layout:

double_bed {

length: 213px;

width: 250px;

height: 113px;

left: 125px;

top: 108px;

depth: 56px;

orientation: 90 degrees;

}

Newly

defined

properties

/* LLM Output*/

tower {

height: 149px;

width: 56px;

top: 65px;

left: 100px;

}…

L

3.2 LayoutGPT Prompt Construction

As is shown in Fig. 2, LayoutGPT prompts consist of three main components: task instructions , and in-context exemplars in CSS structures with normalization .

CSS Structures In autoregressive layout generation, o j is usually modeled as a plain sequence of values, i.e. ( c 1 , x 1 , y 1 , w 1 , h 1 , c 2 , x 2 , . . . ) [17, 23]. However, such a sequence can be challenging for LLMs to understand due to underspecified meaning of each value. Therefore, we seek a structured format that specifies the physical meaning of each value for LLMs to interpret spatial knowledge. We realize that image layouts are highly similar to how CSS (short for Cascading Style Sheets) formats the layout of a webpage and defines various properties of the img tag in HTML. For instance, x j , y j corresponds to the standard properties left and top , while w j , h j corresponds to width and height in CSS. As LLMs like GPT-3.5/4 are trained with code snippets, formatting image/scene layouts in CSS structures potentially enhances the LLMs' interpretation of the spatial meaning behind each value. Therefore, as is shown in Fig. 2, we place category name c j as the selector and map other attribute values into the declaration section following standard CSS styles.

Task Instructions & Normalization Similar to previous work in improving the prompting ability of LLMs [48, 42, 37], we prepend task instructions to the prompt to specify the task goal, define the standard format, unit for values, etc. Besides, as the common length unit of CSS is pixels (px), we normalize each property value based on a fixed scalar and rescale the value to a maximum of 256px. As will be shown in later sections (Sec. 4.4 & 5.4), all three components play important roles in injecting visual commonsense into LLMs and improving generation accuracy.

3.3 In-Context Exemplars Selection

Following previous work [1, 54], we select supporting demonstration exemplars for in-context learning based on retrieval results. Given a test condition C j and a support set of demonstrations D ' tp C k , o k q| k ' 1 , 2 , . . . u , we define a function f p C k , C j q P R that measures the distances between two conditions. For 2D text-conditioned image layout generation, we adopt the CLIP [39] model to extract text features of C j (usually a caption) and the image feature of C k and measure the cosine similarity between them. For the 3D scene synthesis task where each room has length rl and width rw , we measure distance with f p C k , C j q ' } rl k ´ rl j } 2 `} rw k ´ rw j } 2 . We select supporting demonstrations with the topk least distance measures and construct them as exemplars following the CSS structure in Fig. 2. These supporting examples are provided to GPT-3.5/4 in reverse order, with the most similar example presented last.

3.4 Image and Scene Generation

For text-conditioned image synthesis, we utilize a layout-to-image generation model to generate images based on the generated layouts. As for each object layout in 3D scene synthesis, we retrieve a 3D object based on the predicted category, location, orientation, and size following [38]. We directly render the scene with the retrieved 3D objects. See Sec. 4 & Sec. 5 for more details.

4 LayoutGPT for Text-Conditioned Image Synthesis

In this section, we provide an extensive evaluation of LayoutGPT for 2D text-to-image (T2I) synthesis and compare it with SOTA T2I models/systems. An ablation study is conducted to demonstrate the effect of individual components from LayoutGPT. We also showcase qualitative results and application scenarios of our method.

4.1 Experiment Setup

Datasets & Benchmarks To evaluate the generations in terms of specified counts and spatial locations, we propose NSR-1K, a benchmark that includes template-based and human-written (natural) prompts from MSCOCO [29]. Table 1 summarizes our dataset statistics with examples. For template-based prompts, we apply a set of filters to obtain images with only 1-2 types of object and then create prompts based on object categories and bounding box information. As for natural prompts, we extract COCO captions with keywords to suit the task of numerical reasoning (e.g. 'four') or spatial reasoning (e.g. 'on top of') and ensure that all objects from the bounding box annotations are mentioned in the caption to avoid hallucination. Each prompt from NSR-1K is guaranteed to have a corresponding ground truth image and layout annotations. Detailed benchmark construction processes are described in Appendix B.1.

Evaluation Metrics To evaluate generated layouts, we report precision, recall, and accuracy based on generated bounding box counts and spatial positions [9, 16]. For spatial reasoning, each prompt falls into one of the four types of relations ({ left, right, top, below }) and we use the bounding box center for evaluation following PaintSkills [7]. To evaluate generated images, we first obtain bounding boxes from GLIP [26] detection results and then compute average accuracy based on the bounding box counts or spatial relations. We also report CLIP cosine similarity between text prompts and generated images for reference. Detailed metric descriptions are listed in Appendix B.2.

Baselines As we consider both layout evaluation and image evaluation, we compare LayoutGPT with end-to-end T2I models (Stable Diffusion [41], Attend-and-Excite [4]) 2 and two-stage systems that generate layouts first and then apply GLIGEN [27] as the layout-to-image model. We also evaluate ground truth layouts and human-drawn layouts as the theoretical upper bounds. The human-drawn layouts are collected through crowdsourcing, in which we specifically ask human annotators to draw layouts given text prompts. We slightly modify LayoutTransformer [17] as a baseline for supervised conditional layout generation. Detailed descriptions of baseline setups and human annotating are discussed in the Appendix A and E.

Numerical Reasoning

Spatial Reasoning

Layout Eval.

Image Eval.

Layout Eval.

Image Eval.

Methods

Precision

Recall

Acc. (GLIP)

Acc. (GLIP)

Text

ÝÑ

Image

1

Stable Diffusion (v1.4) [41]

-

-

-

32.22

0.256

-

16.89

0.252

2

Stable Diffusion (v2.1)

-

-

-

42.44

0.256

-

17.81

0.256

3

Attend-and-Excite (SD v1.4) [4]

-

-

-

38.96

0.258

-

24.38

0.263

4

Attend-and-Excite (SD v2.1)

-

-

-

45.74

0.254

-

26.86

0.264

Text

Ñ

Layout

Ñ

Image

5

LayoutTransformer [17]

75.70

61.69

22.26

40.55

0.247

6.36

28.13

0.241

6

LayoutGPT (GPT-3.5)

94.81

96.49

86.33

51.20

0.258

82.54

52.86

0.264

7

LayoutGPT (Codex)

90.19

88.29

72.02

46.64

0.254

74.63

45.58

0.262

8

LayoutGPT (GPT-3.5, chat)

81.84

85.47

75.51

54.40

0.261

85.87

56.75

0.268

9

LayoutGPT (GPT-4)

78.36

86.29

78.43

55.64

0.261

91.73

60.64

0.268

10

GT layouts

100.00

100.00

100.00

53.23

0.256

100.00

62.54

0.261

11

Human

99.26

96.52

92.56

56.07

0.258

91.17

51.94

0.258

Three clocks at a train station under a concrete arch (Numerical)

Stable Diffusion

LayoutGPT

LayoutGPT+GLIGEN Human+GLIGEN

Stable Diffusion

LayoutGPT

LayoutGPT+GLIGEN

Human+GLIGEN

A yellow surfboard sits next to a bicycle on a brick sidewalk (Spatial)

A cat is sitting on a basket under a bench (Spatial)

Two teddy bears and a stuffed snowman wearing hats (Numerical)

hat

hat

concrete arch

teddybear

teddybear

clock

clock

clock

stuffed

snowman

surfboard

bicycle

bench

cat

bosket

brick sidewalk

4.2 Evaluation Results

Quantitative Results As shown in Table 2, among the variants of LayoutGPT ( #6-#9 ), GPT-3.5 achieves the best performance in numerical reasoning while GPT-4 performs the best in generating correct spatial positions. LayoutGPT outperforms LayoutTransformer ( #5 ) by large margins, proving the strong cross-modal reasoning skills of LLMs. As for image-level evaluation, LayoutGPT surpasses end-to-end T2I models ( #1-#3 ) by 20-40% in GLIP-based accuracy and relatively 1-6% in CLIP similarity. Therefore, using layouts as an intermediate representation indeed leads to more reliable and faithful generation outcomes. In addition, LayoutGPT achieves similar layout accuracy as human users (numerical #6 vs. #11 (86.33% v.s. 92.56%); spatial #9 vs. #11 (91.73% v.s. 91.17%)), which implies its potential to spare users from drawing layouts manually. The discrepancy between layout accuracy and GLIP-based accuracy suggests that the bottleneck mainly stems from layout-guided image generation and GLIP grounding results.

In addition, LayoutGPT binds attributes to each object's bounding box with 100% accuracy on HRS [2] color prompts. We further evaluate the attribute correctness rate (accuracy) on the final generated images when combining LayoutGPT with GLIGEN/ReCo. As shown in Table 3, our system largely improves the color correctness over Stable Diffusion with multiple objects.

Qualitative results Weshow the qualitative results of LayoutGPT and baselines in Fig. 3. LayoutGPT can understand visual commonsense such as the clock sizes at a train station (top left) or complex spatial relations between multiple objects (bottom right), while SD fails to generate correct numbers or positions. Besides, LayoutGPT demonstrates a similar layout design to human users (bottom left). Fig. 11 in the Appendix visualizes the results of attribute binding using LayoutGPT and ReCo [56].

A bathroom featuring a sink and toilet

Di

ff

erent kinds of doughnuts in a

display case

A big brown cow walking down a

sidewalk near homes

A bathroom featuring a sink and toilet

A cat is lying on top of a suitcase.

a black and white cat lying on its side with

its eyes closed

a purple suitcase with a design of a blue,

A big brown cow walking down a

sidewalk near home

Grey and black cat sitting under a bench.

a grey and black cat with its tail tucked in

and its head down, looking at the viewer

a wooden bench with two legs

A cat is sitting on a basket under a bench.

a black and white cat with its tail curled up in

the basket

a light brown basket with a handle on the side

a wooden bench with a curved backrest

Different kinds of doughnuts in a

display case

Dense layout

Planning

Text-based

Inpainting

LayoutGPT

+GLIGEN

LayoutGPT

+GLIGEN

LayoutGPT

+GLIGEN

wall:

mirror

cabinet

pap

food

bupna

nouse

cpunter

sink

bttle

dorut

Gonu

ldohut

donut

donut

dorut

Goodut

dohut

dor

tree

coW

woll

doo

towe

pavement

floor

rug

yellow, and white flower

4.3 Application Scenarios

By utilizing LLMs as layout generators, LayoutGPT can be applied to a diverse set of scenarios for accurate and creative image generation.

Dense Layout Planning : In Fig. 4 (top), we apply random in-context examples from COCO17 panoptic annotations with 6 ' 15 bounding boxes per image. LayoutGPT can be applied to scenarios that imply numerous objects (e.g. different kinds of donuts) or various categories (e.g. bathroom or street view). Though only a few objects are mentioned in the prompts, LayoutGPT predicts layouts for the whole scene and imagines common objects that are usually visible in each scene.

Text-based Inpainting : In addition, the inherent language generation ability of LLMs enables our method to generate fine-grained regional descriptions from coarse global prompts (Fig. 4 bottom). LayoutGPT can enrich the description of each object with details that are not mentioned in the prompt, producing suitable outputs for models like ReCo [56].

Counterfactual Scenarios : We test LayoutGPT on counterfactual prompts provided by GPT-4 [35]. The in-context examples are randomly drawn from MSCOCO 2017[29], which greatly differs from the counterfactual prompts. As shown in Fig. 5, LayoutGPT manages to generate reasonable layouts on these challenging prompts and handles the relationship between objects well.

4.4 Ablation Study

Component Analysis Table 4 presents the component analysis of our CSS-style prompt on spatial reasoning prompts. Comparisons between line 1-3 entails that the task instructions ( #2 ) and CSS format ( #3 ) effectively improve layout accuracy. Format in-context exemplars in CSS structures

a rubber duck floating on the surface of a desert oasis, surrounded by cacti

a row of crabs collaborating to carry a

hammock made of interwined licorice strands

a rooster perched atop a giant typewriter,

proudly announcing the sunrise in a new font

sky

sky

forest

tree

tree

sond

typewriter

hommock

petoptntoce

coctis.

woter

sn150

rubber-duck

ground

sand

crab

crab

crab

show a more significant effect on accuracy. Pairwise comparisons of line 5-7 support the argument that the CSS style is the most essential component. While solely applying normalization degrades accuracy in line 4 , line 5&8 shows that it slightly improves the performance when combined with other components.

Model-Agnostic Property We show that LayoutGPT is agnostic to layout-guided image generation models in line 9-10 in Table 4. We feed the same generated layouts from LayoutGPT to LayoutGuidance [6] and compute image-level metrics. Compared to using ground truth layouts ( #10 ), LayoutGPT ( #9 ) shows a minor gap in GLIP-based accuracy and a comparable CLIP similarity score. The discrepancy in GLIP-based accuracy is similar to that in Table 2, implying that the layouts generated by our method are agnostic to the downstream model.

5 LayoutGPT for Indoor Scene Synthesis

5.1 Task Setup

Datasets & Benchmarks For indoor scene synthesis, we use an updated version of the 3D-FRONT dataset [12, 13] following ATISS [38]. After applying the same pre-processing operations, we end up with 4273 bedroom scenes and 841 scenes for the living room. We only use rectangular floor plans of the test set for evaluation since LayoutGPT is not compatible with irregular ones. Hence, we end up with 3397/453/423 for train/val/test split of bedroom scenes and 690/98/53 for train/val/test split of living room scenes.

Evaluation Metrics We follow prior work [38] to report KL divergence between the furniture category distributions of predicted and ground truth scenes. We also render scene images from four camera angles for each scene and report FID scores [19]. In addition, we report out-of-bound rates, i.e. the percentage of scenes with furniture exceeding the floor plan boundary.

5.2 Evaluation Results

Quantitative Results The evaluation results are recorded in Table 5. We provide a random baseline for comparison denoted as 'Random Scenes', in which the scene is randomly sampled from the in-context exemplars for each inference run. 3

ATISS

LayoutGPT

Living room

Living & Dining room

Overlapped

furniture

Overlapped

furniture

Out of bound

Out of bound

For both bedrooms and living rooms planning, LayoutGPT attains lower out-of-bound rates than ATISS (bedrooms: 43.26% vs. 49.88%; living rooms: 64.16% vs. 83.02%), which verifies LayoutGPT's spatial reasoning ability in 3D environments. In addition, LayoutGPT has lower FID compared to ATISS (bedrooms: 28.37 vs. 30.02; living rooms: 76.34 vs. 85.40), which indicates that the planned scene has higher quality. Noted here that the living room split contains much more objects on average (11 for living rooms vs. 5 in bedrooms) and is a low-resource split with only 690 training scenes. Therefore, while living rooms are challenging for both methods, LayoutGPT shows more significant improvement over ATISS as supervised methods tend to overfit in early epochs.

Meanwhile, ATISS performs better in terms of KL divergence, which means that the overall furniture distribution predicted by ATISS is closer to the test split. We observe that LayoutGPT tends to avoid furnitures that are extremely rarely seen in each scene (e.g. coffee tables for bedrooms) as these objects appear less frequently in the in-context demonstrations. The limited in-context demonstration size also restricts LayoutGPT to have a universal observation of the furniture distributions.

Qualitative Results As shown in Fig. 6, LayoutGPT manages to understand common 3D concepts, such as 'the pendant lamp should be suspended from the ceiling' and 'nightstands should be placed by the headboard of the bed' (bottom row). When given a floor plan size for both living and dining rooms, LayoutGPT can also generate complicated 3D planning with dining tables and chairs on one side, and a sofa, a coffee table, and a TV stand on the other side (bottom right).

5.3 Application Scenarios

Text-guided Synthesis : LayoutGPT can follow text captions to arrange furniture in the scene (see Fig. 7). When the captions enumerate a complete list of furniture, LayoutGPT strictly follows the captions to generate the furniture and achieve a KL Div. value close to zero.

Partial Scene Completion : Thanks to the autoregressive decoding mechanism, LayoutGPT can complete a scene with partial arrangments such that the additional furniture remains coherent with the existing ones. Through in-context demonstrations, LayoutGPT learns critical (visual) commonsense such as visual symmetric (e.g. nightstands in Fig. 8 (a)), positional relations (e.g. stool at the end of the bed in Fig. 8 (b)), and room functions (e.g. desks and chairs in the dining area in Fig. 8 (d)).

Completed

'A bedroom with a double bed, a

wardrobe, two tables, a cabinet and a

pendant lamp.'

Room Size: max length 214px, max

width 306px

'A bedroom with a double bed, two

wardrobes and a pendant lamp.'

Room Size: max length 262px, max

width 207px

'A living room with three coffee

tables, a tv stand, a multi seat sofa, a

dining table, four dining chairs, a

console table and two pendant lamps.'

Room Size: max length 256px, max

width 490px

'A living room with a multi seat sofa,

a coffee table, a tv stand, a dining

table, four dining chairs, a ceiling

lamp and a pendant lamp.'

Room Size: max length 413px, max

width 256px

Partial

Completed

Partial

Completed

Partial

Partial

Completed

Partial

Completed

Partial

Partial

Completed

Partial

Completed

Partial

Completed

Completed

6.8m

×

6.8m

×

6.8m

Partial

Completed

Partial

Completed

Partial

Completed

Bedroom, 3.8m

×

3.3m

×

3.0m

Living & Dining room, 3.6 m

×

6.8m

(a)

(b)

(c)

Bedroom, 3.5m

×

4.2m

Living & Dining room, 5.1m

×

3.3m

Living & Dining room, 5.4m

×

3.9m

(a)

(b)

(c)

(d)

(e)

(f)

Bedroom, 3.5m

×

4.2m

Living & Dining room, 5.1m

×

3.3m

Living & Dining room, 5.4m

×

3.9m

(a)

(b)

(c)

(d)

(e)

(f)

Bedroom, 3.5m

×

4.2m

Living & Dining room, 5.1m

×

3.3m

Living & Dining room, 5.4m

×

3.9m

(a)

(b)

(c)

(d)

(e)

(f)

Partial

Completed

Partial

Completed

Partial

Completed

Bedroom, 3.8m

×

3.3m

×

3.0m

Living & Dining room, 3.6 m

×

6.8m

(a)

(b)

(c)

Partial

Completed

Partial

Completed

Bedroom, 3.8m

×

3.3m

×

3.0m

(a)

(b)

Partial

Completed

Bedroom, 3.8m

×

3.3m

(a)

Partial

Completed

Partial

Completed

Partial

Completed

Bedroom, 3.8m

×

3.3m

×

3.0m

Living & Dining room, 3.6 m

×

6.8m

(a)

(b)

(c)

(a)

(b)

(c)

(d)

价

Bedroom, 3.5m

×

4.2m

Living & Dining room, 5.1m

×

3.3m

Living & Dining room, 5.4m

×

3.9m

Bedroom, 3.5m

×

4.2m

Living & Dining room, 5.1m

×

3.3m

Living & Dining room, 5.4m

×

Partial

Completed

Partial

Completed

Living & Dining room, 3.6 m

×

6.8m

×

3.0m

(c)

(b)

Living & Dining room, 5.1m

3.9m

Bedroom, 3.5m

×

4.2m

Living & Dining room, 5.1m

×

3.3m

Living & Dining room, 5.4m

Bedroom, 3.5m

×

4.2m

Bedroom, 3.5m

×

4.2m

Living & Dining room, 5.1m

×

3.3m

Living & Dining room, 5.4m

×

3.9m

(d)

(d)

(d)

5.4 Ablation Study

Similar to Sec. 4.4, we study the effect of task instructions, CSS structure, and normalization on indoor scene synthesis (see Table 6). In contrast to our conclusion for 2D planning in Sec. 4.4, comparisons between line 1-4 show that normalization ( #4 ) is the most critical component for suppressing the out-of-bound rate while the CSS structure is also effective. We observe that LLMs occasionally copy attribute values directly from in-context exemplars even though the room sizes are different. Therefore,

(e)

(f)

(e)

(f)

(d)

(e)

(d)

(e)

(f)

normalizing all exemplars to the same scale can reduce the out-of-bound rate. CSS style facilitates LLMs to understand the physical meaning behind each attribute value and hence leads to almost the best result when combined with normalization ( #7 ).

6 Conclusion

In this work, we address a new direction of generative model collaborations. Specifically, we are interested in how Large Language Models (LLMs) can collaborate with visual generative models. To this end, we propose LayoutGPT, an approach that turns an LLM into a visual planner through in-context learning and CSS style prompts. LayoutGPT can generate plausible visual arrangements in both image space and 3D indoor scenes. LayoutGPT can effectively improve image compositions by generating accurate layouts and achieves comparable performance in indoor scene synthesis compared to supervised methods. Besides, LayoutGPT can improve user efficiency in image generation and serve as an essential part of a unified system for all types of multimodal tasks.

×

3.3m

×

(f)

(e)

3.9m

Acknowledgments

The work was funded by an unrestricted gift from Google and a gift from the Robert N. Noyce Trust to the University of California via the Noyce Initiative. We would like to thank Google and the Robert N. Noyce Trust for their generous sponsorship. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the sponsors' official policy, expressed or inferred.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv , abs/2204.14198, 2022. 4

Eslam Mohamed Bakr, Pengzhan Sun, Xiaogian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-toImage Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 20041-20053, October 2023. 3, 6, 7, 19

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. CVPR2023 , 2022. 2

Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG) , 42(4):1-10, 2023. 3, 5, 6, 21

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. ArXiv , abs/2107.03374, 2021. 15

Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.03373 , 2023. 3, 8

Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3043-3054, 2023. 5, 16

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium on user interface software and technology , pages 845-854, 2017. 3

Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W.Taylor. Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction. In International Conference on Computer Vision (ICCV) , 2019. 5

Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. ICLR , 2023. 2, 3

Stella Frank, Emanuele Bugliarello, and Desmond Elliott. Vision-and-language or vision-forlanguage? on cross-modal influence in multimodal transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 9847-9857, 2021. 3

Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021. 8

Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d furniture shape with texture. International Journal of Computer Vision , 129:3313-3337, 2021. 8

Tsu-Jui Fu*, Linjie Li*, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 3

Tsu-Jui Fu, William Yang Wang, Daniel McDuff, and Yale Song. DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents. In Association for the Advancement of Artificial Intelligence (AAAI) , 2022. 3

Tsu-Jui Fu, Xin Eric Wang, Scott Grafton, Miguel Eckstein, and William Yang Wang. SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020. 5

Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry S Davis, Vijay Mahadevan, and Abhinav Shrivastava. Layouttransformer: Layout generation and completion with self-attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10041014, 2021. 2, 3, 4, 5, 6, 9

Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. CVPR 2023 , 2022. 3

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017. 8

Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. In International Conference on Machine Learning , 2023. 2

Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochastic scene layout generation from a label set. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9895-9904, 2019. 2, 3

Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. In International Conference on Machine Learning , 2023. 3

Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, and Irfan Essa. Blt: bidirectional layout transformer for controllable layout generation. In Computer VisionECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII , pages 474-490. Springer, 2022. 2, 3, 4

Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 , 2023. 2, 3

Jianan Li, Tingfa Xu, Jianming Zhang, Aaron Hertzmann, and Jimei Yang. LayoutGAN: Generating graphic layouts with wireframe discriminator. In International Conference on Learning Representations , 2019. 2, 3

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10965-10975, 2022. 5, 17

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 2, 3, 5, 6, 8, 21

Zhuowan Li, Cihang Xie, Benjamin Van Durme, and Alan Yuille. Localization vs. semantics: How can language benefit visual representation learning? arXiv preprint arXiv:2212.00281 , 2022. 3

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014. 5, 7, 21

Jingyu Liu, Wenhan Xiong, Ian Jones, Yixin Nie, Anchit Gupta, and Barlas O˘ guz. Clip-layout: Style-consistent indoor scene synthesis with semantic furniture embedding. arXiv preprint arXiv:2303.03565 , 2023. 2

Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842 , 2023. 3

Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. LLMScore: Unveiling the power of large language models in text-to-image synthesis evaluation. In Thirtyseventh Conference on Neural Information Processing Systems (NeurIPS) , 2023. 3

Andrew Luo, Zhoutong Zhang, Jiajun Wu, and Joshua B Tenenbaum. End-to-end optimization of scene layout. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3754-3763, 2020. 2

Wan-Duo Kurt Ma, JP Lewis, W Bastiaan Kleijn, and Thomas Leung. Directed diffusion: Direct control of object placement through attention guidance. arXiv preprint arXiv:2302.13153 , 2023. 2

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023. 7, 15

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv , abs/2203.02155, 2022. 15

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022. 2, 4

Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Information Processing Systems , 34:12013-12026, 2021. 3, 5, 8, 20

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , 2021. 4

Daniel Ritchie, Kai Wang, and Yu-an Lin. Fast and flexible indoor scene synthesis via deep convolutional generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6182-6190, 2019. 3

Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10674-10685, 2021. 5, 6, 21

Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations , 2022. 4

Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language , pages 70-80, 2015. 3

Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023. 3

Bo Wang, Tao Wu, Minfeng Zhu, and Peng Du. Interactive image synthesis with panoptic layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7783-7792, 2022. 2

Kai Wang, Yu-An Lin, Ben Weissmann, Manolis Savva, Angel X Chang, and Daniel Ritchie. Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Transactions on Graphics (TOG) , 38(4):1-15, 2019. 3

Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor scene generation with transformers. In 2021 International Conference on 3D Vision (3DV) , pages 106-115. IEEE, 2021. 3

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations , 2022. 4

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research , 2022. Survey Certification. 2

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. 2

Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 , 2023. 2, 3

Qiucheng Wu, Yujian Liu, Handong Zhao, Trung Bui, Zhe Lin, Yang Zhang, and Shiyu Chang. Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7766-7776, 2023. 2, 3

Cheng-Fu Yang, Wan-Cyuan Fan, Fu-En Yang, and Yu-Chiang Frank Wang. Layouttransformer: Scene layout generation with conceptual and spatial diversity. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3732-3741, 2021. 2

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In AAAI Conference on Artificial Intelligence , 2021. 4

Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381 , 2023. 3

Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14246-14255, 2023. 2, 3, 6, 7

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023. 2

Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923 , 2023. 3

Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau. Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics (TOG) , 38(4):1-15, 2019. 3

Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 1015-1022. IEEE, 2019. 3

A Implementation Details

In this section, we provide a detailed description of our prompt construction and instantiate instructions examples.

Task instructions As is shown in Table 7, the specific task instructions start with verbalized descriptions of the task and are followed by the formal definition of the CSS style. As for the indoor scene synthesis, we additionally provide a list of available furniture and the normalized frequency distribution for fair comparisons with the supervised method. Yet we discover that the provided frequency distribution has little effect on the generation results, based on the trivial change in the KL divergence. In some cases, it is important to make LLMs sample from a defined distribution instead of learning the distribution from in-context exemplars, which we leave for future work.

Base LLMs We use four variants of GPT models, (1) Codex [5] ( code-davinci-002 ), an LLM that is fine-tuned with large-scale code datasets and can translate natural language into functioning code snippets; (2) GPT-3.5 [36] ( text-davinci-003 ), which is trained to generate text or code from human instructions; (3) GPT-3.5-chat ( gpt-3.5-turbo ) and (4) GPT-4 [35] ( gpt-4 ), which are both optimized for conversational tasks. For the last two models, we first feed the in-context exemplars as multiple turns of dialogues between the user and the model to fit into the API design. However, we generally observe that GPT-3.5-chat and GPT-4 are not as strong as GPT-3.5 in learning from the in-context demonstrations, especially when the dialogue format follows a certain structure instead of free-form descriptions.

Hyperparameters For all LLMs, we fix the sampling temperature to 0.7 and apply no penalty to the next token prediction. For image layouts evaluation in Table 2, we fix the number of exemplars to 16 for numerical reasoning, and 8 for spatial reasoning, based on the best results of a preliminary experiment. However, we do not observe significant gaps in evaluation results when using different amounts of exemplars (see Sec. B.4). For each prompt, we generate five different layouts/images using baselines or LayoutGPT and thus result in 3810 images for numerical reasoning and 1415 images for spatial reasoning in all reported evaluation results. As for indoor scene synthesis, we fix the number of exemplars to 8 for bedrooms and 4 for living rooms to reach the maximum allowed input tokens. We set the maximum output token as 512 for bedrooms and 1024 for living rooms as bedrooms have ' 5 objects per room while living rooms have ' 11 objects per room. We generate one layout for each rectangular floor plan for evaluation.

s

∑

=

∑

∑

1

3

(

|

c

i

x

c

i

x

′

c

i

min

(

x

c

x

∑

min

c

′

k

k

′

(

x

c

x

∑

|

k

c

k

,

x

,

c

′

k

x

-

n

x

2 - 1

c

′

|

′

c

x

k

cat

2

1

)

c

′

k

|

+

k

)

|

=

bed

1

0

1 + 0 + 2

1 + 0 + 3

=

2 + 1 + 2

1 + 0 + 2

1 - 0

B LayoutGPT for 2D Layout Planning

B.1 NSR-1K Benchmark Construction

We rely on the MSCOCO annotations to create NSR-1K with ground-truth layout annotations. Note that each image in COCO is paired with a set of captions and a set of bounding box annotations.

Numerical Reasoning We primarily focus on the competence of T2I models to count accurately, i.e., generate the correct number of objects as indicated in the input text prompt. The prompts for this evaluation encompass object counts ranging from 1 to 5. To design the template-based T2I prompts, we initially sample possible object combinations within an image based on the bounding box annotations. We only use the bounding box annotation of an image when there are at most two types of objects within the image. As a result, the template-based prompts consist of three distinct types: (1) Single Category , wherein the prompt references only one category of objects in varying numbers; (2) Two Categories , wherein the prompt references two categories of distinct objects in varying numbers; and (3) Comparison , wherein the prompt references two categories of distinct objects but specifies the number of only one type of object, while the number of the other type is indicated indirectly through comparison terms including 'fewer than', 'equal number of', and 'more than'. As for natural prompts, we select COCO captions containing one of the numerical keywords from 'one' to 'five' and filter out those with bounding box categories that are not mentioned to avoid hallucination.

Spatial Reasoning We challenge LLMs with prompts that describe the positional relations of two or more objects. Our spatial reasoning prompts consist of template-based prompts and natural prompts from COCO. To construct template-based prompts, we first extract images with only two ground-truth bounding boxes that belong to two different categories. Following the definitions from PaintSkill [7], we ensure the spatial relation of the two boxes belong to (left, right, above, below) . Specifically, given two objects A,B , their bounding box centers p x A , y A q , p x B , y B q and the Euclidean distance d between two centers, we define their spatial relation Rel p A,B q as:

pillow

The definition basically dissects a circle centered at A equally into four sectors that each represent a spatial relation. While the definition may not stand for all camera viewpoints, it allows us to mainly focus on the front view of the scene. Then, we utilize the category labels and the pre-defined relations to form a prompt, as is shown in Table 1. As for the natural COCO prompts, we select prompts that contain one of the key phrases (the left/right of, on top of, under/below) and ensure that the bounding box annotations align with our definition. precicion = ∑ min ( x c k , x ′ c k ) ∑ x ′ c ′ = 1 + 0 + 2 1 + 0 + 3 = 75 %

2

3

B.2 Evaluation Metrics Categories

cat bed

pillow

c

i

Ground Truth recall

=

∑

min

(

x

k

∑

c

k

,

x

′

c

k

)

x

=

1 + 0 + 2

c

= 60 %

We denote the set of n object categories in the ground truth annotation as C GT ' c 1 , c 2 , . . . , c n , where x c 1 , x c 1 , . . . , x c n represent the number of objects for each category. Additionally, we denote the set of m object categories mentioned in GPT-3.5/4's layout prediction as C pred ' c 1 1 , c 1 2 , . . . , c 1 m , where x 1 c 1 1 , x 1 c 1 2 , . . . , x 1 c 1 m represent the number of objects for each category accordingly. If a category c i is not mentioned in C pred , then x 1 c i is assigned a value of 0, and vice versa. = 75 % = 60 % Prediction 1 0 3 x ′ c i MAE = ∑ | x c k -x ′ c k | n = 1 3 ( | 2 - 1 | + | 1 - 0 | + | 2 - 3 | ) = 1

|

+

|

2 - 3

|

) = 1

2

1

2

x

c

i

′

k

2 + 1 + 2

The numerical reasoning ability of GPT-3.5/4 on layout planning is assessed using the following metrics: (1) precision : calculated as ř n k ' 1 min p x c k ,x 1 c k q ř m k ' 1 x 1 c 1 k , is an indication of the percentage of predicted objects that exist in the groundtruth; (2) recall : calculated as ř n k ' 1 min p x c k ,x 1 c k q ř n k ' 1 x c k , indicates the percentage of ground-truth objects that are covered in the prediction; (3) accuracy : In the 'comparison' subtask, an accuracy score of 1 is achieved when the predicted relation, whether it is an inequality or equality, between the two objects is accurately determined. For all other numerical subtasks, accuracy equals to 1 if the predicted categories and object numbers precisely match the ground truth. In other cases, the accuracy is 0. Fig. 9 shows an example of how we compute the precision and recall . The accuracy for this single example is 0 since the predicted object distribution does not match the ground truth in every category.

For spatial reasoning, we evaluate spatial accuracy based on the LLM-generated layouts and GLIPbased layouts. We adopt [26] finetuned on COCO to detect involved objects from the generated images and obtain the bounding boxes. For both types of layouts, we categorize the spatial relation based on the above definition and compute the percentage of predicted layouts with the correct spatial relation. For all evaluation benchmarks, we measure the CLIP similarity, which is the cosine similarity between the generated image feature and the corresponding prompt feature.

B.3 GPT-3.5/4 Prompting

In Sec. 4.4, we investigate the impact of three components in the structured prompts: (1) Instruction , which examines whether detailed instructions explaining the task setup and the format of the supporting examples are included in the prompt. (2) Structure , which evaluates the impact of different formatting settings on the presentation of the bounding box aspects of height, width, top, and left. The 'w/ CSS' setting formats the aspects in CSS, while the 'w/o CSS' setting presents the four aspects in a sequence separated by a comma. (3) Normalization , which investigates the effects of rescaling the bounding box aspects to a specified canvas size and presenting them as integers in pixels in the 'w/ Norm.' setting, while the 'w/o Norm.' setting presents the aspects as relative scales to the canvas size in floats that range from (0, 1).

Table 7 shows the detailed prepending instructions LayoutGPT provided to GPT-3.5/4 models during 2D layout planning. Table 8 compares the formats of supporting examples with ablated structures and normalization settings.

For experiments in Sec. 4.3, to adapt to the nature of the Panoptic task, we add the following additional instruction when prompting LayoutGPT: 'The objects layout might be dense, and objects may overlap with each other. Some objects might not have been mentioned in the prompt, but are very likely to appear in the described scenario. ' To generate counterfactual prompts for text-to-image generation, the following text prompt is provided to GPT-4: 'Please provide a few counterfactual prompts that

In-context

exemplar

LayoutGPT

In-context

exemplar

LayoutGPT

In-context

exemplar

LayoutGPT

adoginthewaterwith

afrisbeeinitsmouth

a close upof a dog

nearabowl

oneorangeandoneapple

aburlapsurface

anorangeanappleand

acloseupofafork

anonion

ahalfconstructedroom

withaoveninsideof

it

stove,andanoven.

Akitchenwitha

refrigerator,

bop

fork

oven

frisbee

bowl

orC

apple

inge

apple

oven

refrigetotor

bop

O

a

abird thathasmany

acloseupofabird

Theviewofaknife

A closeupof anapple

a dog standing on top

colorsonit

onalaptop

slicingintoanapple.

nexttoaknife

ofadogonacow'sback

ofacow

D10

laptop

knife

aiddp

knife

dow

6op

COW

bird

depict rarely seen the spatial relationship between the 80 MSCOCO object categories. An example would be "a monkey riding on top of a bird"' .

B.4 Additional Experiments

Random In-Context Exemplars Empirically, selecting in-context exemplars can be critical for the overall performance of LLMs. Apart from our retrieval-augmented method in Sec. 3, we also experiment with a fixed random set of in-context exemplars. Specifically, we randomly sample k examples from the training (support) set D to form a fixed set of in-context demonstrations for all test conditions C j . Therefore, the fixed random setting results in in-context exemplars that are unrelated to the test condition C j . The minor gap between lines 1&5 in Table 9 verifies that LayoutGPT is not directly copying from the in-context exemplars in most cases. Fig. 10 further justifies the argument with layout visualization of the most similar in-context exemplars and the LayoutGPT outputs.

Number of In-Context Exemplars We take a closer look at the effects of the number of in-context exemplars in the prompt as shown in Table 9. For counting, we observe that the number of exemplars is positively correlated with the counting accuracy. We conjecture that LLMs learn to make more accurate predictions for challenging prompts (e.g., comparison) by learning from more few-shot exemplars. As the layout accuracy also accounts for results where CSS parsing fails, we observe that the LLMs generate more consistent CSS-style code by learning from more examples. However, we cannot observe a similar trend in spatial reasoning prompts. We conjecture that LLMs only require as few as four demonstrations to learn the differences between the four types of spatial relations. The small optimal number of in-context exemplars implies that LLMs already have 2D spatial knowledge and can map textual descriptions to corresponding coordinate values. Yet it is important to find a proper representation to elicit such knowledge from LLMs as implied in Sec. 4.4.

Attribute  Binding

Input (to LayoutGPT) : A red bird fluttering next to a car window.

Output (of LayoutGPT) :

a red bird with wings fluttering

a car window with the window open

Input : A gray and brown cat sitting under a purple umbrella.

Output :

a gray and brown cat in a sitting position

a purple umbrella

Input : A brown horse standing next to a white truck.

Output :

a brown horse standing still

a white truck with four black wheels

(Visualized)

Performance on Numerical Subtasks Table 10 presents the performance of layout generation in various numerical reasoning subtasks. Regarding template-based prompts, the LayoutGPT demonstrates superior performance in the 'Single Category' numerical reasoning task, exhibiting precision, recall, and accuracy values around 86%. However, when it comes to the 'Two Category' numerical reasoning task, while precision and recall experience minimal changes, the accuracy drops to 66%. For the 'Comparison' subtask, the accuracy hovers around 78%. These outcomes indicate that LayoutGPT encounters greater challenges when confronted with multi-class planning scenarios, whether the number of objects is explicitly provided or indirectly implied through comparative clauses.

For natural prompts extracted from MSCOCO, a noteworthy observation is the high recall accompanied by relatively lower precision. This discrepancy arises due to the ground truth bounding box annotations encompassing only 80 object classes, whereas the natural prompts may mention objects beyond the annotated classes. Consequently, our LayoutGPT may predict object layouts corresponding to classes not present in the ground truth, which, despite lowering precision, aligns with the desired behavior.

Performance on Size Comparison Reasoning Task We evaluate LayoutGPT's reasoning ability regarding object size. We use the standard HRS benchmark [2] which is designed for benchmarking compositional text-to-image models. HRS prompts for size reasoning contain comparison terms between randomly sampled common objects. The size relations described in HRS size prompts are often counterfactual and rarely seen (e.g., 'a person which is smaller than a chair and larger than horse' , 'a car which is smaller than a banana and chair and bigger than airplane'. ). LayoutGPT achieves an accuracy of 98.0% / 93.1% / 92.1% when the prompt involves size comparison between 2/3/4 objects. Meanwhile, the best size reasoning performance of nine text-to-image models reported by the HRS benchmark is only 31.1% / 0.2% / 0%. The results further verify that LayoutGPT acquires decent reasoning ability on rare scenarios / counterfactual prompts.

B.5 Failure Cases

Fig. 12 shows typical failure cases in numerical and spatial relations. As previously discussed, we observe in Table 10 that numerical prompts that involves two type of objects ('Two Categories' and 'Comparison') are more challenging to LayoutGPT and the image generation model. In these subtasks, LayoutGPT tends to predict much smaller bounding boxes to fit all objects within the limited image space. The small boxes further challenge GLIGEN to fit the object within the limited region, as shown in Fig. 12 (right).

Input

: Grey and black cat sitting under

Output

:

a grey and black cat with its tail tucked its head down, looking at the viewer

a wooden bench with two legs

'Four zebras grazing in enclosed area with shade trees'

'A light shines on five clocks

showing times in different zones'

'A vintage photo of a dog laying

on a couch'

Numerical failure

Spatial failure

Incompatible layouts

LayoutGPT

LayoutGPT+GLIGEN

LayoutGPT

LayoutGPT+GLIGEN

LayoutGPT

LayoutGPT+GLIGEN

Human

Human+GLIGEN

zebra

bop

doa

clock

zebra

zebra

MM

zebra

zebra

Modification

based on

exemplars

Duplication

from exemplars

Generation

(new objects, significant differences in object sizes or locations)

In-context exemplar

LayoutGPT

In-context exemplar

LayoutGPT

In-context exemplar

LayoutGPT

differences

30

20

scene

10

Q

Q

100

200

300

400

sortedsamples

C LayoutGPT for 3D Scene Synthesis

Due to the limitation in datasets, the conditions are room type and room size instead of text descriptions. While ATISS [38] utilizes the floor plan image as the input condition, LLMs are not compatible with image inputs. Therefore, we convert the floor plan image into the specification of the room size. Therefore, the input conditions are similar to ' Room Type: Bedroom, Room Size: max length 256px, max width 256px '.

C.1 Exemplar Selection

Similar to Sec. B.4, we investigate the effect of using a random set of in-context exemplars for indoor scene synthesis. When we apply 8 random bedroom layouts from the training set as in-context exemplars, the out-of-bound rate increases from 43.26% in Table 5 to 85.58%. The significant differences suggest that LayoutGPT heavily relies on rooms with similar floor plans to maintain objects within the boundary. Yet we verify that the generated layouts from LayoutGPT are not duplicates of the in-context exemplars in most cases.

We first define a training scene layout as a set of objects S t ' t o t 1 , . . . , o t m u , and a generated scene layout as S g ' t o g 1 , . . . , o g n u . Note that o j consists of category c j , location t j P R 3 , size s j P R 3 , and orientation r j P R , i.e. o j ' p c j , t j , s j , r j q We define the scene difference D p¨|¨q between S t and S t as

We set t t j , s t j to 0 if S t does not have a single object that belongs to the same category as c g i . For each testing sample of the bedroom, we compute the scene differences between the generated layout

Out-of-Bound Furniture

Overlapped Objects

Inharmonious placement

门

and all eight in-context exemplars and use the minimum value as the final scene difference. Note that all parameters used for computation are in 'meters' instead of 'pixels'.

We plot the scene differences of all 423 testing samples in Fig. 13. We empirically discover that a scene difference below 1.0 means S g is highly similar to S t , which we conclude as duplication from in-context exemplars. A scene difference below 6.0 shows moderate differences in object sizes or locations between two scenes, representing a modification based on S t to generate S g . Finally, a scene difference larger than 6.0 represents new objects or significant differences in object sizes or locations between the exemplar and the generated layouts, i.e. true generation . Fig. 13 shows that 34/111/278 scenes belong to duplication/modification/generation. Among each category, 30/67/143 scenes have no out-of-bound furniture. Therefore, LayoutGPT is performing generation instead of duplicating in-context exemplars in most cases.

C.2 Failure Cases

While LayoutGPT achieves comparable results as ATISS, LayoutGPT cannot avoid typical failure cases as shown in Fig. 14, such as out-of-bound furniture and overlapped objects. Fig. 14 (right) shows an incorrect placement of nightstands on the same side of the bed while they are commonly placed on each side of the bed headboard. Future work could focus on more sophisticated in-context learning or fine-tuning methods to improve the LLMs' understanding of 3D concepts.

D LayoutGPT for 2D Keypoint Planning

In addition to its application in 2D and 3D layout planning, we investigate the feasibility of leveraging LayoutGPT for 2D keypoint planning to facilitate text-conditioned image generation. In this approach, we utilize LayoutGPT to predict keypoint distributions based on a given text prompt, and subsequently employ GLIGEN [27] for keypoint-to-image generation. The keypoint format used aligns with the specifications outlined in MSCOCO2017 [29], focusing on 17 keypoints that correspond to the human skeleton. Similar to our methodology for selecting supporting examples in the context of 2D layout planning (Section B), we retrieve the k -most similar examples from the training set of MSCOCO2017 and utilize these examples to provide keypoint distributions as input to GPT-3.5/4. Table 11 presents an illustrative example of the input format employed for keypoint planning with GPT-3.5.

Fig. 15 presents several illustrative examples that compare the images generated by conditioning on keypoints planned by our LayoutGPT with those generated by end-to-end models such as StableDiffusion-v2.1 [41] and Attend-and-Excite [4]. In this preliminary demonstration, we observe that LayoutGPT exhibits promising potential in offering inherent control over specific movements or actions through keypoint planning.

Nevertheless, it is worth noting that keypoints planning presents considerably greater challenges compared to bounding box layout planning, attributable to several evident factors. Firstly, keypoints planning necessitates the prediction of the positions of 17 nodes, which is significantly more complex than the 2D layout planning involving four aspects or the 3D layout planning encompassing seven aspects. Secondly, the distribution of keypoints encompasses a much larger array of spatial relations due to the numerous possible body movements. In contrast, previous 2D layout planning tasks only involve four types of spatial relations. These inherent complexities render keypoint planning heavily reliant on in-context demonstrations. However, the limited availability of annotations pertaining to

Instruction:

Given a sentence prompt that will be used to generate an image, plan skeleton keypoints layout of the mentioned objects. The skeleton keypoints include the following 17 nodes: nose, left_eye, right_eye, left_ear, right_ear, left_shoulder, right_shoulder, left_elbow, right_elbow, left_wrist, right_wrist, left_hip, right_hip, left_knee, right_knee, left_ankle, right_ankle. The generated keypoints layout should follow the CSS style, where each line starts with the keypoint node name and is followed by its absolute position.

Formally, each line should be like "node_name {left: ?px; top: ?px; }". Please follow this format strictly. Do not display in other variation of formats. Notice that some keypoint nodes may not be visible on the canvas. In such cases, simply put "node_name {left: 0px; top: 0px; }" for the invisible nodes. The image is 64px wide and 64px high. Therefore, all properties of the positions should not exceed 64px.

Prompt: a man on a surfboard in a river near a couple of trees and branches

Keypoints:

person#1:

[MORE SUPPORTING EXAMPLES]

Prompt: a man leaning on a surfboard in the water riding a wave Keypoints:

body movements in the MSCOCO dataset further exacerbates the challenges associated with reliable keypoint planning. Therefore, we leave the exploration of this potential direction to future research endeavors.

E Ethical Statement

In addition to the layouts predicted by GPT-3.5/4, we also incorporate human-planned layouts as a natural baseline for comparative analysis. To facilitate this, we provide annotators with an interface featuring a blank square space where they can draw bounding boxes. Alongside the input text prompt, we also present the noun words or phrases from the prompt to human annotators, instructing them to draw a bounding box for each corresponding element. We intentionally refrain from imposing additional constraints, enabling annotators to freely exercise their imagination and create layouts based on their understanding of reasonable object arrangements. To compensate annotators for their efforts, we offer a payment rate of $0.2 US dollars per Human Intelligence Task (HIT). The average completion time of approximately 30 seconds per HIT, which corresponds to an average hourly payment rate of $24.

F Limitations

The current work has several limitations that provide opportunities for future research. Firstly, while this work focuses on 2D and 3D bounding box layouts and makes a preliminary attempt at keypoints, there exist various other methods for providing additional spatial knowledge in image/scene

A close up of a monkey driving a motorcycle on a road

StableDiffusion v2.1

Attend-and-Excite

A chimpanzee holds a toothbrush in their hand

A person is standing in some water flying a kite

StableDiffusion v2.1

Attend-and-Excite

LayoutGPT

+ GLIGEN

StableDiffusion v2.1

Attend-and-Excite

fightettigbt_shouldeshoull

Vefteft

eftl

tarithigCiaht-wris

telbpwright_elbow

kighdeshoulde

cight_shoundeleft_elb

left_shoulder

left_wrist

Lright_eyonbthio

-rignt_wrist

generation, such as segmentation masks and depth maps. Future work could explore integrating LLMs with these alternative visual control mechanisms to broaden the scope of visual planning capabilities. Secondly, the current work primarily addresses visual generation tasks and lacks a unified framework for handling other visual tasks like classification or understanding. Extending the proposed framework to encompass a wider range of visual tasks would provide a more comprehensive and versatile solution. Thirdly, this work is a downstream application that attempts to distill knowledge from LLMs' extensive knowledge bases. Future research could explore more fundamental approaches that directly enhance the visual planning abilities of various visual generation models. By developing specialized models that are explicitly designed for visual planning, it may be possible to achieve more refined and dedicated visual generation outcomes. Overall, while the current work demonstrates the potential of using LLMs for visual planning, there are avenues for future research to address the aforementioned limitations and further advance the field of visual generation and planning.

G Broader Impact

The utilization of LLMs for conducting visual planning in compositional 2D or 3D generation has significant broader impacts. Firstly, LLMs alleviate the burden on human designers by simplifying the complex design process. This not only enhances productivity but also facilitates scalability, as LLMs can efficiently handle large-scale planning tasks. Secondly, LLMs exhibit remarkable capabilities in achieving fine-grained visual control. By conditioning on textual inputs, LLMs can easily generate precise and detailed instructions for the desired visual layout, allowing for precise composition and arrangement of elements. Moreover, LLMs bring a wealth of commonsense knowledge into the planning process. With access to vast amounts of information, LLMs can incorporate this knowledge to ensure more accurate and contextually coherent visual planning. This integration of commonsense knowledge enhances the fidelity of attribute annotations and contributes to more reliable and realistic visual generation outcomes.

It is worth noting that this work represents an initial foray into the realm of visual planning using LLMs, indicating the potential for further advancements and applications in this area. As research in this field progresses, we can anticipate the development of more sophisticated and specialized visual planning techniques, expanding the scope of LLMs' contribution to diverse domains, such as architecture, virtual reality, and computer-aided design.

H Additional Qualitative Examples

We present additional visual showcases to demonstrate the capabilities of LayoutGPT in different contexts. Fig. 16 showcases examples related to 2D numerical reasoning, Fig. 17 illustrates examples of 2D spatial reasoning, and Fig. 18 displays examples of 3D scene synthesis. These showcases offer further insights into the effectiveness and versatility of our approach across various domains.

Alightshinesonfiveclocksshowingtimesindifferentzones.

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

lclock

ck

Cl

clodnkrdadk

000

totadkck

福

Thesethreebirdsarewalkingalongthebeachlookingforfood

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

ird

Aphotoofatrainstationwithtwo trainsonthetracks.

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

train

train

train

train

train

Therearethreeelephantsstandingbesideapoolofwater.

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

elephant

elephant

elephant

lephondnt

threeairplanesthatarelandednearalargecity

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

airplane

girpdopdan

Therearefourvasesthathavebutterflysculpturesonthem.

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

eas

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

bicycle

water

bicycle

bikepath

bike-

dath

bicycle

water

DIKe

-path

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

doglaptop

laptop

laptop

Ablackcatlayinginthesununderabench

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

bench

behch""

bench

bench

bench

cat

acuptotherightofadonut

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

donut

cup

cup

donut

cup

donut

cup

Ablackcatlayingontopofabednexttopillows.

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

bed

bed

bed

Ded

cat

cat

pillow

Severalglassbeerbottlesunderaparkbench.

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

bottles

bottles

acowtotheleftofacar

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

coW

cOW

car

COW

a cuptotheleftofadog

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

doq

dog

doa

cup

dno

cup

A banana sitting on top of a cup on a desk.

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

banana

banana

cup

cup

Acatsittingonawoodfloorunderawhiteandblueumbrella.

SD2.1

Attend-n-Excite

GPT-3.5

+GLIGEN

GPT-3.5,chat

+GLIGEN

+GLIGEN

cat

cat

cat

GPT-3.5

I

GPT-3.5-chat

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper highlights the challenges faced by existing models in generating objects with specified counts, positions, attributes, and sizes, and emphasizes the need for compositional skills that can effectively arrange components coherently, accurately reflecting object specifications and interactions. \nThe authors propose LayoutGPT which aims to improve the utility of visual planning skills of Large Language Models (LLMs) by in-context learning. \nThe method shows promising results in generating plausible layouts in multiple domains including 2D images and 3D indoor scenes.\nExperiments show that LayoutGPT outperforms text-to-image models and achieves comparable performance to human users in designing visual layouts for numerical and spatial reasonings. \nIn addition, LayoutGPT also shows comparable performance to supervised methods in 3D indoor scene synthesis. \nThe paper also proposes a new benchmark called NSR-1K for evaluating generations in terms of specified counts and spatial locations.\n\nSTRENGTHS:\n1. The authors propose a novel solution for handling the issue that the current visual; generation models lack visual arrangments ability. They adopt LLMs as visual planners to generate layout information of objects in the target 2D images or 3D scenes, with the help of in-context visual demonstrations in style sheet language.\n2. LayoutGPT generates reasonable layouts in multiple domains, including 2D images and 3D indoor scenes.\n3. They build a challenging benchmark that characterizes counting and positional relations for text-to-image generations.\n4. Experiments show the effectiveness of the proposed method.\n5. The paper presents promising results and provides insights into the application of LLMs for visual planning and generation, both in terms of methodology and empirical findings.\n\nWEAKNESSES:\n1. The writing of motivation is not clear due to a lot of inexplicable expressions.\n\n    - The claim that existing visual generative models are not equipped with various reasoning skills that exist in LLMs is dubious, given the disparate nature of reasoning skills required for visual generative models compared to LLMs. Good reasoning skills are not equal to good generative effects. Therefore, the T2I models that fail to generate objects with specific counts, positions, and attributes do not indicate lacking reasoning skills. \n\n    - In the sentence, 'But unlike LLMs, ... discrete categories ...', do the authors mean LLMs are already capable of generating layout? if yes, there should have some citations, and there is no need to further introduce the LLMs as a centralized model because there is no logical correlation with the previous text. In addition, why did the authors strengthen the discrete categories, could the LLMs generate continuous categories?\n\n    - Unclear description of the drawbacks of the existing LLMs-centered systems. \n\n    - The motivations behind constructing layouts with structured programs are not sufficiently persuasive, considering that LLMs also are trained on the plain text, and alternative methods exist for representing image layouts. In fact, the strict imposition of a structured format may pose challenges for LLMs.\n\n    - To my knowledge, visual inputs generally refer to images or videos, and the LLMs are only able to handle textual inputs. Moreover, both tasks in this work involve textual information as input, and there is no experimental evidence supporting the claim that LLMs have the potential in handling complex visual inputs.\n\n\n    - The meaning of \"addressing the inherent multimodal reasoning skills of LLMs\" requires clarification. To my understanding, LLMs already possess visual planning capabilities, and the authors' objective is to effectively leverage these skills rather than fundamentally modify or address them.\n\n\n2. Inconsistent notations.\n \n    - $o_j$ vs $\\mathbf{o}_j$ in the section 3.1.\n\n    - $o_j$ represents the layout of an object, so the sequence should be (c_j, x_j, y_j, w_j, h_j).\n\n\n\n3. The utilization of CSS as a structured format for representing layouts is a bit farfetched and unnecessary. Firstly, LLMs are capable of understanding the meaning of each element in a plain sequence (o) if detailed task instruction is provided. Secondly, structured representation is not solely limited to the CSS format; alternative formats such as, 'layout: c_1(x_1, y_1, w_1, h_1), c_2(x_2, y_2, w_2, h_2), ...'.\n\n\n4. The reasons behind the lower performance observed when GLIGEN generates images based on ground truth (GT) layouts during image evaluation, as well as when GLIGEN synthesizes images using layouts provided by humans, remain unexplained in Table 2.\n\n\n5. It is important to note that LayoutGPT is specifically designed for generating layouts, encompassing object categories and corresponding bounding boxes. Therefore, it is unreasonable to conclude that LayoutGPT can accurately perform attribute binding. Correct attribute binding is only achieved when the attributes of objects in the generated images align with the textual descriptions. However, wrong attribute bindings between the generated images and sentences are apparent, such as the closed car window in the top row of Figure 4 and the absence of a handle on the basket in the bottom row of Figure 4.\n\nQUESTIONS:\n1. The authors mention that the in-context examples are provided in reverse order based on similarity. Does the order have an impact on the effectiveness of the model？\n2. How are the colored sentences in Figure 4 generated?\n3. For 3D layout planning, how do the furniture frequencies in the instruction impact the performance?\n4. Why does the CSS structure have more impact on task performance compared to instruction?\n\nLIMITATIONS:\nI do not see any limitations to this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose LayoutGPT that can generate visual arrangements of objects using the input prompts, providing a way to collaborate with visual generative models for compositional layout based image generation in both 2D and 3D. Experiment results show that such a method can largely improve layout-based generation using reasoning ability of large language models.\n\nSTRENGTHS:\n1. The paper is well written and easy to follow.\n2. The authors have conducted a comprehensive evaluations/ablations on the method.\n3. The paper provides an interesting way to connect large language models with visual generative models for image generation without any additional training.\n\nWEAKNESSES:\n1. **The paper simply proposes a module using LLMs to conduct visual planning**, i.e., extract / reason object relations and the number of objects when given input text descriptions. For image generation, extracted bounding boxes are simply used as inputs to existing layout-based methods, thus no technical contributions to visual generative models.\n2. **The paper over-claimed more or less.** For example, in Line 196, the author claimed \"LayoutGPT can perform accurate attribute binding\". However, in text-based inpainting section of Figure 4, the purple suitcase doesn't have the specified design \"a blue, yellow and white flower\" and the cat isn't black and white. Similarly, the spatial relationships extracted from the method seem to be off. The first example of text-based inpainting, the cat should be sitting under a bench, thus it seems to me that the method isn't that reliable.\n3. **Lacking some baselines**. For 2D image generation, there are a few works that do layout-based image generation, specifically using bounding boxes. It would be good to compare with these methods. One example I can think of is [1], which also doesn't require additional training.\n4. **Related Works**. It would be quite relevant to include compositional image generation, where you generate images conditioned on multiple specifications or objects.\n\n[1] Chen et al., Training-free layout control with cross-attention guidance (CVPR 2023)\n\nQUESTIONS:\n1. In table 4, when generating images, is the mainly prompt used as the input or are those colored sentences used as inputs?\n2. It seems to me that layoutGPT can do a lot of guessing instead of reasoning. What if you use counter-factual examples or simple examples that rarely appear in real life?\n\nLIMITATIONS:\nThe authors don't include limitation section. One thing is that LLMs can be biased such that generated imagery can also biased.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces LayoutGPT, a training-free approach that injects visual commonsense into LLMs and enables generating plausible 2D images and 3D scenes conditioned on layouts based on text conditions. Specifically, the authors experiment with four variants of GPT models: Codex, GPT-3.5, GPT-3.5-chat and GPT-4 and showcase that LLMs can produce meaningful 2D/3D layouts using a CSS (Cascading Style Sheets) format, where every object is modelled as labelled bounding box, parametrized with three random variables indicating its category, size and location. For the case for 3D indoor scene synthesis, they parse the layouts into 3D scenes by simply replacing the bounding boxes with 3D objects from a library of assets. For the case of 2D image synthesis, they rely on GLIGEN that is a layout-to-image model to convert the generated layout to a 2D scene. For the case of 2D image synthesis, the authors compare their model to Stable Diffusion and Attend-and-Excite and evaluate the generated layouts based on the precision, recall and accuracy of the generated bounding boxes. To measure whether the generated image matches the provided text description, the authors report the CLIP/GLIP cosine similarity between text prompts and the generated images. For the task of 3D indoor scene synthesis, the authors compare their model to ATISS and measure the generation quality by reporting the KL-divergence between the object category distributions in the ground-truth and the generated scenes. For both tasks, LayoutGPT outperforms most baselines on most metrics and from the qualitative results, it seems that LayoutGPT faithfully generate layouts that match the input conditioning.\n\nOverall, I think this is a nice work that introduces an elegant way for using LLMs for 2D and 3D layout synthesis. I think that the idea of representing scene layouts in the CSS format is very intuitive and greatly simplifies the task of layout synthesis. My main concern, as also discussed in the Weaknesses section, is related to whether the proposed model can robustly generate (i) layouts with more complex text conditioning for the case of the image synthesis, (ii) 3D scenes conditioned on detailed scene descriptions clearly describing the how many and what objects should be placed in the scene.\n\nSTRENGTHS:\n1. To the best of my knowledge, the idea of using LLM for 2D and 3D layout synthesis is novel and the authors clearly demonstrate that LLMs can produce complex 2D images and 3D scenes in a CSS format. Unlike other concurrent works that try to use LLMs for similar tasks, I think this model is simpler and more generic, hence it can be applied on both 2D scene synthesis and 3D scene synthesis.\n\n2. I particularly liked that the authors provide results on both a 2D and a 3D task. They compare their model with several strong baselines and showcase that the proposed model can consistently produce 2D images/3D that match the input conditioning. Moreover, from the quantitative evaluation, we note that the proposed training-free method achieves state-of-the-art performance w.r.t. most metrics for both tasks.\n\n3. I appreciated the additional ablations as well as the various implementation details provided in the supplementary. In addition, I think that also the proposed NSR-1K benchmark, can potentially be very useful for various tasks. Although the authors don't mention whether they plan on releasing these benchmark, I would like to encourage them to do so as I think such benchmarks can greatly benefit the research community.\n\nWEAKNESSES:\n1. For the case of the 3D indoor synthesis task, I am wondering whether the authors tried to condition the scene generation on more detailed text descriptions that go beyond simply specifying the room type and the size of the room. For example, given a description like \"a bedroom with one double bed, two nightstands and 1 wardrobe\", would LayoutGPT be able to generate a layout that matches this description? I assume that to some extend this should work quite well so I am not sure why the authors did not provide these type of results. I think it would be beneficial for the paper to include them for the final version of the paper.\n\n2. For the experimental evaluation of the 2D scene synthesis, the authors report precision, recall and accuracy. Is there a reason why not also report MeanIoU on the bounding box parameters? I am not sure whether I am missing out something but I think this metric is very important as the model generates bounding boxes in practice. In addition, I think that the section B2 in the supplementary that discusses the Evaluation metrics and in particular the accuracy computation is not very clear. I think it is good to polish this section a bit.\n\n3. Although, I appreciate that the authors proposed a new benchmark for their image synthesis experiment, I think they should have also evaluated their model on the 2017 Panoptic version of COCO dataset that has been previously used by several generative models that perform layout generation. From the description in L152-161, it is not clear to me whether the panoptic version of the COCO dataset is included in the proposed benchmark. Can the authors please clarify this?\n\nQUESTIONS:\n1. I am wondering whether the LLMs can produce diverse layouts conditioned on the same text conditioning. I believe that this is an interesting experiment that the authors should include in their evaluation. In addition, another valuable analysis would be to compare existing GPT models w.r.t. their capabilities for generating diverse layouts conditioned on the same input prompt. Being able to generate diverse layouts is a very important trait of existing models, hence I think the authors should provide additional experiments that demonstrate whether this is possible or not.\n\n2. Looking at the Experimental Evaluation and in particular the Image Synthesis results, I am wondering how robust is the proposed model if the input text prompt contains larger descriptions. Looking at all the results both in the main paper and in the supplement, I think that the authors show layout generations with at most 5 objects (see Fig 3., top row, right column). Have the authors tried to condition the layout generation with more detailed text prompts with more objects? How good would their model work? \n\n3. In Table 1, the authors mention that their proposed NSR-1K benchmark contains several text descriptions that have comparisons e.g. \"A picture of three cars with a few fire hydrants, the number of cars\nis more than that of fire hydrants\". I checked both the main paper and the supplementary but I was not able to find any conditioning like this. I think it would be great if the authors can provide some examples that show that their model works well with these more challenging text conditioning.\n\n4. Can the authors clarify why LayoutGPT cannot work with floor plans of various shapes. In L219-220 of  the main paper they state that it is not compatible with irregular floor plans but I am not sure why this is really an issue?\n\n5. For the quantitative evaluation in Section 5.1 the authors should also mention the image resolution of the rendered images, used to compute the FID scores. Moreover, they mention that to compute the FID score they render scene images from four camera angles (L224-225), are these angles random per scene? I think it is important that the authors clarify this for reproducibility purposes.\n\n6. Some paper references that are missing that I think the authors should include in their final version of their paper are listed below:\n* Variational Transformer Networks for Layout Generation, CVPR 2021\n* BLT: Bidirectional Layout Transformer for Controllable Layout Generation\n* LayoutDM: Discrete Diffusion Model for Controllable Layout Generation, CVPR 2023 (this is a concurrent work but still it might be good to add it in the reference list)\n\nLIMITATIONS:\nThe authors discuss the limitations of their work and show several failure cases in their supplementary material. In addition, they also discuss potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LayoutGPT, a method to compose in-context visual demonstrations in style sheet language to enhance the visual planning skills of LLMs. As the first work to use LLMs to generate layouts from text conditions, LayoutGPT can generate plausible layouts for 2D images and 3D indoor scenes, including challenging language concepts like numerical and spatial relations. Those generated layouts can be further used for image generation. When combined with a region-controlled image generation model, LayoutGPT outperforms existing text-to-image generation methods by 20-40% and achieves comparable performance as human users in generating plausible image layouts and obtaining images with the correct object counts or spatial relations.\n\nSTRENGTHS:\n1. This is the first work to explore the ability of LLMs for layout generation. It reveals the spatial reasoning ability of LLMs and might inspire future explorations in this direction.\n2. The training-free approach is easy to adopt for various applications.\n3. The presentation is clear and easy to follow.\n\nWEAKNESSES:\n1. Although this was no such exploration before, the proposed approach is a straightforward application of LLM. It would be better and more inspiring if authors could provide some in-depth analysis of the spatial reasoning abilities of LLM.\n2. It is unclear if layoutGPT is robust to the selection of in-context exemplars or not. What's the size of the reference set? What if the text condition describes a rare scenario that does not appear in the reference set? Experiments in the supplementary material show that the performance is sensitive to the number selected in-context exemplars. \n3. The evaluation is conducted only on NSR-1K for numerical reasoning and spatial reasoning in text-to-image synthesis and ATISS for indoor scene synthesis. Although the authors claim that layoutGPT can be used for accurate attribute binding and text-based inpainting, only visual results (fig.4) are shown and there are no quantitative experiments on such datasets to demonstrate such abilities and applications.\n\nQUESTIONS:\nPlease refer to the questions in the weakness section.\n\nLIMITATIONS:\nThe authors mention the limitations in the supplementary materials.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}