SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Weixi Feng $^{1*}$  Wanrong Zhu $^{1*}$  Tsu-jui Fu $^{1}$  Varun Jampani $^{2}$  Arjun Akula $^{2}$  Xuehai He $^{3}$  Sugato Basu $^{2}$  Xin Eric Wang $^{3}$  William Yang Wang $^{1}$
<sup>1</sup>University of California, Santa Barbara  
<sup>2</sup>Google
<sup>3</sup>University of California, Santa Cruz
https://github.com/weixi-feng/LayoutGPT
StableDiffusion (v2.1)
Attend-andExcite
[2D Numerical Reasoning] There are three elephants standing beside a pool of water.
[2D Spatial Reasoning] A carrot and some onion next to a knife on a cutting board.
[3D Living Room] Room Type: Living Room; Room Size: 7.7m x 3.6m
StableDiffusion (v2.1)
Attend-and-Excite
LayoutGPT + GLIGEN
[3D Bedroom] Room Type: Bedroom; Room Size: 3.0m x 4.8m
Figure 1: Generated layouts from LayoutGPT in 2D images and 3D indoor scenes. LayoutGPT can serve as a visual planner to reflect challenging numerical and spatial concepts in visual spaces.
Attaining a high degree of user controllability in visual generation often requires intricate, fine-grained inputs like layouts. However, such inputs impose a substantial burden on users when compared to simple text inputs. To address the issue, we study how Large Language Models (LLMs) can serve as visual planners by generating layouts from text conditions, and thus collaborate with visual generative models. We propose LayoutGPT, a method to compose in-context visual demonstrations in style sheet language to enhance the visual planning skills of LLMs. LayoutGPT can generate plausible layouts in multiple domains, ranging from 2D images to 3D indoor scenes. LayoutGPT also shows superior performance in converting challenging language concepts like numerical and spatial relations to layout arrangements for faithful text-to-image generation. When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by  $20 - 40\%$  and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness. Lastly, LayoutGPT achieves comparable performance to supervised methods in 3D indoor scene synthesis, demonstrating its effectiveness and potential in multiple visual domains.
*equal contribution, correspondence to {weixifeng, wanrongzhu}@cs.ucsb.edu
Can Large Language Models (LLMs) comprehend visual concepts and generate plausible arrangements in visual spaces? Recently, LLMs have shown significant advancement in various reasoning skills [50, 49] that remain challenging to visual generative models. For instance, text-to-image generation (T2I) models suffer from generating objects with specified counts, positions, and attributes [10, 24]. 3D scene synthesis models face challenges in preserving furniture within pre-defined room sizes [30]. Addressing these issues necessitates the development of compositional skills that effectively arrange components in a coherent manner, accurately reflecting object specifications and interactions.
Visual layout is an essential symbolic representation that has been widely studied as it reflects the compositions of a visual space [33, 53, 45, 34]. For instance, layout generation models [21, 25, 17, 53, 23] can be combined with region-controlled image generation methods [56, 27] to improve image compositionality [52]. But unlike LLMs, these models are restricted to discrete categories or have limited reasoning skills for complicated text conditions. Recently, LLMs like ChatGPT [37], are adopted as a centralized module of frameworks or systems where multiple foundational computer vision models are integrated. Through defined action items or API calls, LLMs can interact with visual generative models to extend the systems' capability into image generation tasks. [51].
Despite the advancement, existing approaches that involve the collaboration between LLMs and image generation models are either limited to executing the latter through program generation or using LLMs for language data augmentation for image editing [3]. Current LLM-based systems fail to improve the compositional faithfulness of a generated image by simply using T2I models through API calls. While one could additionally integrate models that synthesize images with the guidance of layouts [56, 27], keypoints [27], or sketches [20, 57], users still have to create fine-grained inputs on their own, leading to extra efforts and degraded efficiency compared to pure language instructions.
To address these challenges, we introduce LayoutGPT, a training-free approach that injects visual commonsense into LLMs and enables them to generate desirable layouts based on text conditions. Despite being trained without any image data, LLMs can learn visual commonsense through in-context demonstrations and then apply the knowledge to infer visual planning for novel samples. Specifically, we observe that representing image layouts is highly compatible with how style sheet language formats images on a webpage. Therefore, as LLMs are trained with program data, constructing layouts as structured programs may enhance LLMs' ability to "imagine" object locations from merely language tokens. Our programs not only enable stable and consistent output structures but also strengthen LLMs' understanding of the visual concepts behind each individual attribute value. When combined with a region-controlled image generation model [27], LayoutGPT outperforms existing methods by  $20 - 40\%$  and achieves comparable performance as human users in generating plausible image layouts and obtaining images with the correct object counts or spatial relations.
In addition, we extend LayoutGPT from 2D layout planning to 3D indoor scene synthesis. With a slight expansion of the style attributes, LayoutGPT can understand challenging 3D concepts such as depth, furniture sizes, and practical and coherent furniture arrangements for different types of rooms. We show that LayoutGPT performs comparably to a state-of-the-art (SOTA) supervised method. Our experimental results suggest that LLMs have the potential to handle more complicated visual inputs. Our contribution can be summarized as the following points:
- We propose LayoutGPT, a program-guided method to adopt LLMs for layout-based visual planning in multiple domains. LayoutGPT addresses the inherent multimodal reasoning skills of LLMs and can improve end-user efficiency.
- We propose Numerical and Spatial Reasoning (NSR-1K) benchmark that includes prompts characterizing counting and positional relations for text-to-image generation.
- Experimental results show that LayoutGPT effectively improves counting and spatial relations faithfulness in 2D image generation and achieves strong performance in 3D indoor scene synthesis. Our experiments suggest that the reasoning power of LLMs can be leveraged for visual generation and handling more complicated visual representations.
Image Layout Generation Layout generation has been an important task for automatic graphical design for various scenarios, including indoor scenes [40, 46], document layouts [59, 60, 15], and graphical user interface [8]. Previous work has proposed various types of models that need to be trained from scratch before generating layouts. LayoutGAN [25] is a GAN-based framework to generate both class and geometric labels of wireframe boxes for a fixed number of scene elements. LayoutVAE [21] generates image layouts conditioned on an input object label set. Transformer-based methods are proposed to enhance flexibility in the layout generation process. For instance, LayoutTransformer [17] adopts self-attention to learn contextual relations between elements and achieve layout completion based on a partial layout input. BLT [23] proposes a hierarchical sampling policy so that any coordinate values can be modified at the sampling stage to enable flexible and controlled generation. However, existing methods are restricted to class labels and fail to reason over numerical and spatial concepts in text conditions. In contrast, LayoutGPT can convert challenging textual concepts to 2D layouts and generate free-form, detailed descriptions for each region.
Compositional Image Generation Recent studies have shown that text-to-image generation (T2I) models suffer from compositional issues such as missing objects, incorrect spatial relations, and incorrect attributes [24, 2]. StructureDiffusion [10] proposes to adjust text embeddings by utilizing prior knowledge from linguistic structures. Attend-and-Excite [4] optimizes attention regions so that objects attend on separate regions. Another line of work strives to introduce extra conditions as inputs. For example, ReCo [56], GLIGEN [27], and Layout-Guidance [6] can generate images based on bounding box inputs and regional captions. [52] combines a layout generator and a region-controlled method to achieve accurate generation results. While we focus on layout generation, we also employ layout-to-image models to generate final images and show the effectiveness of LayoutGPT.
Indoor Scene Synthesis Indoor scene synthesis aims at generating reasonable furniture layouts in a 3D space that satisfies room functionality. Early work adopting autoregressive models requires supervision of 2D bounding boxes and other visual maps [40]. Later, SceneFormer [47] proposes to apply a set of transformers to add furniture to scenes. While previous work adopts separate models to predict different object attributes, ATISS [38] demonstrates that a single transformer model can generate more realistic arrangements while being more efficient. In this work, we investigate leveraging LLMs to achieve scene synthesis without any fine-tuning.
LLMs for Vision Language inputs have been an essential part of many vision language tasks [43, 11, 28, 14]. With the strong generalization ability of contemporary LLMs, recent work attempts to adapt the power of LLMs on multimodal tasks [31, 55]. For instance, multimodal chain-of-thought [58] trained a model to incorporate visual inputs as rationales for question answering. [22] proposes to learn translation parameters to map embeddings between visual and language domains such that an LLM can ground on both modalities. VisProg [18] and ViperGPT [44] use LLMs to design modular pseudocode instructions or executable Python programs to achieve visual reasoning. LLMScore [32] leverages LLMs to evaluate text-to-image models. Visual ChatGPT [51] proposes a prompt manager that supports the execution of various image generation models. In this work, we directly involve LLMs in the generation process by leveraging LLMs to design visual layouts through in-context learning and structured representations.
Given a condition  $\mathcal{C}$ , the goal of layout generation is to predict a set of tuples  $\mathcal{O} = \{\mathbf{o}_j|j = 1,2,\ldots ,n\}$  where each tuple  $\mathbf{o}_j$  denotes the layout information of a 2D or 3D bounding box of object  $j$ . In image planning,  $\mathcal{C}$  is the input text prompt,  $\mathbf{o}_j$  consists of a category  $c_{j}$ , bounding box location  $\mathbf{t}_j = (x_j,y_j)\in \mathbb{R}^2$  and bounding box size  $\mathbf{s}_j = (w_j,h_j)\in \mathbb{R}^2$ , i.e.  $\mathbf{o}_j = (c_j,\mathbf{t}_j,\mathbf{s}_j)$ . Similarly, in 3D scene synthesis,  $\mathcal{C}$  specifies the room type and room size,  $\mathbf{o}_j$  consists of category  $c_{j}$ , location  $\mathbf{t}_j\in \mathbb{R}^3$ , size  $\mathbf{s}_j\in \mathbb{R}^3$ , and orientation  $\mathbf{r}_j\in \mathbb{R}$ , i.e.  $\mathbf{o}_j = (c_j,\mathbf{t}_j,\mathbf{s}_j,\mathbf{r}_j)$ . While  $c_{j}$  can be modeled as a discrete value, our method directly predicts the category text.
Figure 2: The overview process of our LayoutGPT framework performing 2D layout planning for text-conditioned image generation or 3D layout planning for scene synthesis.
3.2 LayoutGPT Prompt Construction
As is shown in Fig. 2, LayoutGPT prompts consist of three main components: task instructions, and in-context exemplars in CSS structures with normalization.
CSS Structures In autoregressive layout generation,  $\mathbf{o}_j$  is usually modeled as a plain sequence of values, i.e.  $(c_1, x_1, y_1, w_1, h_1, c_2, x_2, \ldots)$  [17, 23]. However, such a sequence can be challenging for LLMs to understand due to underspecified meaning of each value. Therefore, we seek a structured format that specifies the physical meaning of each value for LLMs to interpret spatial knowledge. We realize that image layouts are highly similar to how CSS (short for Cascading Style Sheets) formats the layout of a webpage and defines various properties of the img tag in HTML. For instance,  $x_j, y_j$  corresponds to the standard properties left and top, while  $w_j, h_j$  corresponds to width and height in CSS. As LLMs like GPT-3.5/4 are trained with code snippets, formatting image/scene layouts in CSS structures potentially enhances the LLMs' interpretation of the spatial meaning behind each value. Therefore, as is shown in Fig. 2, we place category name  $c_j$  as the selector and map other attribute values into the declaration section following standard CSS styles.
Task Instructions & Normalization Similar to previous work in improving the prompting ability of LLMs [48, 42, 37], we prepend task instructions to the prompt to specify the task goal, define the standard format, unit for values, etc. Besides, as the common length unit of CSS is pixels (px), we normalize each property value based on a fixed scalar and rescale the value to a maximum of 256px. As will be shown in later sections (Sec. 4.4 & 5.4), all three components play important roles in injecting visual commonsense into LLMs and improving generation accuracy.
3.3 In-Context Exemplars Selection
Following previous work [1, 54], we select supporting demonstration exemplars for in-context learning based on retrieval results. Given a test condition  $\mathcal{C}_j$  and a support set of demonstrations  $\mathcal{D} = \{(\mathcal{C}_k,\mathbf{o}_k)|k = 1,2,\ldots \}$ , we define a function  $f(\mathcal{C}_k,\mathcal{C}_j)\in \mathbb{R}$  that measures the distances between two conditions. For 2D text-conditioned image layout generation, we adopt the CLIP [39] model to extract text features of  $\mathcal{C}_j$  (usually a caption) and the image feature of  $\mathcal{C}_k$  and measure the cosine similarity between them. For the 3D scene synthesis task where each room has length  $rl$  and width  $rw$ , we measure distance with  $f(\mathcal{C}_k,\mathcal{C}_j) = \| rl_k - rl_j\|^2 + \| rw_k - rw_j\|^2$ . We select supporting demonstrations with the top- $k$  least distance measures and construct them as exemplars following the CSS structure in Fig. 2. These supporting examples are provided to GPT-3.5/4 in reverse order, with the most similar example presented last.
Table 1: Dataset statistics and examples of the NSR-1K benchmark for image layout planning and text-to-image (T2I) generation with an emphasis on numerical and spatial reasoning.
Task	Type	Example Prompt	# Train	# Val	# Test
T2I Numerical Reasoning	Single Category	“There are two giraffes in the photo.”	14890	-	114
Two Categories	“Three potted plants with one vase in the picture.”	7402	-	197
Comparison	“A picture of three cars with a few fire hydrants, the number of cars is more than that of fire hydrants.”	7402	-	100
Natural	“A fenced in pasture with four horses standing around eating grass.”	9004	-	351
T2I Spatial Reasoning	Two Categories	“A dog to the right of a bench.”	360	-	199
Natural	“A black cat laying on top of a bed next to pillows.”	378	84
3.4 Image and Scene Generation
For text-conditioned image synthesis, we utilize a layout-to-image generation model to generate images based on the generated layouts. As for each object layout in 3D scene synthesis, we retrieve a 3D object based on the predicted category, location, orientation, and size following [38]. We directly render the scene with the retrieved 3D objects. See Sec. 4 & Sec. 5 for more details.
4 LayoutGPT for Text-Conditioned Image Synthesis
In this section, we provide an extensive evaluation of LayoutGPT for 2D text-to-image (T2I) synthesis and compare it with SOTA T2I models/systems. An ablation study is conducted to demonstrate the effect of individual components from LayoutGPT. We also showcase qualitative results and application scenarios of our method.
Datasets & Benchmarks To evaluate the generations in terms of specified counts and spatial locations, we propose NSR-1K, a benchmark that includes template-based and human-written (natural) prompts from MSCOCO [29]. Table 1 summarizes our dataset statistics with examples. For template-based prompts, we apply a set of filters to obtain images with only 1-2 types of object and then create prompts based on object categories and bounding box information. As for natural prompts, we extract COCO captions with keywords to suit the task of numerical reasoning (e.g. "four") or spatial reasoning (e.g. "on top of") and ensure that all objects from the bounding box annotations are mentioned in the caption to avoid hallucination. Each prompt from NSR-1K is guaranteed to have a corresponding ground truth image and layout annotations. Detailed benchmark construction processes are described in Appendix B.1.
Evaluation Metrics To evaluate generated layouts, we report precision, recall, and accuracy based on generated bounding box counts and spatial positions [9, 16]. For spatial reasoning, each prompt falls into one of the four types of relations (\{left, right, top, below\}) and we use the bounding box center for evaluation following PaintSkills [7]. To evaluate generated images, we first obtain bounding boxes from GLIP [26] detection results and then compute average accuracy based on the bounding box counts or spatial relations. We also report CLIP cosine similarity between text prompts and generated images for reference. Detailed metric descriptions are listed in Appendix B.2.
Baselines As we consider both layout evaluation and image evaluation, we compare LayoutGPT with end-to-end T2I models (Stable Diffusion [41], Attend-and-Excite  $[4])^{2}$  and two-stage systems that generate layouts first and then apply GLIGEN [27] as the layout-to-image model. We also evaluate ground truth layouts and human-drawn layouts as the theoretical upper bounds. The human-drawn layouts are collected through crowdsourcing, in which we specifically ask human annotators to draw layouts given text prompts. We slightly modify LayoutTransformer [17] as a baseline for supervised conditional layout generation. Detailed descriptions of baseline setups and human annotating are discussed in the Appendix A and E.
2Attend-and-Eccite uses Stable Diffusion (SD) as the generative backbone. For both end-to-end T2I models, we report results on SD v1.4 and SD v2.1.
Table 2: Comparison of our LayoutGPT with baseline methods in terms of counting and spatial correctness. Line 5-11 generates layout and adopts GLIGEN [27] for layout-guided image generation. "Human" (line 11) denotes layouts collected from human users given text prompts. Text in bold shows the best results of LayoutGPT.
Methods	Numerical Reasoning	Spatial Reasoning
Layout Eval.	Image Eval.	Layout Eval.	Image Eval.
Precision	Recall	Accuracy	Acc. (GLIP)	CLIP Sim.	Accuracy	Acc. (GLIP)	CLIP Sim.
Text → Image
1	Stable Diffusion (v1.4) [41]	-	-	-	32.22	0.256	-	16.89	0.252
2	Stable Diffusion (v2.1)	-	-	-	42.44	0.256	-	17.81	0.256
3	Attend-and-Excite (SD v1.4) [4]	-	-	-	38.96	0.258	-	24.38	0.263
4	Attend-and-Excite (SD v2.1)	-	-	-	45.74	0.254	-	26.86	0.264
Text → Layout → Image
5	LayoutTransformer [17]	75.70	61.69	22.26	40.55	0.247	6.36	28.13	0.241
6	LayoutGPT (GPT-3.5)	94.81	96.49	86.33	51.20	0.258	82.54	52.86	0.264
7	LayoutGPT (Codex)	90.19	88.29	72.02	46.64	0.254	74.63	45.58	0.262
8	LayoutGPT (GPT-3.5, chat)	81.84	85.47	75.51	54.40	0.261	85.87	56.75	0.268
9	LayoutGPT (GPT-4)	78.36	86.29	78.43	55.64	0.261	91.73	60.64	0.268
10	GT layouts	100.00	100.00	100.00	53.23	0.256	100.00	62.54	0.261
11	Human	99.26	96.52	92.56	56.07	0.258	91.17	51.94	0.258
Stable Diffusion
Three clocks at a train station under a concrete arch (Numerical)
LayoutGPT LayoutGPT+GLIGEN Human+GLIGEN
LayoutGPT LayoutGPT+GLIGEN Human+GLIGEN
Two teddy bears and a stuffed snowman wearing hats (Numerical)
A yellow surfboard sits next to a bicycle on a brick sidewalk (Spatial)
Figure 3: Qualitative comparison between Stable Diffusion, LayoutGPT, and human annotations regarding numerical (top row) and spatial reasoning (bottom row) skills.
A cat is sitting on a basket under a bench (Spatial)
4.2 Evaluation Results
Quantitative Results As shown in Table 2, among the variants of LayoutGPT (#6-#9), GPT-3.5 achieves the best performance in numerical reasoning while GPT-4 performs the best in generating correct spatial positions. LayoutGPT outperforms LayoutTransformer (#5) by large margins, proving the strong cross-modal reasoning skills of LLMs. As for image-level evaluation, LayoutGPT surpasses end-to-end T2I models (#1-#3) by  $20 - 40\%$  in GLIP-based accuracy and relatively  $1 - 6\%$  in CLIP similarity. Therefore, using layouts as an intermediate representation indeed leads to more reliable and faithful generation outcomes. In addition, LayoutGPT achieves similar layout accuracy as human users (numerical #6 vs. #11 (86.33% v.s. 92.56%); spatial #9 vs. #11 (91.73% v.s. 91.17%)), which implies its potential to spare users from drawing layouts manually. The discrepancy between layout accuracy and GLIP-based accuracy suggests that the bottleneck mainly stems from layout-guided image generation and GLIP grounding results.
In addition, LayoutGPT binds attributes to each object's bounding box with  $100\%$  accuracy on HRS [2] color prompts. We further evaluate the attribute correctness rate (accuracy) on the final generated images when combining LayoutGPT with GLIGEN/ReCo. As shown in Table 3, our system largely improves the color correctness over Stable Diffusion with multiple objects.
Qualitative results We show the qualitative results of LayoutGPT and baselines in Fig. 3. LayoutGPT can understand visual commonsense such as the clock sizes at a train station (top left) or complex spatial relations between multiple objects (bottom right), while SD fails to generate correct numbers or positions. Besides, LayoutGPT demonstrates a similar layout design to human users (bottom left). Fig. 11 in the Appendix visualizes the results of attribute binding using LayoutGPT and ReCo [56].
Table 3: Color binding accuracy evaluated on prompts from HRS-Bench [2]. We follow the benchmark and use a hue-based classifier to identify the color of generated objects.
Models	Attribute binding Accuracy (%)
Prompts w/ 2 objects	Prompts w/ 3 objects	Prompts w/ 4 objects	Overall
SD1.4	18.57	10.10	11.36	12.84
Attend-and-Excite	31.43	19.19	20.45	22.96
LayoutGPT + GLIGEN	22.86	19.19	14.77	18.68
LayoutGPT + ReCo [56]	40.00	37.37	34.09	36.96
Dense layout Planning	LayoutGPT	+GLIGEN	LayoutGPT	+GLIGEN	LayoutGPT
A bathroom featuring a sink and toilet	Different kinds of doughnuts in a display case	A big brown cow walking down a sidewalk near home
Text-based Inpainting	Grey and black cat sitting under a bench. a grey and black cat with its tail tucked in and its head down, looking at the viewer a wooden bench with two legs	A cat is sitting on a basket under a bench. a black and white cat with its tail curled up in the basket a light brown basket with a handle on the side a wooden bench with a curved backrest	A cat is lying on top of a suitcase. a black and white cat lying on its side with its eyes closed a purple suitcase with a design of a blue, yellow, and white flower
Figure 4: Dense layout planning: LayoutGPT can generate rich objects or categories in complex scenes for MSCOCO 2017 Panoptic prompts [29]. Text-based inpainting: LayoutGPT can generate free-form regional descriptions that are not mentioned in the global prompt.
4.3 Application Scenarios
By utilizing LLMs as layout generators, LayoutGPT can be applied to a diverse set of scenarios for accurate and creative image generation.