SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dense Layout Planning: In Fig. 4 (top), we apply random in-context examples from COCO17 panoptic annotations with  $6 \sim 15$  bounding boxes per image. LayoutGPT can be applied to scenarios that imply numerous objects (e.g. different kinds of donuts) or various categories (e.g. bathroom or street view). Though only a few objects are mentioned in the prompts, LayoutGPT predicts layouts for the whole scene and imagines common objects that are usually visible in each scene.
Text-based Inpainting: In addition, the inherent language generation ability of LLMs enables our method to generate fine-grained regional descriptions from coarse global prompts (Fig. 4 bottom). LayoutGPT can enrich the description of each object with details that are not mentioned in the prompt, producing suitable outputs for models like ReCo [56].
Counterfactual Scenarios: We test LayoutGPT on counterfactual prompts provided by GPT-4 [35]. The in-context examples are randomly drawn from MSCOCO 2017[29], which greatly differs from the counterfactual prompts. As shown in Fig. 5, LayoutGPT manages to generate reasonable layouts on these challenging prompts and handles the relationship between objects well.
Component Analysis Table 4 presents the component analysis of our CSS-style prompt on spatial reasoning prompts. Comparisons between line 1-3 entails that the task instructions (#2) and CSS format (#3) effectively improve layout accuracy. Format in-context exemplars in CSS structures
Table 4: Ablation study of LayoutGPT (GPT-3.5) on spatial reasoning prompts. "w/ Instr": with prepended task instructions. "w/ CSS": format in-context demonstrations in CSS style. "w/ Norm": normalizing attribute values to integers by a fixed size.
w/Instr.	w/CSS	w/Norm.	Layout-to-Image Model	Layout Eval	Image Eval
Acc.	Acc. (GLIP)	CLIP Sim
1	55.12	34.35	0.259
2	✓	78.23	47.92	0.263
3	✓	80.82	51.38	0.264
4	✓	44.10	26.43	0.257
5	✓	✓	GLIGEN [27]	81.84	52.08	0.264
6	✓	✓	73.36	44.88	0.262
7	✓	✓	76.61	47.56	0.263
8	✓	✓	✓	82.54	52.86	0.264
9	✓	✓	✓	Layout-Guidance [6]	82.54	31.02	0.258
10	GT layouts	100.00	33.92	0.257
Figure 5: Qualitative examples of LayoutGPT's performance on counterfactual prompts.
show a more significant effect on accuracy. Pairwise comparisons of line 5-7 support the argument that the CSS style is the most essential component. While solely applying normalization degrades accuracy in line 4, line 5&8 shows that it slightly improves the performance when combined with other components.
Model-Agnostic Property We show that LayoutGPT is agnostic to layout-guided image generation models in line 9-10 in Table 4. We feed the same generated layouts from LayoutGPT to Layout-Guidance [6] and compute image-level metrics. Compared to using ground truth layouts (#10), LayoutGPT (#9) shows a minor gap in GLIP-based accuracy and a comparable CLIP similarity score. The discrepancy in GLIP-based accuracy is similar to that in Table 2, implying that the layouts generated by our method are agnostic to the downstream model.
5 LayoutGPT for Indoor Scene Synthesis
5.1 Task Setup
Datasets & Benchmarks For indoor scene synthesis, we use an updated version of the 3D-FRONT dataset [12, 13] following ATISS [38]. After applying the same pre-processing operations, we end up with 4273 bedroom scenes and 841 scenes for the living room. We only use rectangular floor plans of the test set for evaluation since LayoutGPT is not compatible with irregular ones. Hence, we end up with 3397/453/423 for train/val/test split of bedroom scenes and 690/98/53 for train/val/test split of living room scenes.
Evaluation Metrics We follow prior work [38] to report KL divergence between the furniture category distributions of predicted and ground truth scenes. We also render scene images from four camera angles for each scene and report FID scores [19]. In addition, we report out-of-bound rates, i.e. the percentage of scenes with furniture exceeding the floor plan boundary.
Quantitative Results The evaluation results are recorded in Table 5. We provide a random baseline for comparison denoted as "Random Scenes", in which the scene is randomly sampled from the in-context exemplars for each inference run.<sup>3</sup>
3Notice that while the scenes in "Random Scenes" are sampled from the training set, the out-of-boundary rate is larger than 0 since the 3D-FRONT dataset contains a small portion of scenes with out-of-bound furniture.
Table 5: Comparison of LayoutGPT with ATISS on indoor scene synthesis. "Random Scenes" means randomly sampling one training scene from the in-context demonstrations for each inference room sample. (* denotes results reproduced by us)
Models	Bedrooms	Living Rooms
Out of bounds (↓)	KL Div. (↓)	FID (↓)	Out of bounds (↓)	KL Div. (↓)	FID (↓)
Random Scenes	11.16	0.0142	23.76	9.43	0.1239	79.61
ATISS*[17]	49.88	0.0113	30.02	83.02	0.1054	85.40
LayoutGPT (GPT-3.5)	43.26	0.0995	28.37	73.58	0.1405	76.34
LayoutGPT (GPT-3.5, chat)	57.21	0.0846	29.66	81.13	0.2077	89.40
LayoutGPT (GPT-4)	51.06	0.1417	29.88	64.15	0.1613	78.60
Figure 6: Visualization of LayoutGPT across different types of rooms with different floor plan sizes.
For both bedrooms and living rooms planning, LayoutGPT attains lower out-of-bound rates than ATISS (bedrooms:  $43.26\%$  vs.  $49.88\%$ ; living rooms:  $64.16\%$  vs.  $83.02\%$ ), which verifies Layout-GPT's spatial reasoning ability in 3D environments. In addition, LayoutGPT has lower FID compared to ATISS (bedrooms: 28.37 vs. 30.02; living rooms: 76.34 vs. 85.40), which indicates that the planned scene has higher quality. Noted here that the living room split contains much more objects on average (11 for living rooms vs. 5 in bedrooms) and is a low-resource split with only 690 training scenes. Therefore, while living rooms are challenging for both methods, LayoutGPT shows more significant improvement over ATISS as supervised methods tend to overfit in early epochs.
Meanwhile, ATISS performs better in terms of KL divergence, which means that the overall furniture distribution predicted by ATISS is closer to the test split. We observe that LayoutGPT tends to avoid furnitures that are extremely rarely seen in each scene (e.g. coffee tables for bedrooms) as these objects appear less frequently in the in-context demonstrations. The limited in-context demonstration size also restricts LayoutGPT to have a universal observation of the furniture distributions.
Qualitative Results As shown in Fig. 6, LayoutGPT manages to understand common 3D concepts, such as "the pendant lamp should be suspended from the ceiling" and "nightstands should be placed by the headboard of the bed" (bottom row). When given a floor plan size for both living and dining rooms, LayoutGPT can also generate complicated 3D planning with dining tables and chairs on one side, and a sofa, a coffee table, and a TV stand on the other side (bottom right).
5.3 Application Scenarios
Text-guided Synthesis: LayoutGPT can follow text captions to arrange furniture in the scene (see Fig. 7). When the captions enumerate a complete list of furniture, LayoutGPT strictly follows the captions to generate the furniture and achieve a KL Div. value close to zero.
Partial Scene Completion: Thanks to the autoregressive decoding mechanism, LayoutGPT can complete a scene with partial arrangements such that the additional furniture remains coherent with the existing ones. Through in-context demonstrations, LayoutGPT learns critical (visual) commonsense such as visual symmetric (e.g. nightstands in Fig. 8(a)), positional relations (e.g. stool at the end of the bed in Fig. 8(b)), and room functions (e.g. desks and chairs in the dining area in Fig. 8(d)).
"A bedroom with a double bed, two wardrobes and a pendant lamp." Room Size: max length 262px, max width 207px
"A bedroom with a double bed, a wardrobe, two tables, a cabinet and a pendant lamp." Room Size: max length 214px, max width 306px
"A living room with three coffee tables, a tv stand, a multi seat sofa, a dining table, four dining chairs, a console table and two pendant lamps." Room Size: max length 256px, max width 490px
Figure 7: Generation of 3D scenes based on text captions that enumerate the furniture.
"A living room with a multi seat sofa, a coffee table, a tv stand, a dining table, four dining chairs, a ceiling lamp and a pendant lamp." Room Size: max length 413px, max width 256px
Partial
Bedroom,  $3.8\mathrm{m}\times 3.3\mathrm{m}$
(a)
Figure 8: LayoutGPT can successfully complete a partial scene for different rooms. We provide three starting objects for bedrooms and seven objects for living rooms.
Completed
Partial
Completed
Bedroom,  $3.5\mathrm{m}\times 4.2\mathrm{m}$
(b)
Partial
Completed
Living & Dining room,  $5.1\mathrm{m}\times 3.3\mathrm{m}$
(c)
Partial
Completed
Living & Dining room,  $5.4\mathrm{m} \times 3.9\mathrm{m}$
(d)
Similar to Sec. 4.4, we study the effect of task instructions, CSS structure, and normalization on indoor scene synthesis (see Table 6). In contrast to our conclusion for 2D planning in Sec. 4.4, comparisons between line 1-4 show that normalization (#4) is the most critical component for suppressing the out-of-bound rate while the CSS structure is also effective. We observe that LLMs occasionally copy attribute values directly from in-context exemplars even though the room sizes are different. Therefore,
normalizing all exemplars to the same scale can reduce the out-of-bound rate. CSS style facilitates LLMs to understand the physical meaning behind each attribute value and hence leads to almost the best result when combined with normalization (#7).
Table 6: Ablation studies on LayoutGPT on the bedroom split for 3D indoor scene synthesis.
w/Instr.	w/CSS	w/Norm.	Out ofBound ↓	KL Div. ↓	FID ↓
1	55.32	0.1070	56.83
2	✓	54.85	0.1153	58.85
3	✓	51.77	0.0776	55.62
4	✓	46.57	0.1276	58.24
5	✓	✓	51.30	0.0741	57.64
6	✓	✓	46.81	0.0913	58.61
7	✓	✓	43.74	0.0848	57.70
8	✓	✓	✓	43.26	0.0995	56.66
In this work, we address a new direction of generative model collaborations. Specifically, we are interested in how Large Language Models (LLMs) can collaborate with visual generative models. To this end, we propose LayoutGPT, an approach that turns an LLM into a visual planner through in-context learning and CSS style prompts. LayoutGPT can generate plausible visual arrangements in both image space and 3D indoor scenes. LayoutGPT can effectively improve image compositions by generating accurate layouts and achieves comparable performance in indoor scene synthesis compared to supervised methods. Besides, LayoutGPT can improve user efficiency in image generation and serve as an essential part of a unified system for all types of multimodal tasks.
The work was funded by an unrestricted gift from Google and a gift from the Robert N. Noyce Trust to the University of California via the Noyce Initiative. We would like to thank Google and the Robert N. Noyce Trust for their generous sponsorship. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the sponsors' official policy, expressed or inferred.
In this section, we provide a detailed description of our prompt construction and instantiate instructions examples.
Task instructions As is shown in Table 7, the specific task instructions start with verbalized descriptions of the task and are followed by the formal definition of the CSS style. As for the indoor scene synthesis, we additionally provide a list of available furniture and the normalized frequency distribution for fair comparisons with the supervised method. Yet we discover that the provided frequency distribution has little effect on the generation results, based on the trivial change in the KL divergence. In some cases, it is important to make LLMs sample from a defined distribution instead of learning the distribution from in-context exemplars, which we leave for future work.
Table 7: The prepending instructions provided to GPT-3.5/4 during our LayoutGPT's 2D and 3D layout planning process. The instructions listed here are for the setting with CSS structure and with normalization.
Task	Instruction for GPT-3.5/4
2D Layout Planning	Instruction: Given a sentence prompt that will be used to generate an image, plan the layout of the image. The generated layout should follow the CSS style, where each line starts with the object description and is followed by its absolute position. Formally, each line should be like "object {width: ?px; height: ?px; left: ?px; top: ?px;}" . The image is 64px wide and 64px high. Therefore, all properties of the positions should not exceed 64px, including the addition of left and width and the addition of top and height.
3D Layout Planning	Instruction: Synthesize the 3D layout of an indoor scene from the bottom-up view. The generated 3D layout should follow the CSS style, where each line starts with the furniture category and is followed by the 3D size, orientation, and absolute position. Formally, each line should follow the template: FURNITURE {length: ?px: width: ?px; height: ?px; left: ?px; top: ?px; depth: ?px; orientation: ?degrees;} All values are in pixels but the orientation angle is in degrees. Available furniture: armchair, bookshelf, cabinet, ceiling_lamp, chair, children_cabinet, cof-fee_table, desk, doubleBED, dressing_chair, dressing_table, floor_lamp, kidsBED, nightstand, pendant_lamp, shelf, singleBED, sofa, stool, table, tv_stand, wardrobe Overall furniture frequencies: (armchair: 0.0045; bookshelf: 0.0076; cabinet: 0.0221; ceiling_lamp: 0.062; chair: 0.024; children_cabinet: 0.0075; coffee_table: 0.0013; desk: 0.0172; doubleBED: 0.1682; dressing_chair: 0.0063; dressing_table: 0.0213; floor_lamp: 0.0093; kidsBED: 0.0079; nightstand: 0.2648; pendant_lamp: 0.1258; shelf: 0.0086; singleBED: 0.0211; sofa: 0.0018; stool: 0.012; table: 0.0201; tv_stand: 0.0308; wardrobe: 0.1557)
Base LLMs We use four variants of GPT models, (1) Codex [5] (code-davinci-002), an LLM that is fine-tuned with large-scale code datasets and can translate natural language into functioning code snippets; (2) GPT-3.5 [36] (text-davinci-003), which is trained to generate text or code from human instructions; (3) GPT-3.5-chat (gpt-3.5-turbo) and (4) GPT-4 [35] (gpt-4), which are both optimized for conversational tasks. For the last two models, we first feed the in-context exemplars as multiple turns of dialogues between the user and the model to fit into the API design. However, we generally observe that GPT-3.5-chat and GPT-4 are not as strong as GPT-3.5 in learning from the in-context demonstrations, especially when the dialogue format follows a certain structure instead of free-form descriptions.
Hyperparameters For all LLMs, we fix the sampling temperature to 0.7 and apply no penalty to the next token prediction. For image layouts evaluation in Table 2, we fix the number of exemplars to 16 for numerical reasoning, and 8 for spatial reasoning, based on the best results of a preliminary experiment. However, we do not observe significant gaps in evaluation results when using different amounts of exemplars (see Sec. B.4). For each prompt, we generate five different layouts/images using baselines or LayoutGPT and thus result in 3810 images for numerical reasoning and 1415 images for spatial reasoning in all reported evaluation results. As for indoor scene synthesis, we fix the number of exemplars to 8 for bedrooms and 4 for living rooms to reach the maximum allowed input tokens. We set the maximum output token as 512 for bedrooms and 1024 for living rooms as bedrooms have  $\sim 5$  objects per room while living rooms have  $\sim 11$  objects per room. We generate one layout for each rectangular floor plan for evaluation.
B LayoutGPT for 2D Layout Planning
B.1 NSR-1K Benchmark Construction
We rely on the MSCOCO annotations to create NSR-1K with ground-truth layout annotations. Note that each image in COCO is paired with a set of captions and a set of bounding box annotations.
Numerical Reasoning We primarily focus on the competence of T2I models to count accurately, i.e., generate the correct number of objects as indicated in the input text prompt. The prompts for this evaluation encompass object counts ranging from 1 to 5. To design the template-based T2I prompts, we initially sample possible object combinations within an image based on the bounding box annotations. We only use the bounding box annotation of an image when there are at most two types of objects within the image. As a result, the template-based prompts consist of three distinct types: (1) Single Category, wherein the prompt references only one category of objects in varying numbers; (2) Two Categories, wherein the prompt references two categories of distinct objects in varying numbers; and (3) Comparison, wherein the prompt references two categories of distinct objects but specifies the number of only one type of object, while the number of the other type is indicated indirectly through comparison terms including "fewer than", "equal number of", and "more than". As for natural prompts, we select COCO captions containing one of the numerical keywords from "one" to "five" and filter out those with bounding box categories that are not mentioned to avoid hallucination.
Spatial Reasoning We challenge LLMs with prompts that describe the positional relations of two or more objects. Our spatial reasoning prompts consist of template-based prompts and natural prompts from COCO. To construct template-based prompts, we first extract images with only two ground-truth bounding boxes that belong to two different categories. Following the definitions from PaintSkill [7], we ensure the spatial relation of the two boxes belong to (left, right, above, below). Specifically, given two objects  $A$ ,  $B$ , their bounding box centers  $(x_A, y_A)$ ,  $(x_B, y_B)$  and the Euclidean distance  $d$  between two centers, we define their spatial relation  $\operatorname{Rel}(A, B)$  as:
$$
\operatorname {R e l} (A, B) = \left\{ \begin{array}{l l} B \text {a b o v e} A & \text {i f} \frac {y _ {B} - y _ {A}}{d} \geqslant \sin (\pi / 4) \\ B \text {b e l o w} A & \text {i f} \frac {y _ {B} - y _ {A}}{d} \leqslant \sin (- \pi / 4) \\ B \text {o n t h e l e f t o f} A & \text {i f} \frac {x _ {B} - x _ {A}}{d} <   \cos (3 \pi / 4) \\ B \text {o n t h e r i g h t o f} A & \text {i f} \frac {x _ {B} - x _ {A}}{d} > \cos (\pi / 4) \end{array} \right. \tag {1}
$$
The definition basically dissects a circle centered at  $A$  equally into four sectors that each represent a spatial relation. While the definition may not stand for all camera viewpoints, it allows us to mainly focus on the front view of the scene. Then, we utilize the category labels and the pre-defined relations to form a prompt, as is shown in Table 1. As for the natural COCO prompts, we select prompts that contain one of the key phrases (the left/right of, on top of, under/below) and ensure that the bounding box annotations align with our definition.
We denote the set of  $n$  object categories in the ground truth annotation as  $\mathcal{C}_{GT} = c_1, c_2, \ldots, c_n$ , where  $x_{c_1}, x_{c_1}, \ldots, x_{c_n}$  represent the number of objects for each category. Additionally, we denote the set of  $m$  object categories mentioned in GPT-3.5/4's layout prediction as  $\mathcal{C}_{pred} = c_1', c_2', \ldots, c_m'$ , where  $x_{c_1}', x_{c_2}', \ldots, x_{c_m}'$  represent the number of objects for each category accordingly. If a category  $c_i$  is not mentioned in  $\mathcal{C}_{pred}$ , then  $x_{c_i}'$  is assigned a value of 0, and vice versa.
Categories	ci	cat	bed	pillow
Ground Truth	xcj	2	1	2
Prediction	xcj'	1	0	3
$$
\begin{array}{l} \text {precion} = \frac {\sum \min \left(x _ {c _ {k}} , x _ {c _ {k}} ^ {\prime}\right)}{\sum x _ {c _ {k}} ^ {\prime}} = \frac {1 + 0 + 2}{1 + 0 + 3} = 75 \% \\ \text{recall} = \frac{\sum\min(x_{c_i},x_{c_i}^{\prime})}{\sum x_{c_i}^{\prime}} = \frac{1 + 0 + 2}{2 + 1 + 2} = 60\% \\ \end{array}
$$
Figure 9: An closeup example of how we compute the layout automatic evaluation metrics for numerical reasoning.
Table 8: Closeup of various in-context example formats with ablated CSS structure and normalization for 2D layout planning.
CSS Structure	Normalization	In-context Example Format Demo
Prompt: a teddy bear to the right of a book
Layout:
teddy bear: 0.50, 0.71, 0.50, 0.15
book: 0.50, 0.61, 0.00, 0.26
✓	Prompt: a teddy bear to the right of a book
Layout:
teddy bear {width: 0.50; height: 0.71; left: 0.50; top: 0.15; }
book {width: 0.50; height: 0.61; left: 0.00; top: 0.26;}
✓	Prompt: a teddy bear to the right of a book
Layout:
teddy bear: 32, 45, 31, 9
book: 31, 38, 0, 16
✓	✓	Prompt: a teddy bear to the right of a book
Layout:
teddy bear {width: 32px; height: 45px; left: 31px; top: 9px; }
book {width: 31px; height: 38px; left: 0px; top: 16px;}
The numerical reasoning ability of GPT-3.5/4 on layout planning is assessed using the following metrics: (1) precision: calculated as  $\frac{\sum_{k=1}^{n}\min(x_{c_k},x_{c_k}')}{\sum_{k=1}^{m}x_{c_k}'}$ , is an indication of the percentage of predicted objects that exist in the groundtruth; (2) recall: calculated as  $\frac{\sum_{k=1}^{n}\min(x_{c_k},x_{c_k}')}{\sum_{k=1}^{n}x_{c_k}}$ , indicates the percentage of ground-truth objects that are covered in the prediction; (3) accuracy: In the "comparison" subtask, an accuracy score of 1 is achieved when the predicted relation, whether it is an inequality or equality, between the two objects is accurately determined. For all other numerical subtasks, accuracy equals to 1 if the predicted categories and object numbers precisely match the ground truth. In other cases, the accuracy is 0. Fig. 9 shows an example of how we compute the precision and recall. The accuracy for this single example is 0 since the predicted object distribution does not match the ground truth in every category.
For spatial reasoning, we evaluate spatial accuracy based on the LLM-generated layouts and GLIP-based layouts. We adopt [26] finetuned on COCO to detect involved objects from the generated images and obtain the bounding boxes. For both types of layouts, we categorize the spatial relation based on the above definition and compute the percentage of predicted layouts with the correct spatial relation. For all evaluation benchmarks, we measure the CLIP similarity, which is the cosine similarity between the generated image feature and the corresponding prompt feature.
B.3 GPT-3.5/4 Prompting
In Sec. 4.4, we investigate the impact of three components in the structured prompts: (1) Instruction, which examines whether detailed instructions explaining the task setup and the format of the supporting examples are included in the prompt. (2) Structure, which evaluates the impact of different formatting settings on the presentation of the bounding box aspects of height, width, top, and left. The "w/ CSS" setting formats the aspects in CSS, while the "w/o CSS" setting presents the four aspects in a sequence separated by a comma. (3) Normalization, which investigates the effects of rescaling the bounding box aspects to a specified canvas size and presenting them as integers in pixels in the "w/ Norm." setting, while the "w/o Norm." setting presents the aspects as relative scales to the canvas size in floats that range from (0, 1).
Table 7 shows the detailed preponding instructions LayoutGPT provided to GPT-3.5/4 models during 2D layout planning. Table 8 compares the formats of supporting examples with ablated structures and normalization settings.
For experiments in Sec. 4.3, to adapt to the nature of the Panoptic task, we add the following additional instruction when prompting LayoutGPT: "The objects layout might be dense, and objects may overlap with each other. Some objects might not have been mentioned in the prompt, but are very likely to appear in the described scenario." To generate counterfactual prompts for text-to-image generation, the following text prompt is provided to GPT-4: "Please provide a few counterfactual prompts that