SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 9: The automatic metric scores of LayoutGPT (GPT-3.5) with different in-context sample selection approaches. All values are in percentage (\%).
#	Exemplar Selection	# In-Context Exemplars	Numerical Reasoning	Spatial Reasoning
Precision↑	Recall↑	Layout Accuracy↑	GLIP Accuracy↑	Layout Accuracy↑	GLIP Accuracy↑
1	Fixed Random	16	64.83	92.71	87.66	47.10	80.14	47.07
2	4	88.93	95.02	76.17	50.20	85.30	51.66
3	Retrieval	8	93.32	95.63	82.68	50.58	82.54	52.86
4	16	94.81	96.49	86.33	51.25	82.40	51.09
In-context exemplar
LayoutGPT
Figure 10: Comparison between the most similar in-context exemplar and the generation results of LayoutGPT.
depict rarely seen the spatial relationship between the 80 MSCOCO object categories. An example would be "a monkey riding on top of a bird".
B.4 Additional Experiments
Random In-Context Exemplars Empirically, selecting in-context exemplars can be critical for the overall performance of LLMs. Apart from our retrieval-augmented method in Sec. 3, we also experiment with a fixed random set of in-context exemplars. Specifically, we randomly sample  $k$  examples from the training (support) set  $D$  to form a fixed set of in-context demonstrations for all test conditions  $\mathcal{C}_j$ . Therefore, the fixed random setting results in in-context exemplars that are unrelated to the test condition  $\mathcal{C}_j$ . The minor gap between lines 1&5 in Table 9 verifies that LayoutGPT is not directly copying from the in-context exemplars in most cases. Fig. 10 further justifies the argument with layout visualization of the most similar in-context exemplars and the LayoutGPT outputs.
Number of In-Context Exemplars We take a closer look at the effects of the number of in-context exemplars in the prompt as shown in Table 9. For counting, we observe that the number of exemplars is positively correlated with the counting accuracy. We conjecture that LLMs learn to make more accurate predictions for challenging prompts (e.g., comparison) by learning from more few-shot exemplars. As the layout accuracy also accounts for results where CSS parsing fails, we observe that the LLMs generate more consistent CSS-style code by learning from more examples. However, we cannot observe a similar trend in spatial reasoning prompts. We conjecture that LLMs only require as few as four demonstrations to learn the differences between the four types of spatial relations. The small optimal number of in-context exemplars implies that LLMs already have 2D spatial knowledge and can map textual descriptions to corresponding coordinate values. Yet it is important to find a proper representation to elicit such knowledge from LLMs as implied in Sec. 4.4.
Input (to LayoutGPT): A red bird fluttering next to a car window.
Output (of LayoutGPT):
a red bird with wings fluttering a car window with the window
Figure 11: Attribute binding examples of LayoutGPT and generated images using ReCo.
Input: A brown horse standing next to a white truck.
a brown horse standing still
a white truck with four black wheels
Input: A gray and brown cat sitting under a purple umbrella.
a gray and brown cat in a sitting position a purple umbrella
Table 10: The layout performance on each numerical reasoning subtask. Results reported on LayoutGPT (GPT-4).
Prompt Source	Subtask	Precision	Recall	Accuracy
Template	Single Category	85.96	85.96	85.96
Two Categories	85.14	85.04	66.60
Comparison	-	-	77.80
Natural Prompts from MSCOCO	72.08	87.1	82.79
-	Total	78.36	86.29	78.43
Performance on Numerical Subtasks Table 10 presents the performance of layout generation in various numerical reasoning subtasks. Regarding template-based prompts, the LayoutGPT demonstrates superior performance in the "Single Category" numerical reasoning task, exhibiting precision, recall, and accuracy values around  $86\%$ . However, when it comes to the "Two Category" numerical reasoning task, while precision and recall experience minimal changes, the accuracy drops to  $66\%$ . For the "Comparison" subtask, the accuracy hovers around  $78\%$ . These outcomes indicate that LayoutGPT encounters greater challenges when confronted with multi-class planning scenarios, whether the number of objects is explicitly provided or indirectly implied through comparative clauses.
For natural prompts extracted from MSCOCO, a noteworthy observation is the high recall accompanied by relatively lower precision. This discrepancy arises due to the ground truth bounding box annotations encompassing only 80 object classes, whereas the natural prompts may mention objects beyond the annotated classes. Consequently, our LayoutGPT may predict object layouts corresponding to classes not present in the ground truth, which, despite lowering precision, aligns with the desired behavior.
Performance on Size Comparison Reasoning Task We evaluate LayoutGPT's reasoning ability regarding object size. We use the standard HRS benchmark [2] which is designed for benchmarking compositional text-to-image models. HRS prompts for size reasoning contain comparison terms between randomly sampled common objects. The size relations described in HRS size prompts are often counterfactual and rarely seen (e.g., "a person which is smaller than a chair and larger than horse", "a car which is smaller than a banana and chair and bigger than airplane"). LayoutGPT achieves an accuracy of  $98.0\% / 93.1\% / 92.1\%$  when the prompt involves size comparison between 2/3/4 objects. Meanwhile, the best size reasoning performance of nine text-to-image models reported by the HRS benchmark is only  $31.1\% / 0.2\% / 0\%$ . The results further verify that LayoutGPT acquires decent reasoning ability on rare scenarios / counterfactual prompts.
B.5 Failure Cases
Fig. 12 shows typical failure cases in numerical and spatial relations. As previously discussed, we observe in Table 10 that numerical prompts that involves two type of objects ("Two Categories" and "Comparison") are more challenging to LayoutGPT and the image generation model. In these subtasks, LayoutGPT tends to predict much smaller bounding boxes to fit all objects within the limited image space. The small boxes further challenge GLIGEN to fit the object within the limited region, as shown in Fig. 12 (right).
"A light shines on five clocks showing times in different zones"
Numerical failure
"A vintage photo of a dog laying on a couch"
Spatial failure
"Four zebras grazing in enclosed area with shade trees"
Incompatible layouts
Figure 12: Typical failure cases of LayoutGPT and the generation results using GLIGEN.
Figure 13: Sorted scene differences between LayoutGPT generated scenes and the most similar in-context exemplars of 423 testing bedroom samples. We partition the distribution into three segments representing different behaviors of LayoutGPT. Duplication: The generated scene is a duplication of the exemplar. Modification: LayoutGPT slightly modifies one exemplar as the generated layout. Generation: LayoutGPT generates novel scenes that are highly different from the exemplars.
C LayoutGPT for 3D Scene Synthesis
Due to the limitation in datasets, the conditions are room type and room size instead of text descriptions. While ATISS [38] utilizes the floor plan image as the input condition, LLMs are not compatible with image inputs. Therefore, we convert the floor plan image into the specification of the room size. Therefore, the input conditions are similar to "Room Type: Bedroom, Room Size: max length 256px, max width 256px".
C.1 Exemplar Selection
Similar to Sec. B.4, we investigate the effect of using a random set of in-context exemplars for indoor scene synthesis. When we apply 8 random bedroom layouts from the training set as in-context exemplars, the out-of-bound rate increases from  $43.26\%$  in Table 5 to  $85.58\%$ . The significant differences suggest that LayoutGPT heavily relies on rooms with similar floor plans to maintain objects within the boundary. Yet we verify that the generated layouts from LayoutGPT are not duplicates of the in-context exemplars in most cases.
We first define a training scene layout as a set of objects  $S^t = \{\mathbf{o}_1^t,\dots ,\mathbf{o}_m^t\}$ , and a generated scene layout as  $S^{g} = \{\mathbf{o}_{1}^{g},\ldots ,\mathbf{o}_{n}^{g}\}$ . Note that  $\mathbf{o}_j$  consists of category  $\mathbf{c}_j$ , location  $\mathbf{t}_j\in \mathbb{R}^3$ , size  $\mathbf{s}_j\in \mathbb{R}^3$ , and orientation  $\mathbf{r}_j\in \mathbb{R}$ , i.e.  $\mathbf{o_j} = (\mathbf{c}_j,\mathbf{t}_j,\mathbf{s}_j,\mathbf{r}_j)$ . We define the scene difference  $D(\cdot |\cdot)$  between  $S^t$  and  $S^t$  as
$$
D \left(S ^ {t} \mid S ^ {g}\right) = \sum_ {i = 1} ^ {n} \min  _ {j, \mathbf {c} _ {j} ^ {t} = \mathbf {c} _ {i} ^ {g}} \left(\left\| \mathbf {t} _ {j} ^ {t} - \mathbf {t} _ {i} ^ {g} \right\| _ {1} + \left\| \mathbf {s} _ {j} ^ {t} - \mathbf {s} _ {i} ^ {g} \right\| _ {1}\right). \tag {2}
$$
We set  $\mathbf{t}_j^t, \mathbf{s}_j^t$  to  $\mathbf{0}$  if  $S^t$  does not have a single object that belongs to the same category as  $\mathbf{c}_i^g$ . For each testing sample of the bedroom, we compute the scene differences between the generated layout
Figure 14: Typical failure cases of LayoutGPT.
and all eight in-context exemplars and use the minimum value as the final scene difference. Note that all parameters used for computation are in "meters" instead of "pixels".
We plot the scene differences of all 423 testing samples in Fig. 13. We empirically discover that a scene difference below 1.0 means  $S^g$  is highly similar to  $S^t$ , which we conclude as duplication from in-context exemplars. A scene difference below 6.0 shows moderate differences in object sizes or locations between two scenes, representing a modification based on  $S^t$  to generate  $S^g$ . Finally, a scene difference larger than 6.0 represents new objects or significant differences in object sizes or locations between the exemplar and the generated layouts, i.e. true generation. Fig. 13 shows that 34/111/278 scenes belong to duplication/modification/generation. Among each category, 30/67/143 scenes have no out-of-bound furniture. Therefore, LayoutGPT is performing generation instead of duplicating in-context exemplars in most cases.
C.2 Failure Cases
While LayoutGPT achieves comparable results as ATISS, LayoutGPT cannot avoid typical failure cases as shown in Fig. 14, such as out-of-bound furniture and overlapped objects. Fig. 14 (right) shows an incorrect placement of nightstands on the same side of the bed while they are commonly placed on each side of the bed headboard. Future work could focus on more sophisticated in-context learning or fine-tuning methods to improve the LLMs' understanding of 3D concepts.
D LayoutGPT for 2D Keypoint Planning
In addition to its application in 2D and 3D layout planning, we investigate the feasibility of leveraging LayoutGPT for 2D keypoint planning to facilitate text-conditioned image generation. In this approach, we utilize LayoutGPT to predict keypoint distributions based on a given text prompt, and subsequently employ GLIGEN [27] for keypoint-to-image generation. The keypoint format used aligns with the specifications outlined in MSCOCO2017 [29], focusing on 17 keypoints that correspond to the human skeleton. Similar to our methodology for selecting supporting examples in the context of 2D layout planning (Section B), we retrieve the  $k$ -most similar examples from the training set of MSCOCO2017 and utilize these examples to provide keypoint distributions as input to GPT-3.5/4. Table 11 presents an illustrative example of the input format employed for keypoint planning with GPT-3.5.
Fig. 15 presents several illustrative examples that compare the images generated by conditioning on keypoints planned by our LayoutGPT with those generated by end-to-end models such as StableDiffusion-v2.1 [41] and Attend-and-Excite [4]. In this preliminary demonstration, we observe that LayoutGPT exhibits promising potential in offering inherent control over specific movements or actions through keypoint planning.
Nevertheless, it is worth noting that keypoints planning presents considerably greater challenges compared to bounding box layout planning, attributable to several evident factors. Firstly, keypoints planning necessitates the prediction of the positions of 17 nodes, which is significantly more complex than the 2D layout planning involving four aspects or the 3D layout planning encompassing seven aspects. Secondly, the distribution of keypoints encompasses a much larger array of spatial relations due to the numerous possible body movements. In contrast, previous 2D layout planning tasks only involve four types of spatial relations. These inherent complexities render keypoint planning heavily reliant on in-context demonstrations. However, the limited availability of annotations pertaining to
Table 11: The prompting input provided to GPT-3.5 for LayoutGPT keypoint planning.
Instruction:
Given a sentence prompt that will be used to generate an image, plan skeleton keypoints layout of the mentioned objects. The skeleton keypoints include the following 17 nodes: nose, left_eye, right_eye, left_ear, right_ear, left_shoulder, right_shoulder, left/elbow, right/elbow, left_wrist, right_wrist, left_hip, right_hip, left_knee, right_knee, left_ankle, right_ankle. The generated keypoints layout should follow the CSS style, where each line starts with the keypoint node name and is followed by its absolute position.
Formally, each line should be like "node_name {left: ?px; top: ?px;}" . Please follow this format strictly. Do not display in other variation of formats. Notice that some keypoint nodes may not be visible on the canvas. In such cases, simply put "node_name {left: 0px; top: 0px;}" for the invisible nodes. The image is 64px wide and 64px high. Therefore, all properties of the positions should not exceed 64px.
Prompt: a man on a surfboard in a river near a couple of trees and branches
keypoints: person#1: nose {left: 36px; top: 33px;} left_eye {left: 36px; top: 33px;} right_eye {left: 36px; top: 33px;} left_ear {left: 37px; top: 33px;} right_ear {left: 0px; top: 0px;} left_shoulder {left: 38px; top: 34px;} right_shoulder {left: 36px; top: 35px;} left/elbow {left: 35px; top: 34px;} right_elbow {left: 35px; top: 38px;} left_wrist {left: 33px; top: 32px;} right_wrist {left: 33px; top: 39px;} left_hip {left: 39px; top: 39px;} right_hip {left: 37px; top: 40px;} left_knee {left: 38px; top: 44px;} right_knee {left: 37px; top: 44px;} left_ankle {left: 39px; top: 49px;} right_ankle {left: 37px; top: 48px;}
[MORE SUPPORTING EXAMPLES]
Prompt: a man leaning on a surfboard in the water riding a wave
Keypoints:
body movements in the MSCOCO dataset further exacerbates the challenges associated with reliable keypoint planning. Therefore, we leave the exploration of this potential direction to future research endeavors.
E Ethical Statement
In addition to the layouts predicted by GPT-3.5/4, we also incorporate human-planned layouts as a natural baseline for comparative analysis. To facilitate this, we provide annotators with an interface featuring a blank square space where they can draw bounding boxes. Alongside the input text prompt, we also present the noun words or phrases from the prompt to human annotators, instructing them to draw a bounding box for each corresponding element. We intentionally refrain from imposing additional constraints, enabling annotators to freely exercise their imagination and create layouts based on their understanding of reasonable object arrangements. To compensate annotators for their efforts, we offer a payment rate of  $0.2 US dollars per Human Intelligence Task (HIT). The average completion time of approximately 30 seconds per HIT, which corresponds to an average hourly payment rate of$ 24.
The current work has several limitations that provide opportunities for future research. Firstly, while this work focuses on 2D and 3D bounding box layouts and makes a preliminary attempt at keypoints, there exist various other methods for providing additional spatial knowledge in image/scene
A chimpanzee holds a toothbrush in their hand
A person is standing in some water flying a kite
A close up of a monkey driving a motorcycle on a road
Figure 15: Plausible examples of LayoutGPT(GPT-4) planning keypoints distributions before conducting text-conditioned image generation.
A woman in glasses holding a laptop on a couch
generation, such as segmentation masks and depth maps. Future work could explore integrating LLMs with these alternative visual control mechanisms to broaden the scope of visual planning capabilities. Secondly, the current work primarily addresses visual generation tasks and lacks a unified framework for handling other visual tasks like classification or understanding. Extending the proposed framework to encompass a wider range of visual tasks would provide a more comprehensive and versatile solution. Thirdly, this work is a downstream application that attempts to distill knowledge from LLMs' extensive knowledge bases. Future research could explore more fundamental approaches that directly enhance the visual planning abilities of various visual generation models. By developing specialized models that are explicitly designed for visual planning, it may be possible to achieve more refined and dedicated visual generation outcomes. Overall, while the current work demonstrates the potential of using LLMs for visual planning, there are avenues for future research to address the aforementioned limitations and further advance the field of visual generation and planning.
The utilization of LLMs for conducting visual planning in compositional 2D or 3D generation has significant broader impacts. Firstly, LLMs alleviate the burden on human designers by simplifying the complex design process. This not only enhances productivity but also facilitates scalability, as LLMs can efficiently handle large-scale planning tasks. Secondly, LLMs exhibit remarkable capabilities in achieving fine-grained visual control. By conditioning on textual inputs, LLMs can easily generate precise and detailed instructions for the desired visual layout, allowing for precise composition and arrangement of elements. Moreover, LLMs bring a wealth of commonsense knowledge into the planning process. With access to vast amounts of information, LLMs can incorporate this knowledge to ensure more accurate and contextually coherent visual planning. This integration of commonsense knowledge enhances the fidelity of attribute annotations and contributes to more reliable and realistic visual generation outcomes.
It is worth noting that this work represents an initial foray into the realm of visual planning using LLMs, indicating the potential for further advancements and applications in this area. As research in this field progresses, we can anticipate the development of more sophisticated and specialized visual planning techniques, expanding the scope of LLMs' contribution to diverse domains, such as architecture, virtual reality, and computer-aided design.
H Additional Qualitative Examples
We present additional visual showcases to demonstrate the capabilities of LayoutGPT in different contexts. Fig. 16 showcases examples related to 2D numerical reasoning, Fig. 17 illustrates examples of 2D spatial reasoning, and Fig. 18 displays examples of 3D scene synthesis. These showcases offer further insights into the effectiveness and versatility of our approach across various domains.
A light shines on five clocks showing times in different zones.
SD2.1
Attend-n-Excite
+GLIGEN
GPT-3.5 chat
+GLIGEN
These three birds are walking along the beach looking for food
Attend-n-Excite
GPT-3.5, chat
A photo of a train station with two trains on the tracks.
GPT-3.5
Attend-n-Eclicate
There are three elephants standing beside a pool of water.
GPT-3.5
three airplanes that are landed near a large city
GPT-3.5
GPT-3.5 chat
I lone bicycle next to a bike path near the water.
SD2.1
Figure 16: Qualitative examples of variants of LayoutGPT on numerical reasoning prompts.
a dog laying on a bed near a laptop
GPT-3.5
A black cat laying in the sun under a bench
GPT-3.5
a cup to the right of a donut
GPT-3.5
Several glass beer bottles under a park bench.
GPT-3.5
a cow to the left of a car
+GLIGEN
a cup to the left of a dog
+GLIGEN
A banana sitting on top of a cup on a desk.
GPT-3.5
A cat sitting on a wood floor under a white and blue umbrella.
Attend-n-Excite
+GLIGEN
Figure 17: Qualitative examples of variants of LayoutGPT on spatial reasoning prompts.
GPT-3.5-chat
Figure 18: Additional qualitative examples of variants of LayoutGPT in bedroom scene synthesis.