SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Text-to-Layout. We leverage the CLIP [29] text encoder to encode input texts into embeddings. The constraint similarity  $s(x_{\mathrm{test}},x_j)$  is defined as the cosine similarity of input text embeddings  $n_{\mathrm{test}}$ ,  $n_j$ :
$$
s \left(x _ {\text {t e s t}}, x _ {j}\right) \triangleq \frac {n _ {\text {t e s t}} \cdot n _ {j}}{\| n _ {\text {t e s t}} \| \| n _ {j} \|}. \tag {6}
$$
3.4 Layout Ranker
People usually judge the quality of generated layouts from two perspectives: (1) whether they are visually pleasing; (2) whether they look like the real layouts. Therefore, our proposed layout ranker follows the same principles to evaluate layout quality. To be more specific, it measures the quality of an output layout using a combination of metrics:
$$
q \left(y _ {\text {t e s t}}\right) = \lambda_ {1} \text {A l i g n m e n t} \left(y _ {\text {t e s t}}\right) + \lambda_ {2} \text {O v e r l a p} \left(y _ {\text {t e s t}}\right) + \lambda_ {3} \left(1 - \mathrm {m I o U} \left(y _ {\text {t e s t}}\right)\right). \tag {7}
$$
Dataset	Domain	Associated Task	# Training Set	# Test Set	# Element Types
RICO	Android	constraint-explicit layout generation	31,694	3,729	25
PubLayNet	document	constraint-explicit layout generation	311,397	10,998	5
PosterLayout	poster	content-aware layout generation	9,974	905	3
WebUI	web	text-to-query	3,835	487	10
Table 3: Dataset statistics. Note that these datasets are only used on specific tasks.
Figure 4: Qualitative comparison between LayoutPrompter and the state-of-the-art baseline Layout-Former++ [15] on constraint-explicit layout generation tasks (better view in color and  $2 \times$  zoom).
Here,  $\lambda_1$ ,  $\lambda_2$  and  $\lambda_3$  are hyper-parameters to balance the importance of each metric. Alignment and Overlap reflect quality from the perspective (1), while mIoU mainly focuses on perspective (2). We will introduce them in Section 4.1. The output layout with the lowest  $q$  value (lower  $q$  indicates better quality) is returned as the final output.
Datasets. We conduct experiments on 4 datasets, including RICO [27], PubLayNet [40], Poster-Layout [9] and WebUI [24]. Their statistics and usages are illustrated in Table 3. For RICO and PubLayNet, we adopt the same dataset splits as LayoutFormer++ [15]. While for PosterLayout, the training set includes 9,974 poster-Layout pairs, and the remaining 905 posters are used for testing. Regarding the WebUI dataset, we adopt the dataset splits provided by parse-then-place [24]. In all cases, the in-context exemplars are retrieved from the full training set.
Baselines. Since constraint-explicit layout generation tasks have task-specific and task-generic methods, we compare LayoutPrompter against both kinds of state-of-the-art methods on these tasks. Concretely, we choose LayoutFormer++ [15] as the common task-generic baseline. The task-specific baselines are (1) BLT [19] for generation conditioned on types (Gen-T), (2) BLT [19] for generation conditioned on types with sizes (Gen-TS), (3) CLG-LO [18] for generation conditioned on relationships (Gen-R), (4) LayoutTransformer [7] for completion, and (5) RUITE [30] for refinement. Moreover, we compare LayoutPrompter with DS-GAN [9] and CGL-GAN [42] on content-aware layout generation. We compare with Mockup [11] and parse-then-place [24] on text-to layout.
Val ↑	Ove ↓	Ali ↓	Undi ↑	Unds ↑	Uti ↑	Occ ↓	Rea ↓
CGL-GAN	0.7066	0.0605	0.0062	0.8624	0.4043	0.2257	0.1546	0.1715
DS-GAN	0.8788	0.0220	0.0046	0.8315	0.4320	0.2541	0.2088	0.1874
LayoutPrompter (Ours)	0.9992	0.0036	0.0036	0.8986	0.8802	0.2597	0.0992	0.1723
Table 4: Quantitative comparison with baselines on content-aware layout generation task.
Figure 5: Qualitative results generated by DS-GAN and LayoutPrompter on content-aware layout generation. There are three element types, including logo (red), text (green) and underlay (yellow).
Evaluation Metrics. To evaluate the performance of LayoutPrompter and baselines, we use the following quantitative metrics. For constraint-explicit layout generation and text-to layout, we employ four standard metrics. Alignment (Align.) [22] gauges how well the elements in a layout are aligned with each other. Overlap [22] computes the overlapping area between two arbitrary elements in a layout. Maximum IoU (mIoU) [18] calculates the highest Intersection over Union (IoU) between a generated layout and real layouts. Fréchet Inception Distance (FID) [18] measures how similar the distribution of the generated layouts is to that of real layouts. Additionally, we introduce another metric Constraint Violation Rate (Vio. %) [15] to evaluate how well the generated layouts satisfy their input constraints. It is the ratio of violated constraints to all constraints. In text-to layout, as a textual description may involve the type, position and size constraints of elements, we follow parse-then-place [24] and further break down this metric into Type Vio. % and Pos & Size Vio.%. As for content-aware layout generation, we adopt the eight metrics defined in DS-GAN [9]. Some of them belong to graphic metrics, such as Val, Ove, Ali, Undl and Unds. Others are content-aware metrics, including Uti, Occ and Rea. Please refer to [9] for more details.
Implementation Details. In this work, we conduct experiments on GPT-3 [3] text-davinci-003 model. We place  $N = 10$  exemplars in the prompt  $\mathbf{P}$ . For each test sample, we generate  $L = 10$  different outputs  $y_{\mathrm{test}}$ . The hyper-parameters involved in the layout ranker module are set to  $\lambda_1 = 0.2, \lambda_2 = 0.2$ , and  $\lambda_3 = 0.6$ . When running GPT-3, we fix the parameters to the default values of the OpenAI API, where the sampling temperature is 0.7 and the penalty-related parameters are set to 0.
Tables 2, 4, 5 and Figures 4, 5, 6 show the quantitative and qualitative results on various layout generation tasks (see more qualitative results in Section C of the supplementary material). Although LayoutPrompt has not undergone model training and fine-tuning, the experimental results demonstrate that it can achieve comparable or even better performance than baselines, which proves that LayoutPrompt is a versatile and training-free layout generation approach. Below, we conduct a detailed analysis of the experimental results.
Constraint-Explicit Layout Generation. Table 2 shows the quantitative results. On each constraint-explicit layout generation task, LayoutPrompter is compared with a task-specific method and another common baseline, LayoutFormer++ [15]. Although not trained on these downstream tasks, LayoutPrompter still exhibits competitive quantitative results. Furthermore, it even outperforms the baselines on some metrics (e.g., Align. and Overlap on Gen-T task, RICO dataset). The corresponding qualitative results are shown in Figure 4. Here, we only compare with the state-of-the-art baseline (measured by quantitative metrics), LayoutFormer++. The qualitative comparison indicates
mIoU ↑	FID ↓	Align. ↓	Overlap ↓	Type Vio. % ↓	Pos & Size Vio. % ↓
Mockup	0.1927	37.0123	0.0059	0.4348	31.49	44.92
parse-then-place	0.6841	2.9592	0.0008	0.1380	11.36	19.14
LayoutPrompter (Ours)	0.3190	10.7706	0.0009	0.0892	15.09	23.78
Table 5: Quantitative comparison with baselines on text-to layout.
(a) A page for advertising holiday photo cards. The page should have an image showing the card, a title "Holiday Photo Cards", a further description, and a button "SHOP CARDS" for the user to click to purchase cards.
Figure 6: Qualitative results of Mockup, parse-then-place (short as PTP) and LayoutPrompter on text-to layout (better view in color and  $2 \times$  zoom).
(d) A footer page for an online store selling jewelry. On the page, there should have four lists of links for more info. In each list, there should be a title and 4-6 links.
(b) The page introducing the products to the user. There should be three parts, and each part includes one title, one picture of the product, and one link for the user to click on to view more about the product.
(e) A page to introduce payments to users. There is a title 'accepted payments' on the top. There are also four logos of payment companies under it.
(c) A page for navigation of Mailreach. The page should have a logo of Mailreach, 5 links for users to check the email spam test, blog, features, pricing, and to log in, and a button for users to click to sign up.
(f) A page for showing different work types. There should be six groups. Each of them includes one logo and one text which shows the title of the work.
that LayoutPrompter achieves as good controllability and generation quality as LayoutFormer++. First, the layouts generated by our approach satisfy various input constraints well, including type constraints, size constraints, relationship constraints, etc. Second, our approach can also produce visually pleasing layouts with well-aligned elements and small overlapping areas. Both qualitative and quantitative results demonstrate the effectiveness of LayoutPrompter.
Content-Aware Layout Generation. The quantitative and qualitative results are presented in Table 4 and Figure 5, respectively. Remarkably, LayoutPrompter surpasses the training-based baselines on almost all metrics. This indicates that LayoutPrompter is capable of producing higher-quality and more content-aware layouts compared to the baselines. The rendered results further validate the conclusion. For example, in columns (f) and (g) of Figure 5, the layouts from DS-GAN [9] contain serious misalignments and overlaps. And column (e) shows that DS-GAN sometimes fails to generate content-aware layouts. In contrast, our approach can not only produce aesthetic layouts but also avoid the most salient objects in the input canvas, such as the person, teddy bear, car, etc.
Text-to-Layout. The quantitative and qualitative comparisons are shown in Table 5 and Figure 6. Since text-to layout is one of the most challenging layout generation tasks, LayoutPrompter slightly lags behind the current state-of-the-art method parse-then-place [24], especially on mIoU and FID metrics. However, on the other four metrics, LayoutPrompter is comparable to the baselines. Thanks to the excellent understanding capability of LLMs, our approach can better satisfy the constraints specified in textual descriptions in some cases. For example, in cases (d) and (e) of Figure 6, LayoutPrompter successfully generates 4-6 links and four logos, while parse-then-place makes wrong predictions about the number of elements.
Effect of Introduced Components. LayoutPrompter has three key components, including input-output serialization, dynamic exemplar selection and layout ranking. To investigate their effects, we perform the following ablation studies (see Table 6). (1) Since LayoutFormer++ [15] has proven the effectiveness of constraint sequences relative to other formats, we only study the effect of HTML representation, which is not covered in previous works. Specifically, we replace HTML with a plain sequence proposed by LayoutFormer++ [15] (denoted as w/o HTML) to represent the output layout. This results in a significant drop in FID and overlap metrics on Gen-T. (2) To understand the contribution of dynamic exemplar selection, we compare against its variant (w/o dynamic selection) that adopts random sampling for exemplar retrieval. LayoutPrompter achieves significantly better FID and mIoU across the board. Though the variant has better Align. and Overlap scores in some tasks, its noticeably poor FID and mIoU scores indicate that it fails to acquire the layout patterns in specific domains (e.g., the generated layout does not look like a real UI layout). (3) To understand the
Tasks	Methods	RICO
mIoU ↑	FID ↓	Align. ↓	Overlap ↓	Vio. % ↓
Gen-T	LayoutPrompter	0.429	3.233	0.109	0.505	0.64
w/o HTML	0.460	7.009	0.106	0.663	0.
w/o dynamic selection	0.251	8.154	0.053	0.399	0.24
w/o layout ranker	0.367	3.149	0.142	0.498	0.45
Gen-TS	LayoutPrompter	0.552	1.458	0.145	0.544	0.18
w/o dynamic selection	0.337	8.107	0.199	0.400	0.24
w/o layout ranker	0.505	1.528	0.153	0.549	0.13
Gen-R	LayoutPrompter	0.400	5.178	0.101	0.564	10.58
w/o dynamic selection	0.223	14.177	0.067	0.597	15.95
w/o layout ranker	0.341	5.282	0.137	0.545	6.54
Completion	LayoutPrompter	0.667	7.318	0.084	0.428	-
w/o dynamic selection	0.449	17.409	0.062	0.422	-
w/o layout ranker	0.580	11.194	0.093	0.451	-
Refinement	LayoutPrompter	0.745	0.978	0.159	0.478	-
w/o dynamic selection	0.662	1.718	0.208	0.468	-
w/o layout ranker	0.705	1.161	0.188	0.478	-
Table 6: Ablation studies of the introduced components on RICO.
Tasks	# Training samples	LayoutFormer++	LayoutPromptter
mIoU ↑	FID ↓	Align. ↓	Overlap ↓	Vio. % ↓	mIoU ↑	FID ↓	Align. ↓	Overlap ↓	Vio. % ↓
Gen-T	500	0.176	92.643	0.272	0.668	69.27	0.343	7.201	0.105	0.539	0.11
2,000	0.209	48.702	0.165	0.573	62.22	0.362	6.140	0.083	0.527	0.22
10,000	0.368	3.370	0.132	0.572	11.02	0.389	4.658	0.097	0.527	0.11
Full Set	0.432	1.096	0.230	0.530	0.	0.429	3.233	0.109	0.505	0.64
Gen-TS	500	0.171	79.641	0.301	0.808	74.66	0.405	4.068	0.130	0.596	0.13
2,000	0.249	39.673	0.209	0.655	53.07	0.424	3.460	0.143	0.604	0.06
10,000	0.529	2.395	0.215	0.596	1.86	0.464	2.606	0.138	0.580	0.06
Full Set	0.620	0.757	0.202	0.542	0.	0.552	1.458	0.145	0.544	0.18
Table 7: Ablation studies of training set size on RICO.
effect of the proposed layout ranker, we compare it against a variant (w/o layout ranker) that randomly picks a layout from model outputs. We find that the layout ranker consistently yields improvements on the mIoU and Align. metrics of all tasks.
Effect of Training Set Size. We switch training set sizes: 500, 2000, 10000 and full set (see Table 7). In our approach, the training set represents the exemplar retrieval pool. The results show that the performance of LayoutFormer++ drops rapidly as the training data decreases, but our method is much slightly affected. When training samples are limited (e.g., 500 and 2000), our approach significantly outperforms the training-based baseline on all metrics. These observations suggest that LayoutPrompter is a more data-efficient approach, which is effective in low-resource scenarios. Due to space limitations, more experimental results on stability, the effect of the number of examples, and generalization ability can be found in Section B of the supplementary material.
In this work, we concentrate on leveraging Large Language Models (LLMs) for conditional layout generation to address issues present in existing methods. To enhance the performance of our approach, we introduce three crucial components: input-output serialization, dynamic exemplar selection, and layout ranking. We conduct experiments on 7 existing layout generation tasks using 4 public datasets. Both qualitative and quantitative results highlight that LayoutPrompter is a versatile, data-efficient, and training-free method capable of generating high-quality, constraint-compliant layouts. Despite these promising results, there are still some limitations. First, the performance of our approach is influenced by the number of elements in the layouts, with more elements leading to more failure cases. Notably, this is not a problem specific to our approach and has been observed in prior work [2] as well. Second, we have not studied whether LayoutPrompter is equally effective for other LLMs such as PaLM and LLaMa. Third, with the rapid development of large multimodal models such as GPT-4V, PaLI [5] and LLaVA [25], we get a promising chance to extend LayoutPrompter to supporting layout constraints specified in a wide range of modalities. We leave them for future research.
[33] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
[34] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
[35] K. Yang, D. Klein, N. Peng, and Y. Tian. Doc: Improving long story coherence with detailed outline control. arXiv preprint arXiv:2212.10077, 2022.
[36] K. Yang, N. Peng, Y. Tian, and D. Klein. Re3: Generating longer stories with recursive reprompting and revision. arXiv preprint arXiv:2210.06774, 2022.
[37] J. Zhang, J. Guo, S. Sun, J.-G. Lou, and D. Zhang. Layoutdiffusion: Improving graphic layout generation by discrete diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7226-7236, October 2023.
[38] Z. Zhang, A. Zhang, M. Li, H. Zhao, G. Karypis, and A. Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023.
[39] X. Zheng, X. Qiao, Y. Cao, and R. W. Lau. Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics (TOG), 38(4):1-15, 2019.
[40] X. Zhong, J. Tang, and A. J. Yepes. Publaynet: largest dataset ever for document layout analysis, 2019.
[41] D. Zhou, N. Scharli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, O. Bousquet, Q. Le, and E. Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022.
[42] M. Zhou, C. Xu, Y. Ma, T. Ge, Y. Jiang, and W. Xu. Composition-aware graphic layout gan for visual-textual presentation designs. arXiv preprint arXiv:2205.00303, 2022.
A Coordinate Discretization
In this work, element coordinates are scaled proportionally into a canvas of size  $C_W \times C_H$ . We follow the baselines to choose these two parameters. Specifically, in RICO,  $C_W = 90\mathrm{px}$ ,  $C_H = 160\mathrm{px}$ . In PubMedNet,  $C_W = 120\mathrm{px}$ ,  $C_H = 160\mathrm{px}$ . In PosterLayout,  $C_W = 102\mathrm{px}$ ,  $C_H = 150\mathrm{px}$ . In WebUI,  $C_W = 120\mathrm{px}$ ,  $C_H = 120\mathrm{px}$ . Then, the coordinates are discretized to the nearest integers.
B Additional Experimental Results and Analysis
B.1 Stability of Generation Performance
The output of LLMs varies with the random seed and hyper-parameters (e.g., the temperature). That is, for the same input constraint, LLMs are able to generate many completely different layouts. Since the hyper-parameters are a trade-off between generation quality and diversity, we fix them to the default values of OpenAI API and study the impact of random seeds on model performance. Specifically, we run inference on the test set 10 times, each using a different random seed. Then, we calculate the mean and variance of each quantitative metric (see Table 8). The small variances indicate the stability of LayoutPrompter's performance under different random seeds.
Tasks	RICO	PubLayNet
mIoU ↑	FID ↓	Align. ↓	Overlap ↓	Vio. % ↓	mIoU ↑	FID ↓	Align. ↓	Overlap ↓	Vio. % ↓
Gen-T	0.368±0.002	3.118±0.045	0.130±0.010	0.498±0.004	0.546±0.148	0.343±0.001	4.014±0.067	0.042±0.007	0.047±0.002	0.490±0.059
Gen-TS	0.504±0.001	1.489±0.037	0.155±0.003	0.550±0.005	0.134±0.025	0.393±0.001	2.016±0.024	0.050±0.008	0.098±0.002	0.
B.2 Effect of Exemplar Number
We conduct ablation experiments on the number of prompting exemplars. Figure 7 shows the zero-shot  $(N = 0)$  qualitative results on Gen-T. It is obvious that LLMs fail to generate reasonable layouts in a zero-shot scheme. Table 9 exhibits the quantitative comparison of the Gen-T task on RICO. The results indicate that the number of prompting exemplars mainly affects mIoU and FID. Specifically, as the number of prompting exemplars increases, mIoU and FID get improved. In summary, the number of exemplars has a positive effect on the performance of LayoutPrompter.
Figure 7: Zero-shot results on the Gen-T task.
Table 8: Effect of random seeds. In this experiment, we disable the layout ranker to eliminate the impact of the ranking mechanism on model performance.
Tasks	# exemplar	RICO
mIoU ↑	FID ↓	Align. ↓	Overlap ↓	Vio. % ↓
Gen-T	1	0.381	5.007	0.115	0.491	0.85
3	0.413	5.098	0.120	0.492	0.51
5	0.414	4.521	0.114	0.492	0.65
10	0.427	3.523	0.092	0.486	0.67
Table 9: Ablation studies on the number of prompting exemplars. We run experiments on 1,000 test samples.
B.3 Generalization Ability
To investigate the generalization ability of LayoutPrompter, we compute the DocSim similarity between the generated layouts and their prompting layouts (see Table 10). The DocSim of LayoutFormer++ is computed between the generated layouts and training layouts. The quantitative results show that LayoutPrompter achieves competitive or even better scores compared to LayoutFormer++.
indicating that LayoutPrompt has a close generalization ability to the training-based method. In addition, we exhibit the qualitative results of the generated layouts and their prompting layouts in Figure 8. The results demonstrate that LayoutPrompt is capable of generating meaningful variations different from the prompting ones.
Method	RICO	PubLayNet
LayoutFormer++	0.8472	0.8266
LayoutPrompter	0.8563	0.8119
Table 10: DocSim on Gen-T task. The smaller the DocSim, the better the generalization ability.
Figure 8: Qualitative results of the generated layouts and corresponding retrieved layouts on Gen-T task. Each case contains 2 generated layouts.
In this section, we present additional qualitative results of LayoutPrompter. These results further demonstrate the versatility and effectiveness of our method, which can generate high-quality and constraint-compliant layouts on multiple layout generation tasks.
C.1 Generation Conditioned on Element Types
Element Type Constraints
(1) icon, input, text*8, text button*3, toolbar
(2) background image, icon*5, image*3, slider, text*3, toolbar
(3) icon, list item*6, text*7, toolbar
(4) background image, image, text button*5
(5) background image, icon, image*2, text*3, text button*3
(6) advertisement, icon\*3, image\*6, text, toolbar, web view\*2
(3)
(b) PubLayNet
(a) RICO
(4)
(5)
Figure 9: Qualitative results of Gen-T on RICO and PubLayNet. The element type constraints are in the table.
(1) text*11, title*2
(4) figure, text*6, title*2
(2) table, text*5, title
(5) table, text*6, title*2
(3) table, text*5
(6) figure, text*6, title*2
C.2 Generation Conditioned on Element Types and Sizes
Element Type and Size Constraints
(1) advertisement 70 10, background image 90 144, image 21 7, text 90 25, text 90 8, text button 54 10, text button 54 10, text button 8 7, text button 33 7, text button 54 10, web view 70 10, web view 70 10
(2) icon 12 12, text 29 5, text button 85 10, text button 85 10, toolbar 90 12
(3) input 90 110, text 48 8, text 38 8, text button 28 9, text button 28 9, toolbar 90 16
(4) icon 7 7, image 69 39, text 90 5, text 90 6, text button 83 38
(5) input 53 11, on/off switch 85 11, radio button 17 11, radio button 19 11, text 15 13, text 74 5, text 26 4, text button 15 9, text button 15 8, text button 45 7, text button 45 7
(6) image 71 60, text 81 9, text 76 26, text button 72 16
(5)
Figure 10: Qualitative results of Gen-TS on RICO and PubLayNet. The element type and size constraints are in the table.
(1) figure 90 56, table 99 46, text 99 4, text 32 1, text 99 3, text 47 11, text 47 11
(2) figure 65 74, text 99 13, text 47 16, text 47 39, text 47 23
(3) figure 47 64, table 97 33, text 89 2, text 39 5, text 47 81, text 44 15, text 47 4
(4) table 97 18, text 69 1, text 58 1, text 47 23, text 47 16, text 47 56, text 47 52, text 47 21, text 47 21, title 27 2
(5) figure 101 93, text 49 25, text 49 25, text 46 7, text 47 7
(6) figure 101 44, figure 101 50, text 99 11, text 99 13
(b) PubLayNet
C.3 Generation Conditioned on Element Relationships
Element Relationship Constraints