LayoutPrompter: Awaken the Design Ability of Large Language Models
Authors: Jiawei Lin
Jiaqi Guo
Shizhao Sun
Zijiang James Yang
Jian-Guang Lou
Dongmei Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: graphic design
graphic layout
large language models
in-context learning

Abstract: Conditional graphic layout generation, which automatically maps user constraints to high-quality layouts, has attracted widespread attention today. Although recent works have achieved promising performance, the lack of versatility and data efficiency hinders their practical applications. In this work, we propose LayoutPrompter, which leverages large language models (LLMs) to address the above problems through in-context learning. LayoutPrompter is made up of three key components, namely input-output serialization, dynamic exemplar selection and layout ranking. Specifically, the input-output serialization component meticulously designs the input and output formats for each layout generation task. Dynamic exemplar selection is responsible for selecting the most helpful prompting exemplars for a given input. And a layout ranker is used to pick the highest quality layout from multiple outputs of LLMs. We conduct experiments on all existing layout generation tasks using four public datasets. Despite the simplicity of our approach, experimental results show that LayoutPrompter can compete with or even outperform state-of-the-art approaches on these tasks without any model training or fine-tuning. This demonstrates the effectiveness of this versatile and training-free approach. In addition, the ablation studies show that LayoutPrompter is significantly superior to the training-based baseline in a low-data regime, further indicating the data efficiency of LayoutPrompter. Our project is available at https://github.com/microsoft/LayoutGeneration/tree/main/LayoutPrompter.

LayoutPrompter: Awaken the Design Ability of Large Language Models
Jiawei Lin" Xi'an Jiaotong University kylelin@stu xjtu. edu. en
Jiaqi Guo Microsoft j [lagiuuo@micoooofer com
Shizhao Sun
Microsoft sh lriizsu@iccrooofft. com
Zijiang James Yang Xi an Jiaotong University zi jajanngxxjt.. wtt.ijjtuuudu..... edu.cn
Jian-Guang Lou Microsoft
Dongmei Zhang
Microsoft dongmeiz@mnicoosfft.co
j jlou@microsoft.com
Abstract
Conditional graphic layout generation, which automatically maps user constraints high-quality layouts, has attracted widespread attention today. Although cent works have achieved promising performance, the lack of versatility and data efficiency hinders their practical applications. this work, we propose Layout- Prompter, which leverages large language models (LLMs) to address the above problems through in-context learning. LayoutPrompter is made up of three key components, namely input-output serialization, dynamic exemplar selection and layout ranking. Specifically, the input-output serialization component meticulously designs the input and output formats for each layout generation task. Dynamic exemplar selection responsible for selecting the most helpful prompting exem- plars for a given input. And layout ranker is used to pick the highest quality layout from multiple outputs of L.Mss. We conduct experiments on all existing layout generation tasks using four public datasets. Despite the simplicity of our approach, experimental results show that LayoutPrompler can compete with or even outperform state-of-the-art approaches training or fine-tuning. This demonstrates the effectiveness of this versatile and training- free approach. In addition, the ablation studies show that LayoutPrompter is significantly superior to the training- based baseline in low-data regime, further indicating the data efficiency of LayoutPrompter. Our project is available here)
these tasks without any model
Introduction
Layout, which consists of of well-arranged graphic elements, plays critical role in graphic design. To alleviate the workload designers and allow non-expert users to engage in the design process, numerous studies have delved into the automatic layout generation for diverse user needs [7] 18 22 39) (i.e., layout constraints). Based on input layout constraints, existing conditional layout generation tasks be categorized into the following groups: constraint-explis generation (e.g., generating layouts conditioned on element types), content- aawre layout generation, and text-to-layout (see the left side of Figure [ for constraint examples). Early works in this field 19 21 22] primarily focus on individual tasks and develop task fask-specific model architectures and optimization methods. More recently, task gekkreen approaches [[5] 12 [4] have emerged. Compared to task-specific methods, they achieve greater flexibility and controllability on more tasks, while maintaining the quality of the generated layouts.
layout
Work done during an internship at Microsoft Research Asia.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Consiratni Srydeet Gonorsties
Generation Conditiaeed
Generation Conditioned
Types
Types and Sioee toolbor (9012) Ioon (12.12)
toolbar
Generatiea Croditioned Belationshgg
Iualber
bostlter
loxt
LayoutPrompter
Laysut Complettion
Laveud Haffeement
Dynamic Eximplar Selection
loobar R.53512)
22 231
Kon (.5
Layout Ranler
Idl DecumentLayee
00 Roster laghut
Sortaizztion
Costree- dwain lapour Geseration
Trah logass
LMM
introdaiing
from the Fort Mox,
page shoritd bave rite, and
groape to shon 3 siffact prot. aroup abvuld have of
and one bulken to click chock
the post.
Figure I: LayoutPrompter is a versatile method for graphic layout generation, capable of solving various conditional layout generation tasks (as illustrated the left side) across a range of layout domains (as illustrated on the right side) without any model training or fine-tuning.
methods [15 14 9 24 have achieved promising results, they still suffer from some limitations that impede their applications in real-world scenarios. First, the previous approaches struggle to simultaneously cope with all the layout generation tasks depicted in Figure They are typically tailored for specific tasks and cannot be applied to others. For instance, the state-of-the-arr diffusion-bart model LayoutDM [14] proposes to inject explicit layout constraints through masking logit adjustment during inference, but it fails to do so for implicit vague constraints, e.g.. constraints expressed natural language (i.e., text-to-layout), Consequently, distinct models need to be deployed for different tasks, leading inconvenience. This motivates to explore a more versatile approach for layout generation. Second, the existing methods are not data-efficient either: They usually necessitate extensive constraint-layout pair data for model training. For example, LayoutFormer++ [15] relies on the publaynet dataset [40] with a size of 300K for training to generate aesthetically pleasing document layouts. However, collecting such large datasets for some low- resource layout domains (e.g., poster layouts) is prohibitively expensive. Besides, even if such dataset is available, training time-consuming and costly. Hence, there is pressing need develop a data-efficient layout generation method.
Although
In this work, consider leveraging the powerful pre-trained large language models (LLMs) to address the above problems. The intuition behind is as follows. First, recent research has shown the versatility of LLMS in various tasks 28 38.. By carefully designing input-output formats, these tasks can be converted into sequence-to-sequeen
generation problems and effectively addressed by
LLMs. This emerging trend inspires us utilize LLMs to tackle all conditional layout generation tasks in a unified manner. Second, since the training corpus contains layout source code (28] 4 (c.g., HTML code and XML code), LLMs have acquired some layout-related knowledge during pre-training. For example, they may inherently possess the ability to align graphic elements and avoid unnecessary overlap between them, which beneficial for producing high-quality and visually appealing layouts. Consequently, an LLMs-based approach holds promise to enhance data efficiency compared existing models that are trained from scratch. Third, an additional advantage of LLMs lies in their remarkable in-context learning performance [3] 28
36 35 13] It means that instead
of fine-tuning LLMs individually for each layout generation task, we can simply prompt them to perform the desired task with a few input-output demonstrations. This characteristic further allows LLMs to generate layouts in training-fret free manner without any parameter updates.
To this end, we propose Layout (see Figure l). It formulates all conditional layout generation tasks as requence-to-sequent transformation problems and leverages LLMs tackle them through in-context learning. To unleash the full potential of LLMs for layout generation, two key issues need be addressed. First, how awaken the layout-related knowledge LLMs for achieving decent performance? Second, how to facilitate LLMs understanding diverse user constraints and layout characteristics distinct domains? LayoutPrompter tackles the two issues with the input-output serialization module and the dynamic exemplar selection module, respectively. I. Input-Output Serialization. Since prevalent L.MSs can only read token sequences, this module is responsible for representing constraints and layouts sequences that LLMs can sufficiently exploit their

---

Methods
Versatile
Data- Eata-Efficient
Training-Foo Free
LanooTTranffommrr 7 BLT [19] and so
9
24
16
30
x partially
x
X
LayoutFormer++ 15 LayouDDM
[4] LGDM
12
X
X
LayoutPrompter (ours)
Table 1: A comparison between existing conditional layout generation methods and LayoutPrompter.
related knowledge. To represent input layout constraints sequences, we borrow the successful experience of LayoutFormer++ [T]. where they present two simple but effective principles (i.c., constraint representation and constraint combination) to serialize constraints. We experimentally find that the serialization scheme is also effective for LLMs. To represent layouts as sequences, our principle is to convert them into format resembling what LLMs have encountered during pre me--trannngg thereby leveraging the existing layout-relate layout into the corresponding source code (e.g.. HTML) to obtain the output sequence. II. Dynamic Exemplar Selection. This module is used to select prompting exemplars that have similar layout constraints to the test samples. In contrast random exemplars, dynamic exemplars ensure that LLMMs receive the most relevant context, so they can better comprehend the desired constraints and produce plausible layouts accordingly. To support this technique, we develop an evaluation suite measure the constraint similarities between a given test sample and all candidate exemplars from the training set. Then, we select those with the highest similarity scores as prompting exemplars. In addition. we introduce a layout ranker to further improve LayoutPrompter's performance. Considering that LLMs can produce distinct outputs through sampling, we generate multiple layouts with the same input constraints, and use the ranker to select the highest-quality one as the final output.
knowledge within LLMs. Specifically, we serialize the
We conduct extensive experiments on various tasks and layout domains to evaluate LayoutPrompterr Experimental results show that LayoutPrompter can tackle all existing conditional layout generation tasks, demonstrating its versatility. Despite without any model training or fine-tuning, LayoutPrompter is on par or even better than the statc-of-the-art approaches. Besides, our ablation studies exhibit that LayoutPrompter can still achieve good performance when there is only a small set of candidate exemplars, indicating that it is superior to existing training- based methods in terms of data efficiency. In summary, LayoutPrompter a versatile, cata-efficieet and training- -free layout generation method.
2 Related Work
Graphic Layout Generation. Automatic graphic layout generation is an emerging research topic in recent years. To meet diverse user requirements, existing methods have defined various layout generation tasks, including layout generation conditioned on element types [2] 19 18, layout generation conditioned on element types and sizes [19], layout generation conditioned on element relationships [1, 21 layout completion 7 23 and refinement [30]. In addition to these constraint- explicit tasks, some works consider more challenging but useful tasks, such as content-aware layout generation [39 9 and text-to-layout [|] 24). Content-aware layout generation aims at arranging spatial space for pre-defined elements on a given canvas. The generated layouts not only need to be visually pleasing, but also avoid salient areas of the canvas. Text-to-layout is to generate layouts according to human language descriptions.
Early works in this field primarily focus individual task and propose task-specific approaches based on Generative Adversarial Networks (GANs) 22 9. Variational Autoencoders (VAEs) (2 [6]
and Transformers [19] 11 24
37 have ap- peared. LayoutFormer++ 15] proposes to represent various constraints as sequences and then leverages Transformer
30
Recently, some general approaches [15]
12
14
32 encoder-decoder architecture to generate layouts from constraint se- quences. [2] [4] develop diffusion-basee models for constraint-explici layout generation. However, none of the existing methods can simultaneously handle all layout generation tasks. Furthermore, these methods are highly dependent on large amounts of training data, which hinders their practical applications. In this work, we introduce techniques such dynamic exemplar selection, input-output serialization and layout ranking to effectively utilize LLMs to overcome the above limitations, making LayoutPrompter a versatile and data-efficient approach (see Ta.le
Large Language Models. Large language models (LLMMs) with billions of parameters, such as GPT 3 28 PaLM [6] and LLaMa [31], have demonstrated excellent few-shot performance
3

---

PREAMBLE
IN-CONTEXT LEARNING
generate layout
given information..
Preamblc
Tisk Description: comlent-sware layoul generation
Layout Domalu: poster layout
Cauves Size: carvas width 102p, canvas beight is 1.00p
EXEMPLARS
Ingun Constrantt
EXEMPLAR
Beenplor
Ourputlayeut
Coutent Constraing: left 26px. top 62px. width 50px, beight 69ps Type Constriiet logo text text underlay text
INPUT
INPUT CONSTRAINT
Dommslart
Layouut
<hon3>
<bady:
classe tego" stylew left: sper 2po0 widts:
OUTPUT LAYOUT
<dix teect stylew Jeft: Hipes too 12p1s wlath: Saper height: 189 byd ties odis "tea"" stylew left: tpus topt 25pe: aidres whaer neight taps siitiio
TEST
odii elaceslume fundeclly styleeltedes
topt
heiget:
neiights
Constraitt
odtv clacee styles Teiti 20pe; tap dipas wiath: spper neights ipe scidtiis </bndy
X
Ourgut
Ranker
Figure 2: An overview of LayoutPrompter. The complete prompt consists of task-specific preamble, N in-context exemplars and a test input. The exemplars are dynamically retrieved from the training set according to the test input. Subsequently, the prompt is fed into LLM to generate L distinct layouts. We employ a layout ranker to select the best one as the final output.
various natural language processing (NLP) tasks. Thanks to the emergent ability [3] brought by the scale of model and data, they largely outperform prior supervised approaches and match human-level performance on some tasks, without any finetuning. The versatility and effectiveness of LLMs inspire to develop layout generation method based them.
Recent studies show that the prompting strategy plays a crucial role model performance. For example, chain-of-thought (CoT) prompting [34 is proposed to improve the reasoning ability of LLMs by incorporating intermediate reasoning steps in the exemplars. Least-to-most prompting [4] (also known decomposed prompting) is introduced to solve complex multi-step reasoning tasks. To enhance contextual knowledge, [16] use a retrieval module to dynamically select in-context exemplars. They experimentally find that exemplars semantically similar to test samples can better unleash the model's knowledge. Specifically, they use a sentence encoder to convert model inputs to vector representations. Then, for each test sample, they retrieve the nearest neighbors in the encoded sentence embedding space to construct prompts. Motivated by them, we propose a similar prompting strategy in this work. Since the input of layout generation tasks is different from prior works, we introduce customized evaluation suite to measure sample distances. Experimental results demonstrate its effectiveness in LayoutPrompter.
17
LayoutPrompter
In this section, we elaborate on LayoutPrompter, versatile, data-efficient and training-free layout generation method built upon LLMs. Our main contribution lies in proposing a set of useful techniques for applying LLMs to layout generation. Specifically, to support sequence-to-sevee to-seqeence transformation and make maximum use of the design knowledge within LLMs, we carefully consider the serialization scheme that represents task inputs and outputs as sequences (Section 3.2) Moreover, to enhance the comprehension of user-specified layout constraints, we propose a dynamic exemplar selection module to retrieve the most relevant exemplars from the training set to perform in-context learning (Section 3.3). Besides, layout ranker designed to evaluate layout quality and rank multiple layouts generated under the same constraints, further improving model performance (Section 3.4).
3.1 Overview
Let's consider a conditional layout generation task. We denote its training set as D = {Zz 9;) 3 Here, (ır5, Uj) represents the j-th sample of D, which is an (input constraint, output layout) pair, and M the total number of samples. As illustrated in Figure 2 for given test query Tiess the in-context learning prompt P is composed by sequentially concatenating task-specific preamble R.

---

N exemplars and the query itself:
(1) To be more specific, the preamble R provides the essential information about the target task, such as the task description, layout domain and canvas size. (- and Fy(.) are serialization functions that transform task input and output y into sequences, respectively. G(:,-) denotes an exemplar selection function, which retrieves the in-context exemplars from according to Tiest- The details of Fx, Fy and G will be elaborated in the following sections.
P - [R; Fx(m.) Fr ....; Px (#RN)) ); (y Fx
(k.) G( D).
Notably, when the number of exemplars N is zero-shot learning. where LLMs predict the test output Best solely based on the preamble R and Tust- In our experiments (see Section B.2 in Appendix). we find that additional exemplar guidance can help L.MSs better comprehend the task and grasp the rough pattern of the required layouts. Hence, we opt for few-shot learning (N > 0) instead of zero-shot learning (N 0) in this work.
few-shot in-context learning degenerates to
3.2 Input-Output Serialization
To begin, first establish some notations. For each element that constitutes layout, we describe it by its element type c, left coordinate top coordinate t, width w and height h, i.c., e = (c, L, t, w, h). Here, is categorical attribute. The other four are numerical geometric attributes, which will be discretized the implementation (see Section A in Appendix).
Input Constraint Serialization. For constraint-explicit layout generation, the input constraints are element-wise We serialize such constraints in the
Cameis
Ouitaut buleeny Mal
constraints
same way as LayoutFormer+++ [15), where they repre- sent each constraint sequence and then combine dif- ferent constraints through concatenation. For example, if specifies the element types and sizes, (z) takes the form of (x) = "ciwih /e2wgh2| ..... ". this work, we adopt these ready-made sequences for constraint-explict layout generation tasks. Re- garding content-aware layout generation, the image nature of the input canvas poses unique challenge for serialization, i.e., enabling LLMs that can only read text perceive image content. Inspired by DS DS-GAN ( we recognize that the saliency map [8] can well capture the key con- tent shape of a canvas while discarding other high-frequency, irrelevant details (see Figure 3) To facilitate serialization, we further convert it into a rectified saliency map m = (m lm Wm, hm) by detecting region boundaries with pixel values greater than certain threshold. After prepro- cessing, the input canvas can be represented in a format understandable by LLMs: Fx "Content Constraint: left 4mpx, top fmpx, width WmPX, heightt hm.xx. For the text- to-layout task, where natural language descriptions are used to generate layouts, the constraint sequence is simply the input text itself.
Figure 3: An input canvas converted into saliency map.
Output Layout Serialization. For the output y. we propose to serialize it into the HTML format that LLMs are more familiar with and good at, rather than the plain sequence used in prior works 77 Following common HTML representation, we denote the complete output sequence as concatenation of multiple HTML segments Fy(y)= [ai; (2;....) Here, the i-th segment a; represents the i-th graphic element e; of y. It specifies the element attributes the following format:
<div class="c:" style= 11ft::4xxx top:4pxx; width:w.pp
(2) Thanks to the powerful in-context learning ability of LLMs, the test output ytess will be predicted in the same HTML format, making it easy to extract the required element attributes from the output. More input-output examples can be found in Section of the supplementary material.
3.3 Dynamic Exemplar Selection
As mentioned above, selects N in-context exemplars that have the most similar layout constraints to Trest from D. The selected exemplars are randomly shuffled and combined to construct P (see Equation [T. thereby enhancing I.LMs' understanding of various constraints. To achieve this, we design an evaluation suite measure the constraint similarity between the test query Tiest and each
5

---

RICO
PubLayNer
Thiks
Methods
mlot
FID 4
Align.
Overfap L
Vio. $
mloU
RID.
Align. 4
Overlap J.
Vio. S
BET
0.216
25.635
0.150
0,983
0.140
38.684
0.036
0.196
Gon-T
LayouzFarmer++
0.432
1.0%6
0.230
0.530
0.
0.348
8.411
0.020
0.008
0.
LayounPrompeer
0.429
3233
0.109
0.505
0.64
0.382
3.022
0.037
0,047
0.50
BLT
0.604
0.951
0.181
0.660
D.
0.428
7.914
0.021
0.419
0.
Gen-TS
LayoutFormer++
0.620
0.757
0.202
0,542
0.
0.471
0.720
0.024
0.037
0.
LayoutPrompier
0.552
1.458
0.145
0.544
0.18
0.453
1.067
0.049
0.091
0.
CLG-LO
0.286
8.898
0.311
0.615
3,66
0.277
19.738
0.123
0.200
6.66
Gen-R
LayoutFormur++
0.424
5.972
0.332
0.537
11.84
0,353
4,954
0.025
0.076
3.9
LayoutPrompter
0.400
5.178
0.101
0,561
10.58
0.347
3.620
0.037
0.161
12.29
LayourTrasffrmer
0.363
6.679
0.194
0.478
0.077
14,769
0.019
0.0013
Completion
Layoutormer++
0.752
4.574
0.077
0,487
0.477
10.251
0.020
0.0022
LayoutPrompker
0.667
7,318
0.084
0.428
0.476
2,132
0.023
0.017
RUTTE
0.811
0.107
0.135
0.183
0.781
0.061
0.029
0.020
Refinement
Rayontarmer++
0.816
0.032
0.123
0.449
0.785
0.086
0.024
0.006
LayoutPromper
0.745
0.978
0.1.9
014478
0.647
0.278
0.072
0.048
Table 2: Quantitative comparison with baselines on constraint-explictt layout generation tasks.
indicates larger values better, 4, indicates smaller values are better.
candidate exemplar (Ij, Us) E D. Then, G' can be further expressed Top-k selection function:
G D) Top-k(
U (R).);)ED
{ 25),,
Since we divide existing layout generation tasks into three categories, each with distinct input constraints, their similarity measures have different representations. We'll elaborate below.
Constraint-Explicit Layout Generation. As constraint-expliec layout generation tasks only consider element-wise constraints, we define (TEistt using inter-element constraint similarities. Specifi- cally, we construct a bipartite graph between Tiest = {Prest 1 u==1 and x {pf)y where p denotes the element-wise constraint on e. U, V are the constraint numbers of Trest Iy Then, the inter-element similarity W (i.e., the weight of bipartite graph) and the overall constraint similarity s are defined as:
s (Tiest xz))
Σ (PE..PP) JEM
W WPiist Pÿ), W (Pless Pī) =1 (Piss pJ)2 ll - (4)
Mmmxx
Here, a 0-1 function equal to 1 if Plest and p specify the same element type, and otherwise. This ensures that constraint similarity is only considered between elements with the same type. gies and вJ are specified geometric attributes of plr. and pz. Given the edge weight W of the bipartite graph, we adopt Hungarian method (20) to obtain the maximum matching Minaas And s ( Iry) is calculated as the average weight of matched edges (as shown in Equation
Content-Aware Layout Generation. The constraint of content-aware layout generation is the input canvas. The similarity of two canvases Tiest, V is defined as the loU (Intersection over Union) of their rectified saliency maps (see Section 3.2)
Mhest: my:
Miest O m,
s ( xz)) IoU(meat m.)aam)))
(5)
Mest U Um,l
Text-to-Layout We leverage the CLIP (29 text encoder to encode input texts into embeddings. The constraint similarity s ( I.) is defined as the cosine similarity input text embeddings Thest fly:
s( (Deasts 2z) A
Rhest 12
(6)
| [12
3.4 Layout Ranker
People usually judge the quality of generated layouts from two perspectives: (1) whether they are visually pleasing; (2) whether they look like the real layouts. Therefore, our proposed layout ranker follows the same principles evaluate layout quality. To be more specific, it measures the quality of an output layout using a combination of metrics:
X Alignment (Best) + Ag Overlapp((ea)) + A mloUU(hest)..
(7)

---

Dataset
Domain
Associated Task
# Training Set
# Test Set
# Element Types
RICO
Android
constraint- explicit layout generation
31,694
3,729
25
PubLayNett
document
constraint-expliit layout generation
311,397
10,998
5
PosterLayouu
poster
content-aware layout generation
9,974
905
3
WebUI
text-to-layout
3,835
487
Table 3: Dataset statistics. Note that these datasets are only used on specific tasks.
RICO
PublayyNet Pnerpie
Impui
Layour
bepst contraians
Layont Romeze
Layont Proniter
Layent
Layor
Fomner
201
Image
Gen-T
Box Burton
IuATiOn
backkgoood
[SS41
(20/77)
labi::
Gon-TS
kon
text
(100/))
tnd
1805
15431
150.0]
(50.4)
[50U
941
toc
[6624]
(4237)
147.300
1005 43
beet
Gen-R
The aquil
bettore
tabie
20
Completion
Refinement
Figure 4: Qualitative comparison between LayoutPrompter and the state-of-the of-tth--rttt baseline Layout-
+ormer++ [15 on constraint-explici layout generation tasks (better view in color and 2x zoom).
Here, ' A2 and A are hyper-parameters to balance the importance of each metric. Alignment and Overlap reflect quality from the perspective (I), while mloU mainly focuses on perspective (2). We will introduce them in Section The output layout with the lowest g value (lower indicates better quality) is returned as the final output.
4
Experiments
4.1
Setups
Datasets. We conduct experiments on 4 datasets, including RICO 27]. PubLayNet 40). Poster- Layout [9] and WebUT [24]. Their statistics and usages are illustrated in Table 3 For RICO and PubLayNet, we adopt the same dataset splits as Layoutoormer++ (15) While for PosterLayout, training set includes 9,974 poster-layout pairs, and the remaining 905 posters are used for testing. Regarding the WebUI datasct, we adopt the dataset splits provided by parse-then-place 224. In all cases, the in-context exemplars retrieved from the full training set.
Baselines. Since constraint-explici layout generation tasks have task-specific and task-generic methods, we compare LayoutPrompter against both kinds of fate--oftthe-al methods on these tasks. Concretely, we choose LayoutFormer++ [15] the common rask-generic baseline. The task-specific baselines are (1) BLT 19 for generation conditioned on types (Gen-T), (2) BLT [19 for generation conditioned on types with sizes (Gen-TS), (3) CLG-LO [88 for generation conditioned on relationships (Gen-R), (4) LayoutTraassomnnn Moreover, we compare LayoutPrompter with DS-GAN (9) and CGL-GAN layout generation. We compare with Mockup ||] and parse-then-place
7 for completion, and (5) RUITE 30 for refinement.
on content-aware
24
text-to-layout.

---

Val
Ove 4
Ali
Undy
Und, f
Uti 1
Occ
Rea
CGL-GAN
0,7066
0.0605
0.0062
0,8624
0.4043
0.2257
0.1546
0.1715
DS GAN
0,8788
0,0220
0.0046
0,8315
0.4320
0.2541
0.2088
0.1874
LayouPrompper (Ours)
0.9992
0.0036
0.0036
0.8986
0.8802
0.2597
0,0992
0.1723
Table 4: Quantitative comparison with baselines content layout generation task.
DS-GAN
Ours
(a)
(b)
(d)
(e)
(l)
@
(h)
Figure 5: Qualitative results generated by DS-GAN and LayoutPrompter content-aware layout generation. There are three element types, including logo (red), text (green) and underlay (yellow).
Evaluation Metrics. To evaluate the performance of LayoutPrompter and baselines, we use the following quantitative metrics. For constrann-expplcit layout generation and text-to- extttt--lyyout, we employ four standard metrics. Alignment (Align.) [22] gauges how well the elements in a layout are aligned with each other. Overlap [22] computes the overlapping area between two arbitrary elements in layout. Maximum IoU (mloU) [8] calculates the highest Intersection over Union (IoU) between a generated layout and real layouts. Fréchet Inception Distance (FID) [8] measures how similar the distribution of the generated layouts is to that of real layouts. Additionally, we introduce another metric Constraint Violation Rate (Vio. %) [15] to evaluate how well the generated layouts satisfy their input constraints. It is the ratio of violated constraints to all constraints, In text-to-layout, textual description may involve the type, position and size constraints of elements, parse-then-place [24] and further break down this metric into Type Vio. % and Pos & Size Vio. %. As for content-aware layout generation, we adopt the eight metries defined in DS-GAN (9. Some of them belong to graphic metrics, such as Val, Ove, Ali, Und, and Und,. Others are content-aware metrics, including Uii, Occ and Rea. Please refer
follow
for more details.
Implementation Details. In this work, we conduct experiments on GPT-3 3 text-davinci-000 model. We place N 10 exemplars in the prompt P. For each test sample, generate L = 10 different outputs Yest- The hyper-parameters involved in the layout ranker module are set to λ = 0.2.22 0.2, and À3- 0.6. When running GPT-3, fix the parameters to the default values of the OpenAl API, where the sampling temperature is 0.7 and the penalty-related parameters are set to 0.
4.2 Main Results
and Figures 4 5 6 show the quantitative and qualitative results on various layout generation tasks (see more qualitative results in Section Of the supplementary material). Although LayoutPrompter has not undergone model training and fine-tuning, the experimental results demon- strate that it can achieve comparable or even better performance than basclines, which proves that LayouPrompteer is versatile and training-fiee free layout generation approach. Below, we conduct detailed analysis of the experimental results.
Tables 2 4
Constraint-Explicit Layout Generation. Table 2 shows the quantitative results. On each constraint- explicit layout generation task, LayoutPrompter is compared with a task-specific method and another common baseline, LayoutFormer++ [1]]. Although not trained on these downstream tasks, Lay- outPrompter still exhibits competitive quantitative results. Furthermore, it even outperforms the baselines on some metrics (e.g., Align. and Overlap on Gen-T task, RICO dataset). The corre- sponding qualitative results are shown in Figure Here, we only compare with the state-of-the-ae baseline (measured by guantitative metrics), LayoutFormer++. The qualitative comparison indicates
8

---

m
FID
Align
Overlap
Type Vio. %
Pos Size Vio. &
Mockup
0.1927
37.0123
0.0059
0,4348
31.49
44,92
purse then place
0.6841
2.9592
0.0008
0.1380
11.36
19.14
LayoutPrompter (Ours)
0,3190
10.7706
0.0009
0.0892
15.09
23.78
Table 5: Quantitative comparison with baselines text-to-layout.
Mockup
PTP
Ours
Mockup
PTP
Ours
Mockup
PTP
Ours
(6) page for advertising holiday photo cards. The page should have image showing the card, a title "Holiday Photo Cards", a further description, and buttion "SHOP CARDS" for the user to click to purchase cards.
(b) The page introducing the products to the There should three parts, and each part includes one title, one picture of the click on
(e) A page for navigation of Mailreach. The page should have logo of Mailreach, links for users to check the email spam test, blog, features, pricing,
product, and one link to view more about the product.
log in, button for users click sign up.
(d) A footer page for online sture selling jowelty. On the page, there should have four lists of links for should be a title and 4-6 links.
(e) A page introduce payments There is litle ascepted paymentts en the top. There are also four logos of payment companies under it.
(f) page for showing different work types. There should be six groups. Each of them inclades logo and text which shows the title of the work.
info, In each liss, there
Figure 6: Qualitative results of Mockup, parse-then-place (short as PTP) and LayoutPrompter on text-to-layout (better view in color and 2x zoom).
that LayoutPrompter achieves as good controllability and generation quality as LayoutFormer++. First, the layouts generated by our approach satisfy various input constraints well, including type constraints, size constraints, relationship constraints, etc. Second, our approach can also produce visually pleasing layouts with well-aligned elements and small overlapping areas. Both qualitative and quantitative results demonstrate the effectiveness of LayoutPrompter.
Content-Aware Layout Generation. The quantitative and qualitative results are presented in Table and Figure 5 respectively. Remarkably, LayoutPrompter surpasses the graining-based baselines on almost all metries. This indicates that LayoutPrompter is capable of producing higher-quality and more content-aware layouts compared to the baselines. The rendered results further validate the conclusion. For example, in columns (f) and (g) of Figure 5 the layouts from DS-GAN (9] contain serious misalignments and overlaps. And column (c) shows that DS-GAN sometimes fails to generate content-aware layouts. In contrast, our approach can not only produce aesthetic layouts but also avoid the most salient objects in the input canvas, such as the person, teddy bear, car, etc.
Text-to-Layout The quantitative and qualitative comparisons shown in Table and Figure 6 Since text-to-layout is one of the most challenging layout generation tasks, LayoutPrompter slightly lags behind the current state-of-the-e -frt----rttttttt- method parse-then-place (24), especially on mloU and FID metrics. However, on the other four metrics, LayoutPrompter is comparable to the baselines. Thanks to the excellent understanding capability of LLMs, our approach can better satisfy the constraints specified in textual descriptions in some cases. For example, in cases (d) and (c) of Figure 6 LayoutPrompter successfully generates 4-6 links and four logos, while parse-then-place makes wrong predictions about the number of clements.
4.3 Ablation Studies
Effect of Introduced Components. LayoutPrompter has three key components, including input- output serialization, dynamic exemplar selection and layout ranking. To investigate their effects, we perform the following ablation studies (see Table/6p. (1) Since Layoutormert++ (15) has proven the effectiveness of constraint sequences relative to other formats, we only study the effect of HTML representation, which is not covered in previous works. Specifically, we replace HTML with plain sequence proposed by LayoutFormer++ [15] (denoted as w/o HTML) to represent the output layout. This results in a significant drop in FID and overlap metrics on Gen-T. (2) To understand the contribution of dynamic exemplar selection, we compare against its variant (w/o dynamic selection) that adopts random sampling for exemplar retrieval. LayoutPrompter achieves significantly better FID and mIoU across the board. Though the variant has better Align. and Overlap scores in some tasks, its noticeably poor FID and mloU scores indicate that fails to acquire the layout patterns in specific domains (c.g., the generated layout does not look like real UI layout). (3) To understand the

---

RICO
Tasks
Methods
mioU
FTD
Align.
Overlap L
Vio. %
LayoutPrompter
0.429
3.233
0.109
0.505
0.64
w/ HTML
0.460
7,009
0.106
0.663
Gen-T
dynamic selection
8.154
0.053
0.399
0.24
w/o layout ranker
3.149
0.142
Layourrompter
0.552
1,888
0.145
0.544
0.18
Gen-TS
w/o dynamic selection
0.337
8,107
0.199
0.400
0.24
w/o layour ranker
0.505
1.528
0.153
0,549
0.13
LayoutPrompter
0,400
5,178
0.101
0.564
10.58
Gen-R
w/o dynamic selection
0.223
14.177
0.067
0.597
15.95
layout ranker
0.341
5,282
0.137
0.545
6.54
LayoudPrompier
0.667
7.318
0.084
0.428
Completion
w/o dynamic selection
0.449
17.409
0.062
0.422
w/o layour ranker
0.580
11.194
0.093
0.451
Layoutrompter
0.745
0.978
0.159
0.478
Relinement
w/o dynamic selection
0.662
1.718
0.208
0.468
w/o leyout vanker
0,705
1.161
0.188
0.478
Table 6: Ablation studies of the introduced components on RICO.
LyyoutFormer++
Layoutrrompier
Tasks
Training samples
1
J
Align.
Overlap 4
Via. %
mloU 1
FID
Align.
Overlup 1
Via.
500
0.176
92.643
0.272
0.668
69.27
0.343
7.201
0.105
0.539
0.11
2,000
0.209
48.702
0.165
0.573
62.22
0.362
6.140
0.083
0.527
0.22
Gen-T
10,000
0.368
3,370
0.132
0.572
11.02
0.389
4.658
0.097
0.527
0.11
Full Set
0.432
1.096
0,230
0.530
0.
0,429
3,233
0,109
0.505
0.64
500
0.171
79.641
74.66
0.405
4.068
0.596
0.13
2,000
39.673
53.07
0.424
Gen-T8
10,000
0.529
2.395
0.464
0.06
Full Set
0.620
0.757
0.202
0.542
0.
0.552
1.458
0.145
0.544
0.18
Table 7: Ablation studies of training set size on RICO.
effect of the proposed layout ranker, we compare against variant (w/o layout ranker) that randomly picks layout from model outputs. We find that the layout ranker consistently yields improvements on the mIoU and Align. metrics of all tasks.
Effect of Training Set Size. We switch training set sizes: 500, 2000, 10000 and full set (see Table 7. In our approach, the training set represents the exemplar retrieval pool. The results show that the performance of LayoutFormer++ drops rapidly as the training data decreases, but our method much slightly affected. When training samples are limited (e.g., 500 and 2000), our approach significantly outperforms the maaseng-based baseline on all metrics. These observations suggest that LayoutPrompter is a more data-efficient approach, which is effective in low-resource scenarios. Due to space limitations, more experimental results on stability, the effect of the number of examples, and generalization ability can be found in Section of the supplementary material.
5 Conclusion and Limitation
In this work, we concentrate on leveraging Large Language Models (LLMs) for conditional layout generation address issues present in existing methods. To enhance the performance of our approach, we introduce three crucial components: input-output serialization, dynamic exemplar selection, and layout ranking. We conduct experiments on 7 existing layout generation tasks using 4 public datasets. Both qualitative and quantitative results highlight that LayoutPrompter is a versatile, data-efficient,, and training-free free method capable of generating high-quality, constraint-complian layouts. Despite these promising results, there are still some limitations. First, the performance of our approach is influenced by the number of elements the layouts, with more elements leading to more failure cases. Notably, this is not problem specific to our approach and has been observed in prior work [2] as well. Second, we have not studied whether LayoutPrompter is equally effective for other L.MSs such as PaLM and LLaMa. Third, with the rapid development of large multimodal models such as GPT-4V, PaLI (5) and LLaVA (25), we get a promising chance to extend LayoutPromple to supporting layout constraints specified in wide range of modalities. We leave them for future research.
10

---

References
[1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:237 13-3376,,,,,36, 2022.
[2] D. M. Arroyo, J. Postels, and Tombari. Variational transformer networks for layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13642-13652, 2021.
[3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
[4] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, F. Barnes, A. Herbert-Voss W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code, 2021.
[5] X. Chen, X. Wang. S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer, A. Kolesnikov, J. Puigcerver, N. Ding, K. Rong. H. Akbari, G. Mishra, L. Xue, A. V. Thapliyal, J. Bradbury, W. Kuo, M. Seyedhosseini, C. Jia, B. K. Ayan, C. R. Ruiz, A. P. Steiner, A. Angelova, X. Zhai, N. Houlsby, and R. Sorieut. PaLI: A jointly-scaled multilingual language-iman -mmag model. In The Eleventh International Conference on Learning Representations, 2023. URL /tttps://openrevittt
net Seerum?dddnWVoBea
[6] A. Chowdhery, S. Narang. J. Devlin, M. Bosma, G. Mishra, A. Roberts, Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023// 2022.
[7] K. Gupta, Lazarow, A. Achille, L. S. Davis, V. Mahadevan, and A. Shrivastava. Layouttrans- former: Layout generation and completion with self-attention. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1004-1014, 2021.
[8] X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In 2007 IEEE Conference on computer vision and pattern recognition, pages 1-8. Iece, 2007.
[9] H. Hsu, X. He. Y. Peng, H. Kong, and Q. Zhang. Posterlayout: A new benchmark and approach for content-aware visual-textual presentation layout, 2023.
[10] Y. Hu, C.-H. Lee, T. Xie, T. Yu, N. A. Smith, and M. Ostendorf. In-context learning for few-shot dialogue state tracking. arXiv preprint arXiv:2203. 200.00888.8.. 2022.
[11] F. Huang, G. Li, X. Zhou, J. F. Canny, and Y. Li. Creating user interface mock-ups from high-level text descriptions with deep-learning models. arXiv preprint 1rXiv:2110.0777., 2021.
[12] M. Hui, Z. Zhang. X. Zhang, W. Xie, Y. Wang, and Y. Lu. Unifying layout generation with decoupled diffusion model, 2023.
[13] S. Imani, L. Du, and H. Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.055. 053988, 2023.
[14] N. Inoue, K. Kikuchi, E. Simo-Serra, M. Otani, and K. Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. arXiv preprint arXiv:2303. 08137, 2023.
[15] Z. Jiang, H. Deng, Z. Wu, J. Guo, S. Sun, V. Mijovic, Z. Yang, J.-G. Lou, and D. Zhang. Unilayout: Taming unified sequence-t--sqqeeee transformers for graphic layout generation. arXiv preprint N0N0:2208.08033 2022.
11

---

[16] A. A. Jyothi, T. Durand, J. He, L. Sigal, and G. Mori. Layoutvae: Stochastic scene layout generation from label set. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 9895-9904, 2019.
[17] T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, and A. Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprini arXiv: 10.002000 2022.
[18] K. Kikuchi, E. Simo-Serra, M. Otani, and K. Yamaguchi. Constrained graphic layout generation via latent optimization. In Proceedings of the 29th ACM International Conference on Multimedia, pages 88 96, 2021.
[19] X. Kong, L. Jiang, H. Chang, H. Zhang, Y. Hao, H. Gong, and I. Essa. Blt: bidirectional layout transformer for controllable layout generation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pages 474-490. Springer, 2022.
[20] H. W. Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[21] H.-Y. Lee. L. Jiang, I. Essa, P. B. Le, H. Gong, M.-H. Yang, and W. Yang. Neural design network: Graphic layout generation with constraints. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 491-506. Springer, 2020.
[22] J. Li, J. Yang. J. Zhang, C. Liu, C. Wang, and T. Xu. Attribute-onnditioned layout gan for automatic graphic design. IEEE Transactions on Visualization and Computer Graphics, 27(10): 4039-4044 05994404, 2020.
[23] Y. Li, J. Amelot, X. Zhou, S. Bengio, and S. Si. Auto completion of user interface layout design using transformer- based tree decoders. arXiv preprint arXiv:2001.05308, 2020.
[24] J. Lin, J. Guo, S. Sun, W. Xu, T. Liu, J.-G. Lou, and D. Zhang. A parse-then- placce approach for generating graphic layouts from textual descriptions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[25] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In NeurlPS, 2023.
[26] J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2 101.06804, 2021.
[27] T. F. Liu, M. Craft, J. Situ, E. Yumer, R. Mech, and R. Kumar. Learning design semantics for mobile apps. In Proceedings of the 31st Annual ACM Symposium on User Interface Softwrre and Technology, UIST 18, page 569-579, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450359481.
doi: 10. 10.1145/3242587 3244655 URL
https:/ /stps:///oi... org/ 1001 3242587.288887 3242650
[28] OpenAl. Gpt-4 technical report, 2023.
[29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[30] S. Rahman, V. P. Sermuga Pandian, and M. Jarke. Ruite: Retining ui layout aesthetics using trans- former encoder. In 26th International Conference on Intelligent User
pages 81-83, 2021.
[31] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: /3971, 2023.
[32] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
12

---

[33] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, Emergent abilities of large language models. arXiv preprint arXiv:2266.06682, 2022.
[34] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint 2tt:2201.11903., 2022.
[35] K. Yang, D. Klein, N. Peng, and Y. Tian. Doc: Improving long story coherence with detailed outline control. arXiv preprint arXiv:22 10077, 2022.
[36] K. Yang, N. Peng, Y. Tian, and D. Klein. Re3: Generating longer stories with recursive reprompting and revision. arXiv preprint arXiv:2210.0677 2022.
[37] J. Zhang, J. Guo, S. Sun, J.-G. Lou, and D. Zhang. Layoutdiffusion: Improving graphic layout generation by discrete diffusion probabilistic models. Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 7226-7236, October 2023.
[38] Z. Zhang, A. Zhang, M. Li, H. Zhao, G. Karypis, and A. Smola. Multimodal chain-of- off-thuughttt reasoning in language models. arXiv preprint arXiv:2302.00922, 2023.
[39] X. Zheng, X. Qiao, Y. Cao, and R. W. Lau. Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics (TOG). 38(4): 2019.
[40] X. Zhong, J. Tang, and A. Yepes, Publaynet: largest dataset ever for document layout analysis, 2019.
[41] D. Zhou, N. Schärli, I.. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, O. Bousquet, Q. Le, and Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv: /0625, 2022.
[42] M. Zhou, C. Xu, Y. Ma, T. Ge, Y. Jiang, and W. Xu. Composition aware graphic layout gan for visual-textual presentation designs. arXiv preprint arXiv:2205.00300 00303, 2022.
13

---

A Coordinate Discretization
In this work, element coordinates are scaled proportionally into canvas of size Cw x CH. We follow the baselines to choose these two parameters. Specifically, in RICO, Cw = 90px, CH = 160px. In PubLayNet, Cw 120px, Cи 160px. In PosterLayout, Cw 102px, Cи 150px. In WebUI, Cw = 120px, Cи 120px. Then, the coordinates are discretized to the nearest integers.
B Additional Experimental Results and Analysis
B.1 Stability of Generation Performance
The output of LLMs varies with the random seed and hyper-parameters (e.g., the temperature). That is, for the same input constraint, LLMs are able generate many completely different layouts. Since the hyper-parameters are a trade-off between generation quality and diversity, we fix them the default values of OpenAI API and study the impact of random seeds on model performance. Specifically, we run inference on the test set 10 times, each using different random seed. Then, we calculate the mean and variance of each quantitative metric (see Table 8). The small variances indicate the stability of LayoutPrompter's performance under different random seeds.
RICO
Publ aye
Tooks
mloU1
FID
Align.
Overlap
Vio. %
mloU 1
FTD
Align. 1
Overlop
Vio. fi
Gen-T
0.36810.002
3:11880:055
0.13010.010
0.698200044
0.54630.148 1344000055
0,343a0.001
4.01430.
0,012a0.007
0.07200.002 0098-0.002
0.490g0.059
Gea-TS
0.504+0.001
1,8000.037
0.155+0:001
0.550-0005
0.303-0.001
201640.004
0.050+0.008
Table 8: Effect of random seeds. In this experiment, we disable the layout ranker to eliminate the impact of the ranking mechanism on model performance.
B.2 Effect of Exemplar Number
We conduct ablation experiments on the number of prompting exemplars. Figure S shows the zero- shot (N 0) qualitative results on Gen-T. obvious that LLMs fail generate reasonable layouts in zero-shot scheme. Table exhibits the quantitative comparison of the Gen-T task on RICO. The results indicate that the number of prompting exemplars mainly affects mloU and FID. Specifically, as the number of prompting exemplars increases, mloU and FID get improved. In summary, the number of exemplars has positive effect on the performance of LayoutPrompter.
PublayNet
RICO
Figure 7: Zero-shot results on the Gen-T task.
RICO
Tasks
# exeempar
mloU
FTD
Align. J
Overlap
Vio. %
0.381
5.007
0.115
0.491
0.85
3
0.413
5.098
0.120
0.492
0.51
Gen-T
5
0.414
4.521
0.492
10
0.427
3.523
0.092
0.486
0.67
Table 9: Ablation studies on the number of prompting exemplars. We run experiments on 1,000 test samples.
B.3
Generalization Ability
To investigate the generalization ability of LayoutPrompter, we compute DocSim similarity between the generated layouts and their prompting layouts (see Table 10p. The DocSim of Layout- Former++ computed between the generated layouts and training layouts. The quantitative results show that LayoutPrompter achieves competitive even better scores compared to LayoutFormer++,
14

---

indicating that LayoutPrompter has close generalization ability to the training based method. In addition, we exhibit the qualitative results of the generated layouts and their prompting layouts in Figure The results demonstrate that LayoutPrompter is capable of generating meaningful variations different from the prompting ones.
Method
RICO
PubLayNet
LayoutFormer++
0.8472
0.8266
LayoutPrompter
0.8563
0.8119
Table 10: DocSim Gen-T task. The smaller the DocSim, the better the generalization ability.
Generated Layouts
Retrieved Layouts
Generated
Leyouts
PublayNet
RID
Figure 8: Qualitative results of the generated layouts and corresponding retrieved layouts on Gen-T task. Each case contains generated layouts.
15

---

C Additional Qualitative Results
In this section, we present additional qualitative results LayoutPrompier. These results further demonstrate the versatility and effectiveness of our method, which can generate high-quality and constraint-compliat layouts on multiple layout generation tasks.
C.1 Generation Conditioned on Element Types
(1)
(2)
(4)
(5)
Element Type Constraints
(1) icon, input, text*8, text button '3, toolbar
(2) background image, icon*5, 5, image*3, 3, slider, text*3, toolbar
(3) Icon, list item *6, text*7, toolbar
(4) background image, image, text button"5
(5) background image, icon, image*2, 2, text* button*
(6) advertisement, Icon image*5 5, toolbar, web view*2
(a) RICO
(1)
(2)
(3)
(4)
(5)
(6)
Element Type Constraints
(1) 1ex title*2
(4) figure, text*6, title* table, text"6, title
(2) table, text"5, title
(3) table, text"5
figure, text*6, title*
(b) PubLayNet
Figure 9: Qualitative results Gen-T on RICO and PubLayNet. The element type constraints are in the table.
16

---

C.2
Generation Conditioned on Element Types and Sizes
(1)
(2)
(4)
(5)
(6)
Element Type Size Constraints
(1) advertisement 10, background image 90 144, image 21 7, text 25, text text button 10, text button
54 10, text button 8 7, text button 33 7, text button 54 10, web view 70 10, web view 70 10
(2) icon 12 12, text 29 5, text button 85 10, text button 85 10, toolbar 90 12
input 110, text
text button 28 9, text button toolbar
(4) icon image 69 39, text 90 5, text
button
(5) input 53 11, on/off switch 11, radio button 17 11, radio button 19 11, text 15 13, text 74 5, text 26 4, text
button 15 9, text button 15 8, text button 45 text button 45
(6) image 71 60, text 81 9, text 76 26, text button 72 16
(a) RICO
(1)
(2)
(4)
(5)
Element Type and Size Constraints
(1) figure 90 56, table 99 46, text 99 4, text 32 1, text 99 3, text 47 11, text 47 11
(2) figure 65 74, text 99 13, text 47 16, text 47 39, text 47 23
(3) figure 47 table 33,
2, text 39 5,
81,
text 47
(4) table 97 18, text 69 1, text 58 1, text 23, text 1,, text 47 56, text 47 52, text 47 21, text 47 21, title
(5) figure 101 93, text 49 25, text 49 25, text 46 7, text 47 7
(6) figure 101 44, figure 101 50, text 99 11, text 99 13
(b) PubLayNet
Figure 10: Qualitative results of Gen-TS on RICO and PubLayNet. The element type and size constraints are in the table.
17

---

C.3 Generation Conditioned on Element Relationships
Element Relationship Constraints
(1) background image, image,
button"2,
canvas, button 3 canvas, smaller
background image
(2) image, text 4, text button,
bottom image O, larger text 2, button bottom
(3) icon, list item 6, text"7, toolbar, list item 2 top canvas, list item center canvas, list item center canvas, list item
1 bottom icon 0, list item 2 larger icon O, list item larger icon 0, text 7 right icon 0, toolbar 14 top list item 1, list item 3 equal list item 2, list item bottom list item 2, text 7 top list item 2, text 13 smaller list item 2, text 7 smaller
list item
(4) input*2, text*3, text button*4, text button bottom canvas, text equal input text top input text button smaller input text 3 smaller input text button smaller input text button 7 bottom input text button larger text text button bottom text 4
(5) image, text"3, text button" 3, text top canvas, bottom image 0, bottom text text button bottom text text button 5 top text button*3, toolbar, input center canvas, text button canvas, equal icon 1, button larger icon button larger icon input bottom image 3, button smaller image text button bottom input text button larger input text button bottom input button & larger text button
icon f3, image, input*2,
(a) RICO
(1)
(2)
(3)
(4)
(5)
(6)
Element Relationship Constraints
(1) table, text"4, text bottom text text 4 smaller text
figure, text*3, text smaller figure 0, bottom figure
(3) figure, text"5, text center canvas, text 5 bottom canvas, text bottom text 2
(4) tex,"13, title" 2, title 13
text bottom 0, text bottom text text bottom text text 6 smaller text title smaller 1, title smaller text 2, larger text 3, text text 10 larger 4, bottom text text 12 larger smaller
text text larger text title
bottom text text bottom text 4, text bottom
title larger 6,
larger text text smaller text 9,
title smaller 10,
title top text 10, text 12 smaller text
(5) figure, text*5, title, text smaller figure 0, text 5 bottom figure 0, text 5 smaller text 2, title bottom text 3, title center text 5 (6) figure, list*
bottom figure right list
(b) PubLayNet
Figure 11: Qualitative results of Gen-R on RICO and PubLayNet. The element relationship constraints in the table.
18

---

C.4 Layout Completion
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
(a) RICO
(1)
(3)
(4)
(5)
(7)
(9)
(10)
(11)
(12)
(b)
PubLayNet
Figure 12: Qualitative results of completion on RICO and PubLayNet.
19

---

C.5 Layout Refinement
(1)
(2)
(3)
(5)
(6)
RICO
(1)
(2)
(3)
(4)
(b) PubLayNet
Figure 13: Qualitative results of refinement RICO and PubLayNet. Note that each group has two layouts. The left one is the noisy layout, and the right one the refined layout.
20

---

C.6
Content-Aware Layout Generation
EREST
(1)
(3)
(4)
(5)
(6)
(7)
(9)
(10)
(11)
(12)
Figure 14: Qualitative results of content-awar layout generation on Posterl.ayout.
C.7 Text-to-Layout
(1)
(2)
(4)
(5)
(6)
Textual Descriptions
(1) page to an insurance claim. There should be two text boxes. One for the title 'Need to an insurance claim' and other short description. Two buttons are necessary to click to file or manage the claim. (2) footer page showing homepage of company. There should
entries text links. And the text links
should be unfolded. Users can get access to different info through these links.
(3) page to introduce an online editor. The page should contain an image, and the slogan "Online Infographics, Diagrams, & Charts Maker" to have brief introduction, and a further description of the specific service of the editor. Then button is needed for the user to click to learn info.
(4) page for introducing the home tours. The page should have a title, "home tours". Also, there are four groups and each group has image and title.
(5) page for introducing the service provided by website. The page should have three groups of information sets:
"Web Development" "Social Media", and "Link Building". Each group should include small image, title, further introduction, and a "READ MORE" more info by clicking (6) header page for navigation out stock, a logo, and two links "HOME" view products to shop. Also, two buttons for search and shopping cart are needed.
online shop. page should have announcement to explain the product user to return the homepage
DITTO HANGERS"
Figure 15: Qualitative results of text-to-layout on WebUI. The textual descriptions are in the table.
21

---

D Prompt Examples
Here, we show prompt example for each layout generation task, including Gen-T (Table T.
Gen-TS (Table 12 Gen-R (Table 13). completion (Table 14), refinement (Table 15 content-aware
layout generation (Table 16 and lext-to-layout (Table 17
Preamble
Please generate layout the given
Task Deseription:
generation conditioned given element types
Layout Domain: android layour
Size: camvas 90px, canvas beight 160ps
Exemplar 1
Element Type Constraint: imagel image immgee image [est
<heml>
<hody>
<div clage="inage" style="left:15p"" top:12pa; xidth:61pr; #2pptt::22===///dr
<div class="inage" style="laft:55prr sop:77ps; width:20ps; 55ight:50pe* <x//ávv
"lnas=" atyle="iett:5dpr top:96px;; kidth: 18px: beight:11per 10px"
<div class="inage" styla="left:0pp top: 143pp; width:00ps: siight:dpe+><///
<div classe"cext" style="left:3pp. top: width:2ips; height 7px <>///i>>>
class-"text" style="left:22px top; sidth:44gs beight:9px t><//di>>
<div clazse"ccett button" ityl: top:167px:; width:
</body>
</html>
Exemplar
Element Type Constraint: image image image image pagerr indicator text text text button
text button
text button
<htel>
<hody>
<div claso-"Inage" style-"lett:Opxz top-spx; :9dth:90pe;; height:93px ">p/////
clazs="inage" style="Jaft:30pt top: 20dth::00p: 10pe
class="Inage" style="left:38pt top:96px; 1idth: height: 12pp">//iv
<div claze="inage" ctyle="left:
d0p:;)px; 2tpx; height: 12pp">//div>
class="pagor indicator" stylen"lett:ppe top: 119px; vidth:90px; tpightts
<div clars="sext" "tyle="left:0ps; 9pp:90p;; vidth:90pr; beight:10px:>c/dts
<div classn"cext" style="left:Ops, top: 109px; 90pa;; height: Spx"></div>
claco=lsext button" msylo=1left:Opx; sop:12fpx; vidth-45px;; he////p
<div clazen"rcct button" :ifft: top: 12p:12 vidth:45px;; height::0pe<>>//ttr
<div claze="sext button' style="left:Ops top: 13p:136px; width 90px; height:1lpp"><</ev
</body>
</htn/>
Test Sample
Element Type Comstraint: icon image
image
button
toelbar
(Generated by LLMs)
Table 11: A prompt example of Gen-T on RICO.
22

---

Preamble
Please generate layout based on the given information.
Tack Description: generation conditioned on given element types and sizes
Layout Domain: amdroid layout
Cumva Size: canvas width 90px, canvas height 160px
Exemplar
Element Type and Constraint:
input (npput
text 4| text button 85 10 button 10 4 toalhar 90 12
<htnl>
<body>
<div clasg="icon" ebyle="leCt:0pp: top:5px; width:12ps helghht-11px*><<ddl
<div clans="image" style="laft: top:1ipx; uidth:Opr; height:0px'></a <//11>>>>>>>>>
<div class="input" style="left: top:40px; vidch:81px; height:10pa"><//G
<div claas="input" style="left::pp: 28px; vidth:81pk; height: 10p="//11
class="taxe" stylo="laft:4pp. top:2ipx; width:81ps; height:
<div clase="tezt" stylo="Ieft:115t top: width:22px; dpx//t
<div claase"text button" style="left: 6op:62pr; vidth:86px; height::0pe""<//at
<div claser"text button" exyle="left: top:43px; width:10ps dpx
<div "Lolaart "tyle="eeft:0pp top:5px; width: 80px; t2igh::12px"><</iie
</hody>
</htm1>
Exemplar 2
Element Type Coustraint: 41
input 12 inpur 12 inpur 9 input
lext
toolbar
<hial>
<bady>
<div class="card' style="left:ip:: top:18px; width:86px; beight=<ip='><//y:
<div claas="icon" atyle="lert:0px; top:5px; width: 12px; height: 12px">>/av>
<div class="input" style="loftt:pp:; 30po; uidth: height; 12pp>></dp
<div clase="input" style="left:SSx: 23p:;3pxr; vidth:78ps; teight::1pa/>></dii
<div claas="input" atyle="laft::pr:; top: 7idth:78px; Spxght:::x>>///t0
<div class="input" style="left:Epx: top:36px; vidth:6ipx; Beight::p:"></dii
<div clase="tezt" atyle="iefv:15p:: 8op:8px; width: 15px; height:
<div clase="text
style="left: 60px: top:51px; 25dth:25px;; height:
<div claas="text button" 7tyl top:38px;
13px; height:7pe"><//tt>
<div claser"text buttan" style="left: top:62px; width: d2pp; boig/t:dpx"><<ddv>>>>
<div clase="toolbar" "tyle="left:Opp: top:5px; vidth: 80px; t2igh::12px""x//iiv>>
</hody>
</htm]>
Sample
Element Type and Coustraint: icon inpot 83 iipput 83 9 text 83 text button text button 77 5 toolbar 90 12 (Generated by L.Mss)
Table 12: A prompt example of Gen-Te on RICO.
23

---

Preamble
Plawe genente layout
given
Task Descriptio:: generation conditioned on given element relationdips
Layont Domain: android layur.
Comoa Sibr: camass width is 90ps, height 160pp
Exemplar
Elemout Type Constraint: image imngg image text text sext text text button boolbar
Elemeut Kelationshpp Constralat; text bottom image
lurger image
0 tnolbar
larger image () image equal image I text smaller inage 2| lext smaller image 2. toolbar top text
<body>
class="Leage" stylon"lede Oppc: top: 7po; 7px; height: Tox"></div>
<d19 clase="Arage" Saylen"latt:lppp top: 2idth:20px; height 20pp">///t s//dioe
class="Lzage" stylentlefe (30p: top: 101px; 28px; height tesghtt:8ppr><</dio
<div clasa="text" style="leCt:Bpp. top: viath: 28px; Spight::Se">//dl
class="sect" teyh22t topu width:40px: hei Spi
<div class="text" 1eCl 18p:; wiath: B2pp: bpigght:Be
class="tect" styleehlfftstt tops 140ge; width:5tp:: 7
clare="text button" "lylle top-lpe: width:1dps; besgtt::lpp"></tty
<div class""socibar" style="lefe:0ps top: 2idth:90px;; height: 11px" ></di>>
</body>
</2tm1>>
Exemplar
Elemout Type Constraiot: text
text
Flemou Relatiouship Consiralar: text horiom 0 sext equai test
<body>
clarg="cest" stylle-t 1pp:: top:/pp:
90pp:
<div class="textt ctylemfledb:3pp; 19p:; sadth: 93dth:83px; Bright::0pxt></dds
class="cext" styze:t2etc:3p: sop-87px 83pth
<aiv class="twict? stylee top:96ge;; width: beight::0ge /x</av>>>
<div clang="sect button" style="Jafs:0pr; 149:140pm; widsh:80pm: beight::1pp/>>///do
</body>
</htmi>>
Tist Sample
Elemout Type Constraist: limaee text jest] Element Relatiouship Consiraiat: test top canvas text top comvas test 2 right ioon image (Genevated by LLMs)
text button bettom icon
Table 13: A prompt example of Gen-R on RICO.
24

---

Preamble
Please genernte layout hased the information.
Task Deseription: layout completion
Layout Domain: android layout
Camas Size. canvas width 90px, canvas height is 160px
Exemplur
Portial Layout: image 5:47 40
<html>
<body>
class="Inage" legl:="left::lpx; top:Epx;; 41dth:47px; 40igh::40pp>>>//ttt
clasge"rext button" atyle="left:Oppr top:53px; vidth:84px; t5p
<div clazs="insge"" style="eeft:7ps: top:74px: width:9pa; bp/
<div class="text" "dyle="left:11px; top:7dpx; width:67px; height: Spx"></ii>>
<dtvv classe"text: button" atyle="laft: top:86px; vidth:84px; height:
<div claxs="text butison' style="efftiipp; top: 104px; 86px; height-:2px"><//he
<div claes="text
style="eeft:ips; top: 136px; width:86px; height:11px""</di>>
</body>
</htnl>
Exemplar
Layout: image
<htnl>
<hody>
<div class="inage" ssy:==" top:5px; width:56px; height:t1p
<div clazs="insge' style="left::px: top:17px; width:90px: height:
claes="text" style="left::po; top:65px; vidth:86px; height toigght:dipx">///ii
classe"inage" spyle=1left:0px; top: 108px; 8idth:80px; heiggt::pa">>//fiv
<div "paze="pager indicator! Myll='1eft top:11dps; videh:12px; beight:3px/></iiiv
<div class="teett butzon" slyle="left: 124p1;;;:; width: beight:15px"><//0
<div classe"text button" style="left Glpp; top:13/px; vidth:17px; 17px; 1esght::0px>><//
class="rext" style="1oft: top: 140px; width:bipx; height:op""></dtv>
</bady>
</htnl>
Test Sample
Partial Layout: image 10:65
(Generated by LLMs)
Table 14: A prompt example of layout completion RICO.
25

---

Preamble
Please generate layost based the information.
Task Description: layout refinement
Layout Domain: layout
Canver Size: width is canvas height 160px
Exemplar
Noise Layont: adveriseneet
76
image
text 29 11 toolbar 5 88 16 web view 69 12 web view 11 70 12 web view 20 90 140
<btel>>
<body>
<div class=""dvertieenne atyle="Iaft:10pss top: wipth:70px; heighh::1px">>//dv
<div clare="icon" style="left:77ps top16px; width:12ps; beight::1lp<><</ir>
<div clans="lcon" atyle="laft::0p; topsfpx;; width:11pos 1eighh::1
<d1v clazs="Image" style="laft (1pp: top:opa;; wiath: 1dpo; heigtt::lppe>>//tv>>
claas="taxt" atyle="left: top:6pe; vidth:2ipe; beight
<div clazs="text" style="ledt: top: 25pa; height:bpx>></dir>
<div class="tocibar" style="left /dpr; tap: width width: Bopx; height: <>px1><
clase="eeb style="ledt:100a; top:18px; 7ipsh:70ps; brigh::11px:<<//ir>>
class="geb itylen"iete:10p: top:l rodth: toiga::1ipr"><//1r>
<div clare="veb view" "let:o00 top: width width:90px:; beight::41pxx></// /xc/div>>
</body>
</hsnl>
Exemplar
Noise Layonf: adventisement 0 11 background image 145 icon 17.67 icon 194-6 image 13 10 text 35 18.5 text button 80 button 16 64 button 1497]
text button 10 29 68 11] text button 39 88 12, web view 10 69 12 web view 9 69 10
<btmi>>
<body>
class""sdrertiseett" style="Left: Uper; top:tipx; :idth:99px;; beight: "></di>>>
<div clase="background inage" seyle="left::0e; top:Spx; widsh:90ps; 1iigh::-1dpp/><//ir
<div clasg="icon" style="leds:Spp; top:19ps; widthripx; hoighh::dss>></dtv
<div class="icon" style="lefsrilpy top:19ps; vidthr4ps; 4piphh=-p
<div class="inage" style="left:2ppt top:7ps; Sidth:Spx; beighh:Sps"></dii
<div clans="inage" atyle="laft: top:16ps; 2idth:2tps: height::pr""></dis
<div clase="text" style="1efs:35pm top:7pss widthr18ps; bright--5x""b</dii
claas="taxt button" atyle="att::8lpr top: width: "////t
<d1v class="text button" stylem"lett:10p;; top: 16po;;;; sidth:tspx; 10//">111>
<div class="taxt button" style="latt:Slpt, top:16pp; i /
<div clase="text button" stylo="lefb:1ip;; top:27px; vidsh:60px;; beight:10px
classe"taxt button" atylee"lett:0p:: 41p4ipp; wieth: 90px; height: 1pt
<div clase="tub view" "tyle="lett:10p:; top:5px; width: 70px; teight::0ps"></dii
<div clasg="seb view" style="left:10pt: top: 70px; hoight::0px"></6a
</body>
</htal>
Sample
Noise Layout: icon 68 12 icon 5 icon 80 13 text 14. toolbar 0 10 web view 18 90 130 web view 19 19 130
(Gencrated by LLMss
Table 15: A prompt example of layout refinement on RICO.
26

---

Preamble
Please generate hased the given information.
Task Description: content-awe laynut generation
Layout Domain: poster layout
Canras width 102ps, canvas height 150px
Exemplar
Content Constrmint: left 25px, top 25px, width 30px, height 12x Type Constraint: logo underlay text
<html>
<body>
<div class="loge" style="left:34px; tap:14px; vidth:66px;
<div class="text" style="lefe: :1lle= tap:25px; width:94px; height:3
<div classr"underlay" mtyl=="laft:19pp top:37px; sidth:85px; 48pp
<div class="text" atyle="lett:3epp top:40px; 6idth:64px; height:4ipx'></dts
<div class="text" style="lett:28pr; top:48px; width:74px; height::3px'></di
</bady>
</html>
Exemplar
Content Constraint: left 23px, 60px, width 56px, height 69px Element Type Constrainr: logo text underlay text
<html>
<bady>
<div clasg="loge" styles")aft: top:0px;; width:66px; height: 10pxx></ivv
<div clasg="text" style="laft: 16pp;; top:22pp;; width:86px; height:38px'><//tt
clasg="underlay" atyle="left: top:3/px; 73dth 49px""/4
claes="text" style="lert:35pr; top cop:40px; width:67px; height:47px'>>>>
</body>
</htnl>
Test Sample
Content Constraint: left 26px, top 62px, width 50px, height 60px Element Type Constraint: logo text text text underlay text (Generated by LLMs)
Table 16: prompt example of content-aware layout generation on PosterLaayou..
27

---

Please genenite liyout based the given information.
Tark Deseription: text-Io-lyont
Layont Domain: web layout
Caraas Size: camvus wodth 120px, canmas height is 120px
Exemplar
header page company Fishionnaly Latellc. page, should include a namo the company, three anvigation links "Home" "Chr Fashionably Tiems" and "Retor/Refind Policy" for the user the homepuge, items company, know to have three icons for login, search, and shopping cart.
refund policy. Besides, necessory
<oody>
<div clase="ion "let: 101p:: top:3pm; widsh: Spt; top
clase
"oglleeleett:00po top: 2pxt boigah::pst>></dd>>>>
<div "ico""" style="left: 110p::110s; top: 2px; widch: height::p4">><//55
style- S1eft:30pp top: vidth:dps; beighe:2px"></dn>
<div classs "link" "iett top: width:17pa; /bigh:::p44">/////
"lin" flyl=="laft::6pp" kop: 1éds beight: "oc/////
<dsv clanse "tstle" style=" top:ōpm;; width:24ps; beight::ps
</body>
</2tml>
Exemplar
Tost header page for websil Pag AAA, There should have a logo un left, and three
links 'LOG 'SIGN LP", 'UPLOAD',
click them
and upload somsthhnng.
<stal>
Goody>
<div clasa- "link" "tyte= (8pp: top: Spdt height: <peiie//
myll="laft::900 top:Opes wide height bpight::pettoceeee
<dsv clans 'Lank" "lotte" top: 2pot; widchrdpms besght t//crr
clase "Logo" style- 220f: (2p:; top; width 15pp;; besght:
</body>
</htm/>>
Tost Sample
header page "hommen", pupe, should include of the website. Five links (' ("Latest". "Topt00", "About", "Register", and "Login' ") button "Create" and an icon are on the LLMs)
Table 17: A prompt example of text-to-layout on WebUI.
28


Reviews:
[{"review_id": "PHR9uyw4Ls", "reviewer": "Reviewer_7S5o", "summary": "The authors propose a new conditional layout generation method using LLMs with few-shot learning. The method uses a pre-trained GPT-3 with two techniques: dynamic prompting and layout ranker. Dynamic prompting is to provide few-shot examples with similar constraints to the input query using ConsSim, and layout ranker is to rank the layout using weighted metrics. The authors show that this method can generate layouts that comply with different types of conditions across domains, and the qualitative and quantitative results show that the method is comparable and sometimes better than existing training-based, domain-specific methods on various metrics.", "questions": "1. The Gen-T and Gen-TS results in RICO in Fig 5 look very similar, and I wondered if the authors intentionally chose them for comparison. Otherwise, it seems impossible since the conditions are generic and have many possible solutions. \n\n2. In the ablation studies (Table 5), the Gen-T results w/o HTML achieve 0% constraint violations, and the proposed method performs the worst. Are there possible reasons? The other tasks do not have results w/o HTML, and I wondered if the same situation can be observed. \n\n3. It would be interesting to see the results in the refinement task with different perturbation levels.\n\n4. Have the authors tried fine-tuning LLMs on these tasks?", "limitations": "Yes, the authors have addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The proposed method addressed the problems of existing methods: reducing the training cost using few-shot learning and providing abilities across domains and tasks by properly designed input/output formats and pre-trained LLMs.\n\n2. In addition to generic layout conditions, the proposed method can be applied to two more challenging tasks: content-aware generation and text-to-layout. \n\n3. The two techniques, dynamic prompting and layout ranker, are simple and effective without extra fine-tuning or complex heuristics.\n\n4. The authors demonstrate the possibility of leveraging LLMs for design-related tasks.", "weaknesses": "1. It will be great to see more comparisons and discussions on content-aware layout generation. For example, some prior works, such as CanvasVAE [1] and CGL-GAN [2], are missing here. \n\n2. It would be great if the authors could examine if the retrieved examples are similar to the given test set layout. For instance, in RICO, multiple layouts might be extracted from the same app, so they might look very similar and provide a way for the model to cheat. \n\n3. Ablation studies can be improved by adding, for example, zero-shot results, results of different ways to represent the constraints, results of varying screen resolutions, results in groups with different numbers of elements, etc.  \n\n4. It would be great if the authors could conduct a user study, ideally on designers, since none of these metrics can perfectly capture human perceptions/judgments.\n\n[1] CanvasVAE: Learning to Generate Vector Graphic Documents, Kota Yamaguchi, ICCV 2021\n[2] Composition-aware Graphic Layout GAN for Visual-textual Presentation Designs, Zhou et al., IJCAI 2022\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness1. It will be better to compare with other content-aware layout generation works, such as CanvasVAE and CGL-GAN.**\n\nWe clarify that content-aware layout generation refers to generating layouts based on the image content, i.e., avoid putting graphic elements on top of main objects (e.g., faces, animals, and products) and create a pleasing feeling. For CanvasVAE, it can not be applied here directly.\nThere are two reasons.\n1. CanvasVAE is an unconditional VAE model, so canvas/content cannot be input into the model as a condition.\n2. CanvasVAE generates the entire design with many attributes, such as the element color, opacity and image, which is beyond the scope of layout generation.\n\nTherefore, we don't compare with it in the current paper and leave this to future work. \n\nFor CGL-GAN, our baseline DS-GAN has outperformed it on most quantitative and qualitative results. We didn't include it in the paper, and will add it in the revised version.\n\n**Weakness2. The retrieved examples could be similar to the given test set layouts.**\n\nWe manually check some test layouts and their corresponding retrieved examples.\nFigure 15 (in the attachment) shows the qualitative results.\nWe can observe that the retrieved examples are close to the test layouts, but not exactly the same.\nWe think they provide good references to generating different and reasonable layouts.\nIn the future, we will try to develop quantitative metrics to make the study along this direction more comprehensive and solid.\n\n**Weakness3. Ablation studies can be improved.**\n\nThanks for your comments.\nWe share the ablation studies we have explored here.\n1. Zero-shot results. \nThe zero-shot qualitative results are shown in Figure 17 (in the attachment).\nIt is obvious that LLMs fail to generate reasonable layouts in a zero-shot scheme.\nWe'll add zero-shot results in the revised version.\n\n2. Results of different ways to represent constraints. \nActually, we have tried different ways to represent constraints but do not find a better way than the one proposed by LayoutFormer++.  Take Gen-T as an example. \nWe find that the constraint format \"4 images\" performs worse than the constraint format \"image | image | image | image\" used in LayoutFormer++. \nBesides, we find that using a fixed order (e.g., alphabetical order) for element types as LayoutFormer++ can improve performance.\nFor example, \"image | image | text | text\" is better than a random order \"image | text | image | text\".\n\n3. Results of varying screen resolutions.\nThe four datasets we used in experiments have different screen resolutions.\nOn RICO, the screen resolution is discretized to $90px\\times160px$.\nPubLayNet has a screen resolution of $120px\\times160px$.\nPosterLayout has a screen resolution of $102px\\times150px$.\nWebUI has a screen resolution of $120px\\times120px$.\n\n4. Results in groups with different numbers of elements.\nPrevious work, such as VTN [1], has found that as the number of elements\nincreases, quantitative measures like FID and Overlap become worse. \nWe will include a thorough analysis of this in the revised version.\n\n**Weakness4. It would be great to add a user study.**\n\nWe will add a user study in the revised version. \nWe would like to point out that most of state-of-the-art work, such as LayoutFormer++ [2], LayoutDM [3] and LDGM [4] do not conduct user studies and we think quantitative and qualitative results are also enough to demonstrate the effectiveness of our method.\n\n**Question1. Some results of Gen-T and Gen-TS in RICO are similar.**\n\nIt is by coincidence.\nWe use Layout Ranker to select the best layout from 10 outputs.\nIt chooses the one similar to the result generated by LayoutFormer++ by chance.\nTo illustrate that our method can generate diverse layouts, we present other generated layouts (who are not ranked as top 1 by Layout Ranker) in Figure 18 (in the attachment).\nWe can observe that they differ from those shown in the paper, and they are also plausible and satisfy the constraints.\n\n**Question2. Possible reasons why the experiment w/o HTML achieves the best constraint violation.**\n\nThe one w/o HTML uses the plain sequence to represent the layouts, i.e., $\\text{type}_1, x_1, y_1, w_1, h_1, \\text{type}_2, x_2 ...$.\nWe conjecture that the better constraint violation comes from that the plain sequence is much shorter than HTML code.\nNote that though w/o HTML achieves 0\\% constraint violation, it has very pool FID and overlap metrics (see Table 5).\nDifferently, the proposed method has negligibly performance drop in terms of constraint violation (from 0\\% to 0.64\\%) and achieves significantly better quality (e.g., FID and overlap).\nThe same result can be observed in Gen-TS, where w/o HTML achieves 0\\% constraint violation.\n\n**Question3. It would be better to consider different perturbation levels for refinement task.**\n\nWe now follow the perturbation noise in LayoutFormer++.\nWe'll try bigger and smaller noises and add the results in the revised version.\n\n**Question4. Whether fine-tuning LLMs is tried.**\n\nWe have not tried fine-tuning LLMs on these tasks, since our target is to develop a training-free layout generation method.\nIn addition, while there are parameter-efficient fine-tuning techniques for LLMs, they are still much more expensive than a training-free method.\n\n[1] Variational transformer networks for layout generation. CVPR2021.\n\n[2] LayoutFormer++: Conditional Graphic Layout Generation via Constraint Serialization and Decoding Space Restriction. CVPR2023.\n\n[3] LayoutDM: Discrete Diffusion Model for Controllable Layout Generation. CVPR2023.\n\n[4] Unifying Layout Generation With a Decoupled Diffusion Model. CVPR2023.\n"}, {"review_id": "T2e9RqLnYp", "reviewer": "Reviewer_HKER", "summary": "This paper leverages large language models to solve the layout generation tasks. It shows a great performance when generalizing to other domains or other tasks. Since in-context learning is used, there is no need to train the model on a vast amount of data.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The prompting method shows a good performance and does not need large amount of training data.\n2. The new method can well generalize to new domains and new tasks without an additional model", "weaknesses": "1. The paper used text-davinci-003 as LLM, but it is also worthwhile to see how other LLMs perform with the proposed prompt-based method. Can these prompts perform well on other LLMs, including the in-house or open-sourced ones.\n2. The in-context learning is not stable with LLMs. The sample prompt may have different results depending on the random seed and temperature parameters. The paper didn't report the variance of the results. \n3. In line 258, it is also necessary to mention the parameters for running LLM to repeat the experiments. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness1. How other LLMs perform with the proposed method.**\n\nInitially, we tried Alpaca 7B.\nHowever, the model is prone to generate heavily overlapping layouts, and sometimes the output is not even in the right format.\nWe then switched to text-davinci-003 and were successful.\nWe agree that it is an interesting topic to investigate the performance of other LLMs.\nWe will continue to experiment with other powerful LLMs, such as GPT-4 and LLaMa 2, and contribute more techniques and results to the community.\n\n**Weakness2. Stability of LLMs results.**\n\nSince temperature is a trade-off between generation quality and diversity, we used a moderate value of 0.7, which is also the default value of OpenAI API.\nHere, we share our exploration for the impact of random seeds on generating layouts.\n1. Qualitative results. In Figure 18 (in the attachment), we show the generated layouts (four layouts in each group) under the same constraints but different random seeds.\nAlthough they look different, they all satisfy the given constraints and are visually pleasing.\n\n2. Quantitative results.\nTo study the effect of random seeds on LLMs, we generate 10 different outputs for each test sample, and then calculate the mean and variance of all metrics (see Table 14 in the attachment).\nWe disable Layout Ranker in this experiment to eliminate the impact of post-processing on model performance.\nThe small variances show the stability of the proposed method to different random seeds.\n\nCombining these two aspects, we conclude that though LLMs generate different layouts under different random seed, the performance is relatively stable.\n\n**Weakness3. The parameters for running LLMs.**\n\nThanks for your advice.\nWe'll add the parameters for running LLMs in the revised version.\nIn fact, for all the parameters, we use default values in OpenAI API.\nFor example, $\\text{temperature} = 0.7$, $\\text{top p} = 1$, $\\text{frequency penalty}=0$, $\\text{presence penalty}=0$."}, {"review_id": "kKbug7r8oO", "reviewer": "Reviewer_kjZH", "summary": "In this paper, the authors introduce LayoutPrompter, a novel approach for conditional graphic layout generation using\nlarge language models with minimal demonstrations.\n\nThe authors frame the graphic layout generation task as a sequence generation problem. They represent layouts as HTML\nfiles, leveraging the fact that large language models are pretrained on a corpus that frequently includes HTML. A unique\naspect of this approach is the use of a Dynamic Prompting strategy. In this strategy, the model retrieves exemplars that are\nmore closely related to each test sample based on a defined constraint similarity metric. This ensures that the in-context\nexamples used for prompting are highly relevant. Furthermore, a Layout Ranker is appended to the model, which selects\nthe optimal layout post-inference. Notably, the method does not require extensive training data and demonstrates\nadaptability across various layout domains.\n\nTo validate the efficacy of LayoutPrompter, the authors carry out experiments across different layout domains. The\nperformance of LayoutPrompter is benchmarked against various task-specific and task-agnostic baselines. Remarkably,\nLayoutPrompter yields competitive results even without domain-specific training or fine-tuning. It is worth noting that\nLayoutPrompter is particularly effective on a custom WebUI dataset, which is constructed to investigate a new text-to-layout\ngeneration task. This outperformance is more pronounced when compared to results on other layout domains and\nagainst other baselines.", "questions": "Ablation Study Details: Could the authors elaborate on the methodology used for the ablation study, and provide insights into why\nDynamic Prompting and Layout Ranker might not have shown significant effectiveness. Would considering different variations of\nthese techniques lead to different outcomes?\n\nDetailed Experimental Analysis: Could the authors provide a more detailed analysis of the experimental results? Specifically, why\nLayoutPrompter performs better on certain metrics compared to others. Moreover, what are the factors that contribute to its varied\nperformance across different domains?\n\nGeneralization Across Domains: The paper claims that the method can be generalized across domains. Can the authors provide\nadditional empirical evidence or examples to substantiate this claim? Specifically, how will the model perform in a zero-shot scheme or\ngiven demos in different domains?\n\nHandling of Failures and Underperformance: It would be helpful if the authors could elaborate on the specific cases where\nLayoutPrompter fails or underperforms. What are the common characteristics of these cases and are there any strategies that could be\nemployed to mitigate these issues?\n\nClarity and Organization of Writing: There are instances where the writing is not very clear or well-organized, as observed in the\nexplanation of closing the gap between existing methods and human designers. Could the authors clarify the intended message and\npossibly consider restructuring this section for better clarity?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: LayoutPrompter is innovative in using large language models for layout generation without fine-tuning. The\nadaptation of dynamically choosing better demos (Dynamic Prompting) and output verification (Layout Ranker) to this\nfield is a clever utilization of existing techniques.\n\nQuality: The method's solid integration of Sequence Generation, HTML representation, Dynamic Prompting, and Layout\nRanker underpins its robustness. Comprehensive benchmarks add credibility to the research.\n\nClarity: The paper is well-structured, with clear explanations of methodology and results, making it accessible and easy to\nfollow.\n\nSignificance: LayoutPrompter's ability to generalize without extensive training data is noteworthy, indicating the potential for\ndiverse applications and research in layout generation.", "weaknesses": "Limited Novelty in Techniques: The application of large language models for layout generation is noteworthy, but the utilization of\nHTML Representation, Dynamic Prompting, and Layout Ranker lack innovation. It would be interesting to see the exploration or\nintroduction of new methodologies tailored for this domain.\n\nInconclusive Ablation Study: The ablation study on the Dynamic Prompting and Layout Ranker does not convincingly demonstrate\nthe effectiveness of these two techniques.\n\nInsufficient Experimental Analysis: While the paper presents experimental results, it lacks in-depth explanations and analyses.\nElaborating on the implications of these results, and providing a more detailed interpretation, would add substantial value.\nIn-depth Analysis of Failures: The paper could provide a more detailed analysis of cases where LayoutPrompter fails or\nunderperforms. Understanding the limitations and areas where the model struggles would offer insights for future research and\nimprovements.\n\nWriting Clarity and Organization: The paper could improve in terms of clarity and organization. For instance, in discussing the\nmotivation for bridging the gap between existing methods and human designers, the text lacks coherence and the last two points seem\nto overlap. Clearer and more structured presentation of concepts would enhance the paper’s readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Limited novelty in techniques.**\n\nLeveraging large language models for layout generation is by no means an easy task.\nIt requires a deep understanding of the layout domain and the according innovations, which can be grounded in our three design choices.\nFirst, layout representation.\nDue to the inclusion of layout source codes in the training data (e.g., HTML code for Web UI), LLMs have acquired some useful layout knowledge during pre-training. \nHence, we represent layouts as corresponding HTML codes to better leverage the existing layout knowledge within LLMs.\nSecond, ConsSim retrieval criterion in dynamic prompting. \nTo help LLMs better understand the constraints, ConsSim is designed to select the examples with the similar constraints in the prompt.\nThird, a layout ranker is proposed to select the layout with the best quality. \nWith these three insightful techniques, our LayoutPrompter achieves competitive performance with state-of-the-art approaches.\n\n\n**Ablation study details**\n\nIn the ablation study, we primarily analyze the effectiveness of Dynamic Prompting and Layout Ranker. \nTo understand the contribution of Dynamic Prompting, we compare LayoutPrompter against its variant (w/o Dynamic Prompting) that performs random sampling for exemplar retrieval and keeps the other components the same as LayoutPrompter.\nSimilarly, to understand the effectiveness of Layout Ranker, we compare LayoutPrompter against a variant that randomly picks a layout from an LLM's outputs.\nAll the comparisons are conducted on 5 conditional layout generation tasks.\n\nTo obtain a more in-depth understanding of Dynamic Prompting, we additionally include 2 studies on the effect of retrieval pool size and the number of exemplars in the prompt.\nBeing mindful of the LLM experiment costs and the limited time for rebuttal, we randomly select 1,000 samples from the test set for the experiments.\nThe results are in Table 15 and Table 16 (in the attachment), from which we can observe that the pool size and the number of exemplars mainly affect the mIoU and FID metrics. \n\n**Inconclusive Ablation Study**\n\nWe kindly argue that Dynamic Prompting and Layout Ranker have shown effectiveness in ablation studies.\nCompared with the variant (w/o Dynamic Prompting), LayoutPrompter achieves significantly better FID and mIoU across the board. \nThough the variant achieves better Alignment and Overlap scores in some tasks, its noticeably poor FID and mIoU scores indicate that it fails to acquire the layout patterns in specific domains (e.g., the generated layout does not look like a real UI layout). \nWe also provide the qualitative results of this variant in Figure 19 (in the attachment).\nBy comparing Figure 19 and Figure 5, we can conclude that Dynamic Prompting yields better layout quality.\nFor Layout Ranker, it consistently yields improvements on the mIoU and Alignment metrics of all tasks.\n\n**Insufficient experimental analysis.**\n\nDue to the black-box nature of deep learning, we cannot precisely tell which factors attribute to the good performance against state-of-the-art methods and the difference in various layout domains.\nStill, we try to discuss two potential factors.\n\n1) Intrinsic layout complexity of each domain. \nPubLayNet is a document layout dataset, while RICO is a dataset of Android UI.\nElements in a document layout are typically well-aligned and do not overlap with each other.\nHowever, overlapping elements in UI are frequently observed (see Figure 15 in the attachment for comparison). \nSuch an intrinsic difference may explain why a method often achieves better Alignment and Overlap scores on PubLayNet than RICO.\nBesides, the less aligned and larger overlapping nature of UI layouts may make training-based methods struggle to acquire the layout patterns, which provides a chance for LayoutPrompter to outstand, as it is only exposed to layouts with similar constraints.\n\n2) Pre-training data distribution. \nIt is known that the distribution of the pre-training dataset has a significant impact on large language models' performance on downstream tasks.\nIn the publicly available web corpus, the number of Android UI relevant corpus (e.g., Android application source code from Github) may be significantly larger than that of documents.\nAs a result, language models are more familiar with Android layouts, which may explain LLMs achieve more state-of-the-art metrics on RICO.\n\n**Generalization across domain.**\n\nFor cross-domain, we intend to mean that LayoutPrompter can solve layout generation tasks on different layout domains using the same methodology, rather than performing in a zero-shot scheme or on given demonstrations in different domains.\nWe will revise this term.\n\nAs suggested, we include the zero-shot results in Figure 17 (in the attachment), from which we can observe that LLMs fail to generate visually pleasing layouts.\n\n**Handling of failures and underperformance.**\n\nOne typical failure happens when the number of elements is large.\nThis is not a problem specific to our approach and can also be observed in LayoutFormer++.\nWe think there are two possible reasons.\nFirst, it is more difficult to place the elements properly as the number of elements increases.\nSecond, the average number of elements in the training set is around 10, which may lead to the issue that existing methods do not perform well when generating layouts whose element number is above average.\n\n**Clarity and organization of writing.**\n\nThanks for your comments. We will polish the writing.\nSpecifically, for the discussion about the gap between existing methods and human designers, we plan to organize it as follows.\nFirst, the 'high cost' (Line 39) will be explained more clearly as high training cost and reliance on a large amount of data.\nSecond, the 'weak cross-domain ability' (line 47) will be revised since it is a bit exaggerated as you pointed out.\nWe will put the discussion about layout domains in the point 'relatively task-specific'."}, {"review_id": "osJG8UfRF9", "reviewer": "Reviewer_mVCb", "summary": "The authors propose LayoutPrompter, which is a new approach for generating high-quality, cross-domain graphic layouts without any model training or fine-tuning. It can handle a variety of conditional layout generation tasks and outperforms existing methods on some metrics. The approach is competitive with state-of-the-art approaches on five traditional conditional layout generation tasks and even extends to solve two challenging problems that have more flexible constraints. ", "questions": "(1) Figure 1 is a bit hard to understand as a teaser figure. It took me a while to figure out that the left parts refer to the “ five typical tasks and two more challenging tasks”, while the right parts are the subdomains of discussion…\n\n(2) Missing reference for saliency map on line 161\n\n(3) Are there any ablation studies on the effect of the number of prompting examples?\n\n(4) Regarding the new dataset: \n- What is the source of the website? \n- What kind of HTML elements are included? \n- What is the portion of each element? \n- How was the quality controlled (e.g., removing ads, removing unrelated banner, etc)? \n- How was the instructions for layout design collected? \n- Does the instruction creation process involve human effort or was it automatically generated?\n- Where is the data sheet for this new dataset?\n\n(5) Out of curiosity, may I ask why the authors did not conduct any human evaluation?\n", "limitations": "The authors have included discussions of the limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The strength of this submission lies in its clear motivation, well-written content, and ease of understanding. The authors have effectively articulated the purpose and significance of their research, making the motivation behind the study readily apparent to the reader. The paper is written in a coherent and structured manner, which contributes to its ease of follow throughout. Furthermore, the study impressively conducts solid experiments that encompass a range of layout design settings, providing robust empirical evidence to support their findings. Overall, the submission showcases a strong foundation and execution, making it a valuable contribution to the field.", "weaknesses": "(1) One major weakness of the submission is the lack of detailed information regarding the new dataset, WebUI. While the authors mention its collection, they provide very limited information about the data collection process or the format of the dataset. This omission raises several questions that remain unanswered. It would be beneficial for the authors to include a section in the main paper or supplementary material that thoroughly describes the data collection process, including the sources, selection criteria, and any preprocessing steps applied. Additionally, specifying the format of the dataset (e.g., file type, structure) would provide valuable insights for readers.\n\n(2) Another weakness pertains to the organization and clarity of the dataset setup explanation. As the submission covers multiple tasks and settings, it would greatly enhance readability if the authors were more specific about which dataset was used for each setting. Currently, the explanation of the dataset setup (lines 227-236) appears disorganized and challenging to follow. (I searched for the experimental results on WebUI and found nothing, then it took me a while to figure out that WebUI was referred to as text-to-layout…) To address this, I recommend the authors consider incorporating a table that lists the tasks and corresponding datasets. This table would help readers quickly identify the datasets utilized for each specific setting, reducing confusion and facilitating a more straightforward interpretation of the experimental results.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness1. The details of WebUI dataset are missing.**\n\nThanks for your advice. We'll include a section in the supplementary material to describe the dataset details.\nHere, we first provide you with the information.\n\n1) Source.\nWe mainly collect the company homepages of the Fortune 500 companies (Microsoft, Google and so on).\nSince these web pages are very long, we divide them into several sections according to the id values in the HTML (Header section, Hero section, Product section, Footer section and so on).\nWe then filter ads and cookies based on text content to control the dataset quality.\n\n2) From HTML to layouts.\nIn the last step, we have acquired many web sections. \nWe convert these web sections into corresponding layouts in this step.\nSince HTML non-leaf nodes are usually used as containers without specific semantics, we only include leaf nodes in the final layouts and filter out all non-leaf nodes.\nThen, we determine the element type by the HTML tag.\nFor example, elements within an \"h\" tag are considered \"title\", elements within an \"a\" tag are considered \"link\", elements within a \"div\" tag are considered \"text\", elements with a \"src\" attribute are treated as \"image\" and so on.\nElement positions can be automatically extracted from the \"top\", \"left\", \"width\" and \"height\" attributes.\nSo far, we have converted HTML into the corresponding layouts.\nWe filter layouts with more than 50 elements. We also filter out heavily overlapping layouts.\n\n3) Text annotation.\nWhen the layouts are ready, we recruit annotators to annotate natural language descriptions for each layout.\nFirst, annotators were asked to label 30 samples. Then, one expert checked their results and gave them feedback. This training was performed in 2 rounds to select 15 qualified annotators. \nAfter annotation, 5 experts evaluated each data sample. A\nsample was accepted only when more than 3 experts agreed\nwith it. \nFinally, we got about 5000 text-layout pairs to construct the first text-to-layout dataset, WebUI.\n\n4) Data sheet.\nEach sample in our constructed dataset is a text-layout pair.\nAnd each layout contains $N$ element types and $N\\times4$ coordinate attributes.\nWebUI has a total of 5212 text-layout pairs.\nThe average text length is 40.5.\nThe average number of elements is 10.1.\nThe maximum number of elements is 50.\n\n**Weakness2. The dataset setup is disorganized.**\n\nThanks for your advice. \nWe'll add a table as you suggested in the revised version for better readability.\nHere, we clarify that RICO and PubLayNet are used in 5 typical layout generation tasks, PosterLayout is used in content-aware layout generation, and our constructed WebUI is used in text-to-layout.\n\n**Question1. The teaser figure is hard to understand.**\n\nWe will revise the figure title, explaining the meaning of each part of the teaser figure, to make it easier for readers to understand.\n\n**Question2. Missing reference for saliency map.**\n\nWe're so sorry we neglected to add the reference for saliency map.\nWe'll add it in the revised version.\n\n**Question3. Ablation studies on the effect of the number of prompting examples.**\n\nWe show results on the Gen-T task of RICO in Table 16 (in the attachment).\nDue to limited time and the experiment costs, we run experiments on 1000 test samples.\nThe results indicate that the number of prompting examples mainly affects mIoU and FID.\nSpecifically, as the number of prompting examples increases, mIoU and FID get improved.\nWe'll experiment with the full test set on more tasks, and add the results in the supplementary material.\n\n**Question4. The dataset details.**\n\nWe have answered it in Weakness1. \nPlease see Weakness1 for the dataset details.\n\n**Question5. Human evaluation.**\n\nWe will add a human evaluation in the revised version. \nWe would like to point out that most of state-of-the-art work, such as LayoutFormer++ [1], LayoutDM [2] and LDGM [3] do not conduct any human evaluation and we think quantitative and qualitative results are also enough to demonstrate the effectiveness of our method.\n\n[1] LayoutFormer++: Conditional Graphic Layout Generation via Constraint Serialization and Decoding Space Restriction. CVPR2023.\n\n[2] LayoutDM: Discrete Diffusion Model for Controllable Layout Generation. CVPR2023.\n\n[3] Unifying Layout Generation With a Decoupled Diffusion Model. CVPR2023."}, {"review_id": "10Y6sRGvyb", "reviewer": "Reviewer_Lp5t", "summary": "This paper presents a set of few-shot techniques to enable large language models to generate geometric layouts, particularly in the domain of User interfaces, posters and documents. These techniques include dynamic prompting and layout ranker modules, which allows the LLMs to learn from relevant examples in the dataset and filter out badly generated UIs from various samples. The presented technique enables cross-domain, task-generic generation of layouts without the need for training, utilizing existing knowledge in LLMs which was unexplored in prior work.\n", "questions": "These questions are directly relevant to the weaknesses of the paper:\n\n1) What is the average qualitative and quantitative similarity of the generated UIs against the most similar UIs in the prompt? It would be great if the author(s) can use an established metric (e.g., DocSim) and compare this similarity against existing work. Existing work that doesn't use prompt can be compared against the most similar entry in the entire dataset.\n\n2) What is the pool of UIs that the prompt UIs can be sourced from? Is this only from the training set of various domains?\n\n3) How does the performance of the model correlate with the size of the dataset that the prompt UIs can be sourced from? It would be great if the author(s) can perform some ablation experiments along this dimension.\n\n4) Can the author(s) discuss further the significant drop of FIDs w/o dynamic prompting? This discussion could be qualitative.\n", "limitations": "Limitations are adequately addressed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Enabled LLMs to perform a novel task. This task is also typically challenging for LLMs to perform due to its highly geometric nature.\n\n- Experiments and prompts that lead to the success of layout generation in LLMs would constitute a significant contribution, given the challenging nature of this task in LLM.\n\n- Introduced dynamic prompting and layout ranker, of which their formulation and implementation can be useful for future layout generation work\n\n- Achieved competitive performance in some layout generation tasks, without the need for training\n", "weaknesses": "Some of these weaknesses are pointed out in the limitations sections by the authors themselves, but I would like to highlight them since I believe they are important for the significance of the current paper:\n\n- **Reliant on existing dataset's coverage of the target task to achieve good performance:** This is shown in the ablation study that the FID is significantly higher without dynamic prompting. This is an important issue in the UI domain where data is difficult and costly to collect.\n\n- **Generalisation ability of LLMs in generating UIs:** Since the dynamic prompting module relies heavily on the provided prompt, it is unclear whether the generated UIs are novel and are meaningful variations for design inspiration. This can be shown by computing the similarity between the generated UIs and the prompt UIs, and comparing this with existing work (which might be generated UIs against the entire dataset for existing techniques that don't use prompts).\n\n- **Diversity and Coverage of the generated UIs:** Related to the above point, it is unclear how much of the entire design space that the generated UIs can cover. Existing work computes the Wasserstein distance between that of the generated UIs and the test set, of which this work did not compute (partially due to the conditional nature of all evaluated tasks). I wonder if the author can perform some unconditional UI generation experiments to explore the knowledge of the overall UI design space that is learned by existing LLMs.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness1. Reliant on existing dataset's coverage of the target task to achieve good performance.**\n\nTo understand the effect of dataset's coverage on model performance, we regard the size of the layout pool that prompt UIs can be sourced from as a proxy for coverage, and we study how the model performance on Gen-T task varies as the pool size increases.\nBeing mindful of the LLM experiment costs and the limited time for rebuttal, we randomly select 1,000 samples from the test set of RICO and PubLayNet for the experiment.\nThe results are reported in Table 15 (in the attachment).\nIt is observed that when the pool size is only 100 (i.e., small coverage), all the metrics are poor on both datasets.\nHowever, when the pool size increases to 500, all the metrics show significant improvement, in particular FID.\nWhen the pool size continues to increase from 500 to 10,000, although the overall trend of FID is still improving, the change speed slows down, and the other metrics only have marginal improvement.\nWith a size of 10,000 layout pool, the model has already been on par with the one using the full training set (Full Set).\nTo sum up, although increasing the coverage of the dataset can indeed improve the model performance, the improvement is limited when the coverage is increased to a certain extent.\nIn the revised version, We will experiment with the full test set on more tasks and add the results in the supplementary material.\n\n\n**Weakness2. Generalization ability of LLMs in generating UIs.**\n\nTo investigate the generalization ability of LayoutPrompter, we take your kind suggestion and compute the similarity (measured by $DocSim_g$) between the generated UIs and prompt UIs.\nWe also compare with LayoutFormer++, for which the $DocSim_g$ is computed between the generated layouts and training layouts.\nWe present the quantitative and qualitative results in Table 13 and Figure 16 (in the attachment).\nQuantitative results show that LayoutPrompter achieves a competitive or even better $DocSim_g$ score with LayoutFormer++, indicating that LayoutPrompter has a close generalization ability to LayoutFormer++.\nQualitative results indicate that LayoutPrompter is capable of generating meaningful variations different from the retrieved layouts.\n\n**Weakness3. Diversity and Coverage of the generated UIs.**\n\nThanks for your advice.\nWe do think our approach can perform unconditional layout generation tasks.\nTo achieve so, we should randomly choose prompt layouts in Dynamic Prompting rather than using ConsSim, as there are no constraints as input for prompt layouts retrieval.\nOur preliminary hypothesis for unconditional layout generation is that with the LLM prior design knowledge and a properly sized layout pool, LayoutPrompter could perform on par or even better performance with state-of-the-art layout approaches.\nWe will implement this and include a fair experiment in the revised version to comprehensively investigate the problem.\n\nIn addition, FID is a measure of the distribution difference between the generated layouts and the real layouts.\nTherefore, we argue that the competitive FID of our method on the representative conditional layout generation tasks could also reflect the diversity and coverage of the generated layouts.\n\n**Question1. The qualitative and quantitative similarity of the generated UIs against the most similar UIs in the prompt.**\n\nPlease refer to our response in Weakness2 for the study results.\n\n**Question2. What is the pool of UIs that the prompt UIs can be sourced from?**\n\nThe prompt UIs are all sourced from the training sets of various domains.\nWe follow the dataset splits in the baselines for fair comparisons.\nSpecifically, the splits of RICO and PubLayNet follow LayoutFormer++, and the splits of PosterLayout follow DS-GAN.\n\n**Question3. How does the performance of the model correlate with the size of the dataset that the prompt UIs can be sourced from?**\n\nPlease refer to our response in Weakness1 for the study results.\n\n\n\n**Question4. The significant drop of FIDs w/o dynamic prompting.**\n\nDynamic Prompting is responsible for selecting prompt layouts from the training set with constraints similar to a given one.\nIt helps the model understand the characteristics of various layout constraints and thus generate high-quality layouts conforming to the given constraint.\nWithout Dynamic Prompting, the model hardly acquires the layout patterns for certain constraints, leading to an inferior FID.\nFor example, the RICO dataset contains a ``drawer'' element, which represents the side navigation bar and is usually placed on the left side of the canvas.\nIf the model is not familiar with its meaning and layout pattern, it is likely that the drawer will not be placed in the correct position, and thus the generated layout will not be a plausible one, ultimately hurting the FID score.\nWe also include more qualitative results for the variant w/o Dynamic Prompting in Figure 19 (in the attachment)."}]

Author rebuttal:
First, we thank the reviewers for recognizing our simple yet effective work that enables LLMs to generate graphic layouts in a training-free manner. We also appreciate all valuable and constructive comments. We attach our added quantitative and qualitative results here.

Meta-review: The reviewers are generally positive about the paper, and some of them increased their score after the rebuttal. Overall, this is a nice piece of work that leverages LLMs for layout generation, a topic that has gained increasing interest in the field. The work proposes and examines two simple but effective techniques to achieve goals such as cross-domain layout generation. The work has demonstrated the applications in both content-aware generation and text-to-layout tasks. Reviewer 7S5o commended that "The two techniques, dynamic prompting and layout ranker, are simple and effective without extra fine-tuning or complex heuristics." Reviewer kjZH noted that "LayoutPrompter is innovative in using large language models for layout generation without fine-tuning. The adaptation of dynamically choosing better demos (Dynamic Prompting) and output verification (Layout Ranker) to this field is a clever utilization of existing techniques." Reviewer mVCb highlighted the same aspect "The strength of this submission lies in its clear motivation, well-written content, and ease of understanding."

The reviewers do have concerns about the evaluation of the techniques, including ablation, datasets and metrics. For example, Reviewer 7S5o mentioned that "I respectfully disagree that current qualitative and quantitative metrics are enough. Since layout design is a professional task, designers can evaluate layouts by considering design rules, aesthetics, styles, usabilities, etc". Reviewer HKER
 is concerned with the stability of the LLMs results. The authors should incorporate your responses to the reviewers in the revision to further strengthen the paper. In addition, the paper seems to miss a few highly relevant works:

PLay: Parametrically Conditioned Layout Generation using Latent Diffusion, CY Cheng, F Huang, G Li, Y Li, ICML 2023

Auto completion of user interface layout design using transformer-based tree decoders, Y Li, J Amelot, X Zhou, S Bengio, S Si, arXiv preprint arXiv:2001.05308
Decision: Accept (poster)