Title: LayoutPrompter: Awaken the Design Ability of Large Language Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: graphic design

graphic layout

large language models

in-context learning
Abstract: Conditional graphic layout generation, which automatically maps user constraints to high-quality layouts, has attracted widespread attention today. Although recent works have achieved promising performance, the lack of versatility and data efficiency hinders their practical applications. In this work, we propose LayoutPrompter, which leverages large language models (LLMs) to address the above problems through in-context learning. LayoutPrompter is made up of three key components, namely input-output serialization, dynamic exemplar selection and layout ranking. Specifically, the input-output serialization component meticulously designs the input and output formats for each layout generation task. Dynamic exemplar selection is responsible for selecting the most helpful prompting exemplars for a given input. And a layout ranker is used to pick the highest quality layout from multiple outputs of LLMs. We conduct experiments on all existing layout generation tasks using four public datasets. Despite the simplicity of our approach, experimental results show that LayoutPrompter can compete with or even outperform state-of-the-art approaches on these tasks without any model training or fine-tuning. This demonstrates the effectiveness of this versatile and training-free approach. In addition, the ablation studies show that LayoutPrompter is significantly superior to the training-based baseline in a low-data regime, further indicating the data efficiency of LayoutPrompter. Our project is available at https://github.com/microsoft/LayoutGeneration/tree/main/LayoutPrompter.

LayoutPrompter: Awaken the Design Ability of Large Language Models

Abstract

Conditional graphic layout generation, which automatically maps user constraints to high-quality layouts, has attracted widespread attention today. Although recent works have achieved promising performance, the lack of versatility and data efficiency hinders their practical applications. In this work, we propose LayoutPrompter, which leverages large language models (LLMs) to address the above problems through in-context learning. LayoutPrompter is made up of three key components, namely input-output serialization, dynamic exemplar selection and layout ranking. Specifically, the input-output serialization component meticulously designs the input and output formats for each layout generation task. Dynamic exemplar selection is responsible for selecting the most helpful prompting exemplars for a given input. And a layout ranker is used to pick the highest quality layout from multiple outputs of LLMs. We conduct experiments on all existing layout generation tasks using four public datasets. Despite the simplicity of our approach, experimental results show that LayoutPrompter can compete with or even outperform state-of-the-art approaches on these tasks without any model training or fine-tuning. This demonstrates the effectiveness of this versatile and training-free approach. In addition, the ablation studies show that LayoutPrompter is significantly superior to the training-based baseline in a low-data regime, further indicating the data efficiency of LayoutPrompter. Our project is available here.

1 Introduction

Layout , which consists of a set of well-arranged graphic elements, plays a critical role in graphic design. To alleviate the workload of designers and allow non-expert users to engage in the design process, numerous studies have delved into the automatic layout generation for diverse user needs [7, 15, 18, 19, 21, 22, 39] (i.e., layout constraints). Based on input layout constraints, existing conditional layout generation tasks can be categorized into the following groups: constraint-explicit layout generation (e.g., generating layouts conditioned on element types), content-aware layout generation , and text-to-layout (see the left side of Figure 1 for constraint examples). Early works in this field [7, 19, 21, 22] primarily focus on individual tasks and develop task-specific model architectures and optimization methods. More recently, task-generic approaches [15, 12, 14] have emerged. Compared to task-specific methods, they achieve greater flexibility and controllability on more tasks, while maintaining the quality of the generated layouts.

Shizhao Sun

Dongmei Zhang

Constraint-Explicit Layout Generation

(a) Document Layout

(b) Poster Layout

(c) Android Layout

(d) Web Layout

Content-Aware Layout Generation

A page introducing the latest

from the ForTheDoers blog. The

page should have a title, and 3

groups to show 3 different posts.

Each group should have one

image, some text boxes to show

the time and headline of the post,

and one button to click to check

the post.

Text-to-Layout

LayoutPrompter

Dynamic Exemplar

Selection

Input-Output

Serialization

LLM

Generation Conditioned on

Types

toolbar

icon

Generation Conditioned on

Types and Sizes

toolbar (90,12)

icon (12,12)

Generation Conditioned on Relationships

toolbar

icon

toolbar top canvas

icon small toolbar

Layout Completion

toolbar (0,5,90,12)

Layout Refinement

toolbar (1,4,88,10)

icon (0,6,14,11)

Generation Conditioned on Types and Sizes

Layout Completion toolbar

Generation Conditioned on Relationships icon

toolbar top canvas icon small toolbar

toolbar (0,5,90,12)

Layout Refinement toolbar (1,4,88,10) icon (0,6,14,11) Although state-of-the-art methods [15, 12, 14, 9, 24] have achieved promising results, they still suffer from some limitations that impede their applications in real-world scenarios. First, the previous approaches struggle to simultaneously cope with all the layout generation tasks depicted in Figure 1 . They are typically tailored for specific tasks and cannot be applied to others. For instance, the state-of-the-art diffusion-based model LayoutDM [14] proposes to inject explicit layout constraints through masking or logit adjustment during inference, but it fails to do so for implicit or vague constraints, e.g., constraints expressed in natural language (i.e., text-to-layout). Consequently, distinct models need to be deployed for different tasks, leading to inconvenience. This motivates us to explore a more versatile approach for layout generation. Second, the existing methods are not data-efficient either. They usually necessitate extensive constraint-layout pair data for model training. For example, LayoutFormer++ [15] relies on the publaynet dataset [40] with a size of 300K for training to generate aesthetically pleasing document layouts. However, collecting such large datasets for some lowresource layout domains (e.g., poster layouts) is prohibitively expensive. Besides, even if such a dataset is available, training is time-consuming and costly. Hence, there is a pressing need to develop a data-efficient layout generation method.

In this work, we consider leveraging the powerful pre-trained large language models (LLMs) to address the above problems. The intuition behind is as follows. First, recent research has shown the versatility of LLMs in various tasks [28, 13, 1, 38]. By carefully designing input-output formats, these tasks can be converted into sequence-to-sequence generation problems and effectively addressed by LLMs. This emerging trend inspires us to utilize LLMs to tackle all conditional layout generation tasks in a unified manner. Second, since the training corpus contains layout source code [28, 4] (e.g., HTML code and XML code), LLMs have acquired some layout-related knowledge during pre-training. For example, they may inherently possess the ability to align graphic elements and avoid unnecessary overlap between them, which is beneficial for producing high-quality and visually appealing layouts. Consequently, an LLMs-based approach holds promise to enhance data efficiency compared to existing models that are trained from scratch. Third, an additional advantage of LLMs lies in their remarkable in-context learning performance [3, 28, 36, 35, 13]. It means that instead of fine-tuning LLMs individually for each layout generation task, we can simply prompt them to perform the desired task with a few input-output demonstrations. This characteristic further allows LLMs to generate layouts in a training-free manner without any parameter updates.

To this end, we propose LayoutPrompter (see Figure 1). It formulates all conditional layout generation tasks as sequence-to-sequence transformation problems and leverages LLMs to tackle them through in-context learning. To unleash the full potential of LLMs for layout generation, two key issues need to be addressed. First, how to awaken the layout-related knowledge in LLMs for achieving decent performance? Second, how to facilitate LLMs understanding diverse user constraints and layout characteristics in distinct domains? LayoutPrompter tackles the two issues with the input-output serialization module and the dynamic exemplar selection module, respectively. I. Input-Output Serialization. Since prevalent LLMs can only read token sequences, this module is responsible for representing user constraints and layouts as sequences so that LLMs can sufficiently exploit their toolbar (90,12)

related knowledge. To represent input layout constraints as sequences, we borrow the successful experience of LayoutFormer++ [15], where they present two simple but effective principles (i.e., constraint representation and constraint combination) to serialize constraints. We experimentally find that the serialization scheme is also effective for LLMs. To represent layouts as sequences, our principle is to convert them into a format resembling what LLMs have encountered during pre-training, thereby leveraging the existing layout-related knowledge within LLMs. Specifically, we serialize the layout into the corresponding source code (e.g., HTML) to obtain the output sequence. II. Dynamic Exemplar Selection. This module is used to select prompting exemplars that have similar layout constraints to the test samples. In contrast to random exemplars, dynamic exemplars ensure that LLMs receive the most relevant context, so they can better comprehend the desired constraints and produce plausible layouts accordingly. To support this technique, we develop an evaluation suite to measure the constraint similarities between a given test sample and all candidate exemplars from the training set. Then, we select those with the highest similarity scores as prompting exemplars. In addition, we introduce a layout ranker to further improve LayoutPrompter's performance. Considering that LLMs can produce distinct outputs through sampling, we generate multiple layouts with the same input constraints, and use the ranker to select the highest-quality one as the final output.

We conduct extensive experiments on various tasks and layout domains to evaluate LayoutPrompter. Experimental results show that LayoutPrompter can tackle all existing conditional layout generation tasks, demonstrating its versatility. Despite without any model training or fine-tuning, LayoutPrompter is on par or even better than the state-of-the-art approaches. Besides, our ablation studies exhibit that LayoutPrompter can still achieve good performance when there is only a small set of candidate exemplars, indicating that it is superior to existing training-based methods in terms of data efficiency. In summary, LayoutPrompter is a versatile, data-efficient and training-free layout generation method.

2 Related Work

Graphic Layout Generation. Automatic graphic layout generation is an emerging research topic in recent years. To meet diverse user requirements, existing methods have defined various layout generation tasks, including layout generation conditioned on element types [21, 19, 18], layout generation conditioned on element types and sizes [19], layout generation conditioned on element relationships [18, 21], layout completion [7, 23] and refinement [30]. In addition to these constraintexplicit tasks, some works consider more challenging but useful tasks, such as content-aware layout generation [39, 9] and text-to-layout [11, 24]. Content-aware layout generation aims at arranging spatial space for pre-defined elements on a given canvas. The generated layouts not only need to be visually pleasing, but also avoid salient areas of the canvas. Text-to-layout is to generate layouts according to human language descriptions.

Early works in this field primarily focus on an individual task and propose task-specific approaches based on Generative Adversarial Networks (GANs) [22, 9], Variational Autoencoders (VAEs) [21, 16] and Transformers [19, 7, 11, 24, 30]. Recently, some general approaches [15, 12, 14, 37] have appeared. LayoutFormer++ [15] proposes to represent various constraints as sequences and then leverages a Transformer [32] encoder-decoder architecture to generate layouts from constraint sequences. [12, 14] develop diffusion-based models for constraint-explicit layout generation. However, none of the existing methods can simultaneously handle all layout generation tasks. Furthermore, these methods are highly dependent on large amounts of training data, which hinders their practical applications. In this work, we introduce techniques such as dynamic exemplar selection, input-output serialization and layout ranking to effectively utilize LLMs to overcome the above limitations, making LayoutPrompter a versatile and data-efficient approach (see Table 1).

Large Language Models. Large language models (LLMs) with billions of parameters, such as GPT [3, 28], PaLM [6] and LLaMa [31], have demonstrated excellent few-shot performance on

EXEMPLAR

Please generate a layout based on the given information.

Task Description

: content-aware layout generation

Layout Domain

: poster layout

Canvas Size

: canvas width is 102px, canvas height is 150px

Content Constraint

: left 26px, top 62px, width 50px, height 69px

Element Type Constraint

: logo | text | text | text | underlay | text

INPUT

CONSTRAINT

<html>

<body>

<div class='logo' style='left: 3px; top: 2px; width: 22px; height: 7px'></div>

<div class='text' style='left: 24px; top: 12px; width: 53px; height: 10px'></div>

<div class='text' style='left: 6px; top: 25px; width: 91px; height: 10px'></div>

<div class='text' style='left: 19px; top: 37px; width: 64px; height: 7px'></div>

<div class='underlay' style='left: 18px; top: 48px; width: 66px; height: 9px'></div>

<div class='text' style='left: 22px; top: 49px; width: 57px; height: 6px'></div>

</body>

</html>

OUTPUT

LAYOUT

EXEMPLARS

Preamble

Exemplar 1

Exemplar N

…

Input

Constraint

Output Layout

Input

Constraint

TEST

INPUT

+

Output Layout

TEST

OUTPUT

LLM

INPUT

OUTPUT

IN-CONTEXT LEARNING

PREAMBLE

Input

Constraint

Output Layout

+

Ranker

✖

L

various natural language processing (NLP) tasks. Thanks to the emergent ability [33] brought by the scale of model and data, they largely outperform prior supervised approaches and even match human-level performance on some tasks, without any finetuning. The versatility and effectiveness of LLMs inspire us to develop a layout generation method based on them.

Recent studies show that the prompting strategy plays a crucial role in model performance. For example, chain-of-thought (CoT) prompting [34] is proposed to improve the reasoning ability of LLMs by incorporating intermediate reasoning steps in the exemplars. Least-to-most prompting [41, 17] (also known as decomposed prompting) is introduced to solve complex multi-step reasoning tasks. To enhance contextual knowledge, [26, 10] use a retrieval module to dynamically select in-context exemplars. They experimentally find that exemplars semantically similar to test samples can better unleash the model's knowledge. Specifically, they use a sentence encoder to convert model inputs to vector representations. Then, for each test sample, they retrieve the nearest neighbors in the encoded sentence embedding space to construct prompts. Motivated by them, we propose a similar prompting strategy in this work. Since the input of layout generation tasks is different from prior works, we introduce a customized evaluation suite to measure sample distances. Experimental results demonstrate its effectiveness in LayoutPrompter.

3 LayoutPrompter

In this section, we elaborate on LayoutPrompter, a versatile, data-efficient and training-free layout generation method built upon LLMs. Our main contribution lies in proposing a set of useful techniques for applying LLMs to layout generation. Specifically, to support sequence-to-sequence transformation and make maximum use of the design knowledge within LLMs, we carefully consider the serialization scheme that represents task inputs and outputs as sequences (Section 3.2). Moreover, to enhance the comprehension of user-specified layout constraints, we propose a dynamic exemplar selection module to retrieve the most relevant exemplars from the training set to perform in-context learning (Section 3.3). Besides, a layout ranker is designed to evaluate layout quality and rank multiple layouts generated under the same constraints, further improving model performance (Section 3.4).

3.1 Overview

Let's consider a conditional layout generation task. We denote its training set as D = { ( x j , y j ) } j M =1 . Here, ( x j , y j ) represents the j -th sample of D , which is an (input constraint, output layout) pair, and M is the total number of samples. As illustrated in Figure 2, for a given test query x test, the in-context learning prompt P is composed by sequentially concatenating a task-specific preamble R ,

N exemplars and the query itself:

To be more specific, the preamble R provides the essential information about the target task, such as the task description , layout domain and canvas size . F X ( · ) and F Y ( · ) are serialization functions that transform task input x and output y into sequences, respectively. G ( · , · ) denotes an exemplar selection function, which retrieves the in-context exemplars from D according to x test. The details of F X , F Y and G will be elaborated in the following sections.

Notably, when the number of exemplars N is set to 0 , few-shot in-context learning degenerates to zero-shot learning, where LLMs predict the test output y test solely based on the preamble R and x test . In our experiments (see Section B.2 in Appendix), we find that additional exemplar guidance can help LLMs better comprehend the task and grasp the rough pattern of the required layouts. Hence, we opt for few-shot learning ( N > 0 ) instead of zero-shot learning ( N = 0 ) in this work. Saliency Detection saliency bbox

(100,0,200,250)

3.2 Input-Output Serialization

Content-Aware Layout Generation

(a)

To begin, we first establish some notations. For each element e that constitutes a layout, we describe it by its element type c , left coordinate l , top coordinate t , width w and height h , i.e., e = ( c, l, t, w, h ) . Here, c is a categorical attribute. The other four are numerical geometric attributes, which will be discretized in the implementation (see Section A in Appendix).

Input Constraint Serialization. For constraint-explicit layout generation, the input constraints are element-wise constraints on e . We serialize such constraints in the same way as LayoutFormer++ [15], where they represent each constraint as a sequence and then combine different constraints through concatenation. For example, if x specifies the element types and sizes, F X ( x ) takes the form of F X ( x ) = " c 1 w 1 h 1 | c 2 w 2 h 2 | . . . " . In this

Output Saliency Map

& Bounding Box

(a)

(b)

work, we adopt these ready-made sequences for constraint-explicit layout generation tasks. Regarding content-aware layout generation, the image nature of the input canvas poses a unique challenge for serialization, i.e., enabling LLMs that can only read text to perceive image content. Inspired by DS-GAN [9], we recognize that the saliency map [8] can well capture the key content shape of a canvas while discarding other high-frequency, irrelevant details (see Figure 3). To facilitate serialization, we further convert it into a rectified saliency map m = ( l m , t m , w m , h m ) by detecting region boundaries with pixel values greater than a certain threshold. After preprocessing, the input canvas x can be represented in a format understandable by LLMs: F X ( x ) = "Content Constraint: left l m px,top t m px,width w m px,height h m px" . For the textto-layout task, where natural language descriptions are used to generate layouts, the constraint sequence is simply the input text itself.

Output Layout Serialization. For the output y , we propose to serialize it into the HTML format that LLMs are more familiar with and good at, rather than the plain sequence used in prior works [15, 7]. Following common HTML representation, we denote the complete output sequence as a concatenation of multiple HTML segments a : F Y ( y ) = [ a 1 ; a 2 ; . . . ] . Here, the i -th segment a i represents the i -th graphic element e i of y . It specifies the element attributes in the following format:

Thanks to the powerful in-context learning ability of LLMs, the test output y test will be predicted in the same HTML format, making it easy to extract the required element attributes from the output. More input-output examples can be found in Section D of the supplementary material.

3.3 Dynamic Exemplar Selection

As mentioned above, G selects N in-context exemplars that have the most similar layout constraints to x test from D . The selected exemplars are randomly shuffled and combined to construct P (see Equation 1), thereby enhancing LLMs' understanding of various constraints. To achieve this, we design an evaluation suite s to measure the constraint similarity between the test query x test and each sentence embeddi

T5

A page for the introduction

The page should have a lo description. The logo…

Text-to-Layout

(b)

candidate exemplar ( x j , y j ) ∈ D . Then, G can be further expressed as a Top-k selection function:

Since we divide existing layout generation tasks into three categories, each with distinct input constraints, their similarity measures have different representations. We'll elaborate below.

Constraint-Explicit Layout Generation. As constraint-explicit layout generation tasks only consider element-wise constraints, we define s ( x test , x j ) using inter-element constraint similarities. Specifically, we construct a bipartite graph between x test = { p u test } U u =1 and x j = { p v j } V v =1 , where p denotes the element-wise constraint on e . U, V are the constraint numbers of x test , x j . Then, the inter-element similarity W (i.e., the weight of bipartite graph) and the overall constraint similarity s are defined as:

Here, 1 is a 0-1 function equal to 1 if p u test and p v j specify the same element type, and 0 otherwise. This ensures that constraint similarity is only considered between elements with the same type. g u test and g v j are specified geometric attributes of p u test and p v j . Given the edge weight W of the bipartite graph, we adopt Hungarian method [20] to obtain the maximum matching M max. And s ( x test , x j ) is calculated as the average weight of matched edges (as shown in Equation 4).

Content-Aware Layout Generation. The constraint of content-aware layout generation is the input canvas. The similarity of two canvases x test , x j is defined as the IoU (Intersection over Union) of their rectified saliency maps (see Section 3.2) m test , m j :

Text-to-Layout. We leverage the CLIP [29] text encoder to encode input texts into embeddings. The constraint similarity s ( x test , x j ) is defined as the cosine similarity of input text embeddings n test , n j :

3.4 Layout Ranker

People usually judge the quality of generated layouts from two perspectives: (1) whether they are visually pleasing; (2) whether they look like the real layouts. Therefore, our proposed layout ranker follows the same principles to evaluate layout quality. To be more specific, it measures the quality of an output layout using a combination of metrics:

RICO

Gen-T

image

text

button

text

×

3

background

×7

Layout

Former++

Layout

Prompter

image

text

button

text

input

×2

Input

constraints

Layout

Former++

Layout

Prompter

Input

constraints

image

icon

text

×2

text

button

×6

toolbar

input

×3

(58,4)

(20,17)

(20,17)

(107,6)

(127,9)

(30,6)

image

text

button

text

×2

(44,24)

(44,8)

(92,5)

(109,7)

Gen-TS

icon

text

×2

text

button

×2

card

text

at the center of

card

text

button

at the bottom of

text

text

button

input

×2

×3

input

the equal size

with other

input

Gen-R

Layout

Former++

Layout

Prompter

Input

constraints

text

×7

title

×2

table

Layout

Former++

Layout

Prompter

Input

constraints

text

×6

title

figure

×2

text

×4

×2

table

(63,1)

(50,40)

(105,3)

(50,40)

(105,6)

(105,45)

text

×6

figure

×3

(50,5)

(50,8)

(50,6)

(50,34)

(50,21)

(105,3)

(46,27)

(47,23)

(98,27)

text

×6

figure

figure

larger than

text

text

at the

bottom of

canvas

text

×4

table

table

at the

center of

canvas

text

at the top of

table

Completion

Refinement

PubLayNet

Here, λ 1 , λ 2 and λ 3 are hyper-parameters to balance the importance of each metric. Alignment and Overlap reflect quality from the perspective (1), while mIoU mainly focuses on perspective (2). We will introduce them in Section 4.1. The output layout with the lowest q value (lower q indicates better quality) is returned as the final output.

4 Experiments

4.1 Setups

Datasets. We conduct experiments on 4 datasets, including RICO [27], PubLayNet [40], PosterLayout [9] and WebUI [24]. Their statistics and usages are illustrated in Table 3. For RICO and PubLayNet, we adopt the same dataset splits as LayoutFormer++ [15]. While for PosterLayout, the training set includes 9,974 poster-layout pairs, and the remaining 905 posters are used for testing. Regarding the WebUI dataset, we adopt the dataset splits provided by parse-then-place [24]. In all cases, the in-context exemplars are retrieved from the full training set.

Baselines. Since constraint-explicit layout generation tasks have task-specific and task-generic methods, we compare LayoutPrompter against both kinds of state-of-the-art methods on these tasks. Concretely, we choose LayoutFormer++ [15] as the common task-generic baseline. The task-specific baselines are (1) BLT [19] for generation conditioned on types (Gen-T), (2) BLT [19] for generation conditioned on types with sizes (Gen-TS), (3) CLG-LO [18] for generation conditioned on relationships (Gen-R), (4) LayoutTransformer [7] for completion, and (5) RUITE [30] for refinement. Moreover, we compare LayoutPrompter with DS-GAN [9] and CGL-GAN [42] on content-aware layout generation. We compare with Mockup [11] and parse-then-place [24] on text-to-layout.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

DS-GAN

Ours

Q7

Evaluation Metrics. To evaluate the performance of LayoutPrompter and baselines, we use the following quantitative metrics. For constraint-explicit layout generation and text-to-layout, we employ four standard metrics. Alignment (Align. ) [22] gauges how well the elements in a layout are aligned with each other. Overlap [22] computes the overlapping area between two arbitrary elements in a layout. Maximum IoU (mIoU ) [18] calculates the highest Intersection over Union (IoU) between a generated layout and real layouts. Fréchet Inception Distance (FID ) [18] measures how similar the distribution of the generated layouts is to that of real layouts. Additionally, we introduce another metric Constraint Violation Rate (Vio. %) [15] to evaluate how well the generated layouts satisfy their input constraints. It is the ratio of violated constraints to all constraints. In text-to-layout, as a textual description may involve the type, position and size constraints of elements, we follow parse-then-place [24] and further break down this metric into Type Vio. % and Pos & Size Vio. % . As for content-aware layout generation, we adopt the eight metrics defined in DS-GAN [9]. Some of them belong to graphic metrics, such as Val , Ove , Ali , Undl and Unds . Others are content-aware metrics, including Uti , Occ and Rea . Please refer to [9] for more details.

Implementation Details. In this work, we conduct experiments on GPT-3 [3] text-davinci-003 model. We place N = 10 exemplars in the prompt P . For each test sample, we generate L = 10 different outputs y test. The hyper-parameters involved in the layout ranker module are set to λ 1 = 0 . 2 , λ 2 = 0 . 2 , and λ 3 = 0 . 6 . When running GPT-3, we fix the parameters to the default values of the OpenAI API, where the sampling temperature is 0.7 and the penalty-related parameters are set to 0.

4.2 Main Results

Tables 2, 4, 5 and Figures 4, 5, 6 show the quantitative and qualitative results on various layout generation tasks (see more qualitative results in Section C of the supplementary material). Although LayoutPrompter has not undergone model training and fine-tuning, the experimental results demonstrate that it can achieve comparable or even better performance than baselines, which proves that LayoutPrompter is a versatile and training-free layout generation approach. Below, we conduct a detailed analysis of the experimental results.

Constraint-Explicit Layout Generation. Table 2 shows the quantitative results. On each constraintexplicit layout generation task, LayoutPrompter is compared with a task-specific method and another common baseline, LayoutFormer++ [15]. Although not trained on these downstream tasks, LayoutPrompter still exhibits competitive quantitative results. Furthermore, it even outperforms the baselines on some metrics (e.g., Align. and Overlap on Gen-T task, RICO dataset). The corresponding qualitative results are shown in Figure 4. Here, we only compare with the state-of-the-art baseline (measured by quantitative metrics), LayoutFormer++. The qualitative comparison indicates

(d) A footer page for an online store selling jewelry. On the page, there should have four lists of links for more info. In each list, there should be a title and 4-6 links.

(e) A page to introduce payments to users. There is a title 'accepted payments' on the top. There are also four logos of payment companies under it.

(f) A page for showing different work types. There should be six groups. Each of them includes one logo and one text which shows the title of the work.

Mockup

PTP

Ours

(a) A page for advertising holiday photo cards. The page should have an image showing the card, a title "Holiday Photo Cards", a further description, and a button "SHOP CARDS" for the user to click to purchase cards.

(b) The page introducing the products to the user. There should be three parts, and each part includes one title, one picture of the product, and one link for the user to click on to view more about the product.

Mockup

PTP

Ours

(c) A page for navigation of Mailreach. The page should have a logo of Mailreach, 5 links for users to check the email spam test, blog, features, pricing, and to log in, and a button for users to click to sign up.

Mockup

PTP

Ours

that LayoutPrompter achieves as good controllability and generation quality as LayoutFormer++. First, the layouts generated by our approach satisfy various input constraints well, including type constraints, size constraints, relationship constraints, etc. Second, our approach can also produce visually pleasing layouts with well-aligned elements and small overlapping areas. Both qualitative and quantitative results demonstrate the effectiveness of LayoutPrompter.

Content-Aware Layout Generation. The quantitative and qualitative results are presented in Table 4 and Figure 5, respectively. Remarkably, LayoutPrompter surpasses the training-based baselines on almost all metrics. This indicates that LayoutPrompter is capable of producing higher-quality and more content-aware layouts compared to the baselines. The rendered results further validate the conclusion. For example, in columns (f) and (g) of Figure 5, the layouts from DS-GAN [9] contain serious misalignments and overlaps. And column (e) shows that DS-GAN sometimes fails to generate content-aware layouts. In contrast, our approach can not only produce aesthetic layouts but also avoid the most salient objects in the input canvas, such as the person, teddy bear, car, etc.

Text-to-Layout. The quantitative and qualitative comparisons are shown in Table 5 and Figure 6. Since text-to-layout is one of the most challenging layout generation tasks, LayoutPrompter slightly lags behind the current state-of-the-art method parse-then-place [24], especially on mIoU and FID metrics. However, on the other four metrics, LayoutPrompter is comparable to the baselines. Thanks to the excellent understanding capability of LLMs, our approach can better satisfy the constraints specified in textual descriptions in some cases. For example, in cases (d) and (e) of Figure 6, LayoutPrompter successfully generates 4-6 links and four logos , while parse-then-place makes wrong predictions about the number of elements.

4.3 Ablation Studies

Effect of Introduced Components. LayoutPrompter has three key components, including inputoutput serialization, dynamic exemplar selection and layout ranking. To investigate their effects, we perform the following ablation studies (see Table 6). (1) Since LayoutFormer++ [15] has proven the effectiveness of constraint sequences relative to other formats, we only study the effect of HTML representation, which is not covered in previous works. Specifically, we replace HTML with a plain sequence proposed by LayoutFormer++ [15] (denoted as w/o HTML) to represent the output layout. This results in a significant drop in FID and overlap metrics on Gen-T. (2) To understand the contribution of dynamic exemplar selection, we compare against its variant (w/o dynamic selection) that adopts random sampling for exemplar retrieval. LayoutPrompter achieves significantly better FID and mIoU across the board. Though the variant has better Align. and Overlap scores in some tasks, its noticeably poor FID and mIoU scores indicate that it fails to acquire the layout patterns in specific domains (e.g., the generated layout does not look like a real UI layout). (3) To understand the

effect of the proposed layout ranker, we compare it against a variant (w/o layout ranker) that randomly picks a layout from model outputs. We find that the layout ranker consistently yields improvements on the mIoU and Align. metrics of all tasks.

Effect of Training Set Size. We switch training set sizes: 500, 2000, 10000 and full set (see Table 7). In our approach, the training set represents the exemplar retrieval pool. The results show that the performance of LayoutFormer++ drops rapidly as the training data decreases, but our method is much slightly affected. When training samples are limited (e.g., 500 and 2000), our approach significantly outperforms the training-based baseline on all metrics. These observations suggest that LayoutPrompter is a more data-efficient approach, which is effective in low-resource scenarios. Due to space limitations, more experimental results on stability, the effect of the number of examples, and generalization ability can be found in Section B of the supplementary material.

5 Conclusion and Limitation

In this work, we concentrate on leveraging Large Language Models (LLMs) for conditional layout generation to address issues present in existing methods. To enhance the performance of our approach, we introduce three crucial components: input-output serialization, dynamic exemplar selection, and layout ranking. We conduct experiments on 7 existing layout generation tasks using 4 public datasets. Both qualitative and quantitative results highlight that LayoutPrompter is a versatile, data-efficient, and training-free method capable of generating high-quality, constraint-compliant layouts. Despite these promising results, there are still some limitations. First, the performance of our approach is influenced by the number of elements in the layouts, with more elements leading to more failure cases. Notably, this is not a problem specific to our approach and has been observed in prior work [2] as well. Second, we have not studied whether LayoutPrompter is equally effective for other LLMs such as PaLM and LLaMa. Third, with the rapid development of large multimodal models such as GPT-4V, PaLI [5] and LLaVA [25], we get a promising chance to extend LayoutPrompter to supporting layout constraints specified in a wide range of modalities. We leave them for future research.

References

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716-23736, 2022.

D. M. Arroyo, J. Postels, and F. Tombari. Variational transformer networks for layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13642-13652, 2021.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code, 2021.

X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer, A. Kolesnikov, J. Puigcerver, N. Ding, K. Rong, H. Akbari, G. Mishra, L. Xue, A. V. Thapliyal, J. Bradbury, W. Kuo, M. Seyedhosseini, C. Jia, B. K. Ayan, C. R. Ruiz, A. P. Steiner, A. Angelova, X. Zhai, N. Houlsby, and R. Soricut. PaLI: A jointly-scaled multilingual language-image model. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=mWVoBz4W0u .

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

K. Gupta, J. Lazarow, A. Achille, L. S. Davis, V. Mahadevan, and A. Shrivastava. Layouttransformer: Layout generation and completion with self-attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1004-1014, 2021.

X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In 2007 IEEE Conference on computer vision and pattern recognition , pages 1-8. Ieee, 2007.

H. Hsu, X. He, Y. Peng, H. Kong, and Q. Zhang. Posterlayout: A new benchmark and approach for content-aware visual-textual presentation layout, 2023.

Y. Hu, C.-H. Lee, T. Xie, T. Yu, N. A. Smith, and M. Ostendorf. In-context learning for few-shot dialogue state tracking. arXiv preprint arXiv:2203.08568 , 2022.

F. Huang, G. Li, X. Zhou, J. F. Canny, and Y. Li. Creating user interface mock-ups from high-level text descriptions with deep-learning models. arXiv preprint arXiv:2110.07775 , 2021.

M. Hui, Z. Zhang, X. Zhang, W. Xie, Y. Wang, and Y. Lu. Unifying layout generation with a decoupled diffusion model, 2023.

S. Imani, L. Du, and H. Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398 , 2023.

N. Inoue, K. Kikuchi, E. Simo-Serra, M. Otani, and K. Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. arXiv preprint arXiv:2303.08137 , 2023.

Z. Jiang, H. Deng, Z. Wu, J. Guo, S. Sun, V. Mijovic, Z. Yang, J.-G. Lou, and D. Zhang. Unilayout: Taming unified sequence-to-sequence transformers for graphic layout generation. arXiv preprint arXiv:2208.08037 , 2022.

A. A. Jyothi, T. Durand, J. He, L. Sigal, and G. Mori. Layoutvae: Stochastic scene layout generation from a label set. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9895-9904, 2019.

T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, and A. Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406 , 2022.

K. Kikuchi, E. Simo-Serra, M. Otani, and K. Yamaguchi. Constrained graphic layout generation via latent optimization. In Proceedings of the 29th ACM International Conference on Multimedia , pages 88-96, 2021.

X. Kong, L. Jiang, H. Chang, H. Zhang, Y. Hao, H. Gong, and I. Essa. Blt: bidirectional layout transformer for controllable layout generation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII , pages 474-490. Springer, 2022.

H. W. Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83-97, 1955.

H.-Y. Lee, L. Jiang, I. Essa, P. B. Le, H. Gong, M.-H. Yang, and W. Yang. Neural design network: Graphic layout generation with constraints. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 491-506. Springer, 2020.

J. Li, J. Yang, J. Zhang, C. Liu, C. Wang, and T. Xu. Attribute-conditioned layout gan for automatic graphic design. IEEE Transactions on Visualization and Computer Graphics , 27(10): 4039-4048, 2020.

Y. Li, J. Amelot, X. Zhou, S. Bengio, and S. Si. Auto completion of user interface layout design using transformer-based tree decoders. arXiv preprint arXiv:2001.05308 , 2020.

J. Lin, J. Guo, S. Sun, W. Xu, T. Liu, J.-G. Lou, and D. Zhang. A parse-then-place approach for generating graphic layouts from textual descriptions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In NeurIPS , 2023.

J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen. What makes good in-context examples for gpt3 ? arXiv preprint arXiv:2101.06804 , 2021.

T. F. Liu, M. Craft, J. Situ, E. Yumer, R. Mech, and R. Kumar. Learning design semantics for mobile apps. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology , UIST '18, page 569-579, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450359481. doi: 10.1145/3242587.3242650. URL https://doi.org/10.1145/3242587.3242650 .

OpenAI. Gpt-4 technical report, 2023.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

S. Rahman, V. P. Sermuga Pandian, and M. Jarke. Ruite: Refining ui layout aesthetics using transformer encoder. In 26th International Conference on Intelligent User Interfaces-Companion , pages 81-83, 2021.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022.

K. Yang, D. Klein, N. Peng, and Y. Tian. Doc: Improving long story coherence with detailed outline control. arXiv preprint arXiv:2212.10077 , 2022.

K. Yang, N. Peng, Y. Tian, and D. Klein. Re3: Generating longer stories with recursive reprompting and revision. arXiv preprint arXiv:2210.06774 , 2022.

J. Zhang, J. Guo, S. Sun, J.-G. Lou, and D. Zhang. Layoutdiffusion: Improving graphic layout generation by discrete diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 7226-7236, October 2023.

Z. Zhang, A. Zhang, M. Li, H. Zhao, G. Karypis, and A. Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923 , 2023.

X. Zheng, X. Qiao, Y. Cao, and R. W. Lau. Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics (TOG) , 38(4):1-15, 2019.

X. Zhong, J. Tang, and A. J. Yepes. Publaynet: largest dataset ever for document layout analysis, 2019.

D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, O. Bousquet, Q. Le, and E. Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 , 2022.

M. Zhou, C. Xu, Y. Ma, T. Ge, Y. Jiang, and W. Xu. Composition-aware graphic layout gan for visual-textual presentation designs. arXiv preprint arXiv:2205.00303 , 2022.

A Coordinate Discretization

In this work, element coordinates are scaled proportionally into a canvas of size C W × C H . We follow the baselines to choose these two parameters. Specifically, in RICO, C W = 90px, C H = 160px. In PubLayNet, C W = 120px, C H = 160px. In PosterLayout, C W = 102px, C H = 150px. In WebUI, C W = 120px, C H = 120px. Then, the coordinates are discretized to the nearest integers.

B Additional Experimental Results and Analysis

B.1 Stability of Generation Performance

The output of LLMs varies with the random seed and hyper-parameters (e.g., the temperature). That is, for the same input constraint, LLMs are able to generate many completely different layouts. Since the hyper-parameters are a trade-off between generation quality and diversity, we fix them to the default values of OpenAI API and study the impact of random seeds on model performance. Specifically, we run inference on the test set 10 times, each using a different random seed. Then, we calculate the mean and variance of each quantitative metric (see Table 8). The small variances indicate the stability of LayoutPrompter's performance under different random seeds.

B.2 Effect of Exemplar Number

We conduct ablation experiments on the number of prompting exemplars. Figure 7 shows the zeroshot ( N = 0 ) qualitative results on Gen-T. It is obvious that LLMs fail to generate reasonable layouts in a zero-shot scheme. Table 9 exhibits the quantitative comparison of the Gen-T task on RICO. The results indicate that the number of prompting exemplars mainly affects mIoU and FID. Specifically, as the number of prompting exemplars increases, mIoU and FID get improved. In summary, the number of exemplars has a positive effect on the performance of LayoutPrompter.

PubLayNet

RICO

B.3 Generalization Ability

To investigate the generalization ability of LayoutPrompter, we compute the DocSim similarity between the generated layouts and their prompting layouts (see Table 10). The DocSim of LayoutFormer++ is computed between the generated layouts and training layouts. The quantitative results show that LayoutPrompter achieves competitive or even better scores compared to LayoutFormer++,

indicating that LayoutPrompter has a close generalization ability to the training-based method. In addition, we exhibit the qualitative results of the generated layouts and their prompting layouts in Figure 8. The results demonstrate that LayoutPrompter is capable of generating meaningful variations different from the prompting ones. Test Layout Retrieved Layouts Test Layout Retrieved Layouts

PubLayNet

PubLayNet

RICO

C Additional Qualitative Results

In this section, we present additional qualitative results of LayoutPrompter. These results further demonstrate the versatility and effectiveness of our method, which can generate high-quality and constraint-compliant layouts on multiple layout generation tasks.

C.1 Generation Conditioned on Element Types

Element Type Constraints

background image, image, text button*5

(1)

(2)

(3)

(4)

(5)

(6)

icon, input, text*8, text button*3, toolbar

icon, list item*6, text*7, toolbar

background image, icon*5, image*3, slider, text*3, toolbar

advertisement, icon*3, image*6, text, toolbar, web view*2

background image, icon, image*2, text*3, text button*3

(a) RICO

Element Type Constraints

(4) figure, text*6, title*2

(1)

(2)

(3)

(4)

(5)

(6)

(1) text*11, title*2

(3) table, text*5

(2) table, text*5, title

(6) figure, text*6, title*2

(5) table, text*6, title*2

(b) PubLayNet

C.2 Generation Conditioned on Element Types and Sizes

(1)

(2)

(3)

(4)

(5)

(6)

Element Type and Size Constraints

icon 12 12, text 29 5, text button 85 10, text button 85 10, toolbar 90 12

input 90 110, text 48 8, text 38 8, text button 28 9, text button 28 9, toolbar 90 16

icon 7 7, image 69 39, text 90 5, text 90 6, text button 83 38

input 53 11, on/off switch 85 11, radio button 17 11, radio button 19 11, text 15 13, text 74 5, text 26 4, text button 15 9, text button 15 8, text button 45 7, text button 45 7

image 71 60, text 81 9, text 76 26, text button 72 16

(a) RICO

Element Type and Size Constraints

(1)

(2)

(3)

(4)

(5)

(6)

figure 90 56, table 99 46, text 99 4, text 32 1, text 99 3, text 47 11, text 47 11

figure 65 74, text 99 13, text 47 16, text 47 39, text 47 23

figure 47 64, table 97 33, text 89 2, text 39 5, text 47 81, text 44 15, text 47 4

table 97 18, text 69 1, text 58 1, text 47 23, text 47 16, text 47 56, text 47 52, text 47 21, text 47 21, title 27 2

figure 101 93, text 49 25, text 49 25, text 46 7, text 47 7

figure 101 44, figure 101 50, text 99 11, text 99 13

C.3 Generation Conditioned on Element Relationships

(1)

(2)

(3)

(4)

(5)

(6)

Element Relationship Constraints

(1) background image, image, text, text button*2, text 2 center canvas, text button 3 center canvas, text 2 smaller background image 0

(2) image, text*4, text button, text 4 bottom image 0, text 3 larger text 2, text button 5 bottom text 4

(3) icon, list item*6, text*7, toolbar, list item 2 top canvas, list item 4 center canvas, list item 6 center canvas, list item 1 bottom icon 0, list item 2 larger icon 0, list item 4 larger icon 0, text 7 right icon 0, toolbar 14 top list item 1, list item 3 equal list item 2, list item 5 bottom list item 2, text 7 top list item 2, text 13 smaller list item 2, text 7 smaller list item 3

(4) input*2, text*3, text button*4, text button 7 bottom canvas, text 2 equal input 0, text 4 top input 0, text button 6 smaller input 0, text 3 smaller input 1, text button 7 smaller input 1, text button 7 bottom input 1, text button 6 larger text 3, text button 6 bottom text 4

(5) image, text*3, text button*3, text 2 top canvas, text 3 bottom image 0, text 3 bottom text 2, text button 5 bottom text 2, text button 5 top text 3

(6) icon*3, image, input*2, text button*3, toolbar, input 5 center canvas, text button 6 center canvas, icon 2 equal icon 1, text button 6 larger icon 1, text button 8 larger icon 1, input 4 bottom image 3, text button 8 smaller image 3, text button 7 bottom input 4, text button 8 larger input 5, text button 8 bottom input 5, text button 8 larger text button 6

(a) RICO

(1)

(2)

(3)

(4)

(5)

(6)

Element Relationship Constraints

(1) table, text*4, text 4 bottom text 1, text 4 smaller text 3

(2) figure, text*3, text 2 smaller figure 0, text 2 bottom figure 0

(3) figure, text*5, text 1 center canvas, text 5 bottom canvas, text 5 bottom text 2

(4) text*13, title*2, title 13 top canvas, text 4 bottom text 0, text 2 bottom text 1, text 4 bottom text 1, text 6 smaller text 1, text 9 larger text 1, title 13 top text 1, title 14 smaller text 1, title 13 smaller text 2, text 8 larger text 3, text 9 bottom text 3, text 6 bottom text 4, text 8 bottom text 4, text 10 larger text 4, text 9 bottom text 6, text 12 larger text 6, title 13 larger text 6, text 10 larger text 8, text 12 smaller text 9, text 12 smaller text 10, title 13 smaller text 10, title 14 top text 10, text 12 smaller text 11

(5) figure, text*5, title, text 5 smaller figure 0, text 5 bottom figure 0, text 5 smaller text 2, title 6 bottom text 3, title 6 center text 5

(6) figure, list*2, text, text 3 bottom figure 0, list 2 right list 1

(b) PubLayNet

C.4 Layout Completion

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

(10)

(11)

(12)

(a) RICO

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

(10)

(11)

(12)

(b) PubLayNet

C.5 Layout Refinement

(1)

(2)

(3)

(4)

(5)

(6)

(a) RICO

(4)

(5)

(6)

(1)

(2)

(3)

(b) PubLayNet

C.6 Content-Aware Layout Generation

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

(10)

(11)

(12)

状元雨

纯牛奶

纯牛奶

VINVTER

NOLKE

C.7 Text-to-Layout

(1)

(2)

(3)

(4)

(5)

(6)

(1) A page to file an insurance claim. There should be two text boxes. One is for the title 'Need to file an insurance claim' and the other is a short description. Two buttons are necessary to click to file or manage the claim.

(2) A footer page showing the homepage of a company. There should be five entries of text links. And the text links should be unfolded. Users can get access to different info through these links.

(3) A page to introduce an online editor. The page should contain an image, and the slogan "Online Infographics, Diagrams, & Charts Maker" to have a brief introduction, and a further description of the specific service of the editor. Then a button is needed for the user to click to learn info.

(4) A page for introducing the home tours. The page should have a title, "home tours". Also, there are four groups and each group has an image and a title.

(5) A page for introducing the service provided by a website. The page should have three groups of information sets: "Web Development" "Social Media", and "Link Building". Each group should include a small image, a title, a further introduction, and a link "READ MORE" for more info by clicking it.

(6) A header page for navigation of an online shop. The page should have an announcement to explain the product is out of stock, a logo, and two links "HOME" and "BUY DITTO HANGERS" for the user to return to the homepage and view products to shop. Also, two buttons for search and shopping cart are needed.

D Prompt Examples

Here, we show a prompt example for each layout generation task, including Gen-T (Table 11), Gen-TS (Table 12), Gen-R (Table 13), completion (Table 14), refinement (Table 15), content-aware layout generation (Table 16), and text-to-layout (Table 17).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a set of few-shot techniques to enable large language models to generate geometric layouts, particularly in the domain of User interfaces, posters and documents. These techniques include dynamic prompting and layout ranker modules, which allows the LLMs to learn from relevant examples in the dataset and filter out badly generated UIs from various samples. The presented technique enables cross-domain, task-generic generation of layouts without the need for training, utilizing existing knowledge in LLMs which was unexplored in prior work.\n\nSTRENGTHS:\n- Enabled LLMs to perform a novel task. This task is also typically challenging for LLMs to perform due to its highly geometric nature.\n\n- Experiments and prompts that lead to the success of layout generation in LLMs would constitute a significant contribution, given the challenging nature of this task in LLM.\n\n- Introduced dynamic prompting and layout ranker, of which their formulation and implementation can be useful for future layout generation work\n\n- Achieved competitive performance in some layout generation tasks, without the need for training\n\nWEAKNESSES:\nSome of these weaknesses are pointed out in the limitations sections by the authors themselves, but I would like to highlight them since I believe they are important for the significance of the current paper:\n\n- **Reliant on existing dataset's coverage of the target task to achieve good performance:** This is shown in the ablation study that the FID is significantly higher without dynamic prompting. This is an important issue in the UI domain where data is difficult and costly to collect.\n\n- **Generalisation ability of LLMs in generating UIs:** Since the dynamic prompting module relies heavily on the provided prompt, it is unclear whether the generated UIs are novel and are meaningful variations for design inspiration. This can be shown by computing the similarity between the generated UIs and the prompt UIs, and comparing this with existing work (which might be generated UIs against the entire dataset for existing techniques that don't use prompts).\n\n- **Diversity and Coverage of the generated UIs:** Related to the above point, it is unclear how much of the entire design space that the generated UIs can cover. Existing work computes the Wasserstein distance between that of the generated UIs and the test set, of which this work did not compute (partially due to the conditional nature of all evaluated tasks). I wonder if the author can perform some unconditional UI generation experiments to explore the knowledge of the overall UI design space that is learned by existing LLMs.\n\nQUESTIONS:\nThese questions are directly relevant to the weaknesses of the paper:\n\n1) What is the average qualitative and quantitative similarity of the generated UIs against the most similar UIs in the prompt? It would be great if the author(s) can use an established metric (e.g., DocSim) and compare this similarity against existing work. Existing work that doesn't use prompt can be compared against the most similar entry in the entire dataset.\n\n2) What is the pool of UIs that the prompt UIs can be sourced from? Is this only from the training set of various domains?\n\n3) How does the performance of the model correlate with the size of the dataset that the prompt UIs can be sourced from? It would be great if the author(s) can perform some ablation experiments along this dimension.\n\n4) Can the author(s) discuss further the significant drop of FIDs w/o dynamic prompting? This discussion could be qualitative.\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose LayoutPrompter, which is a new approach for generating high-quality, cross-domain graphic layouts without any model training or fine-tuning. It can handle a variety of conditional layout generation tasks and outperforms existing methods on some metrics. The approach is competitive with state-of-the-art approaches on five traditional conditional layout generation tasks and even extends to solve two challenging problems that have more flexible constraints.\n\nSTRENGTHS:\nThe strength of this submission lies in its clear motivation, well-written content, and ease of understanding. The authors have effectively articulated the purpose and significance of their research, making the motivation behind the study readily apparent to the reader. The paper is written in a coherent and structured manner, which contributes to its ease of follow throughout. Furthermore, the study impressively conducts solid experiments that encompass a range of layout design settings, providing robust empirical evidence to support their findings. Overall, the submission showcases a strong foundation and execution, making it a valuable contribution to the field.\n\nWEAKNESSES:\n(1) One major weakness of the submission is the lack of detailed information regarding the new dataset, WebUI. While the authors mention its collection, they provide very limited information about the data collection process or the format of the dataset. This omission raises several questions that remain unanswered. It would be beneficial for the authors to include a section in the main paper or supplementary material that thoroughly describes the data collection process, including the sources, selection criteria, and any preprocessing steps applied. Additionally, specifying the format of the dataset (e.g., file type, structure) would provide valuable insights for readers.\n\n(2) Another weakness pertains to the organization and clarity of the dataset setup explanation. As the submission covers multiple tasks and settings, it would greatly enhance readability if the authors were more specific about which dataset was used for each setting. Currently, the explanation of the dataset setup (lines 227-236) appears disorganized and challenging to follow. (I searched for the experimental results on WebUI and found nothing, then it took me a while to figure out that WebUI was referred to as text-to-layout…) To address this, I recommend the authors consider incorporating a table that lists the tasks and corresponding datasets. This table would help readers quickly identify the datasets utilized for each specific setting, reducing confusion and facilitating a more straightforward interpretation of the experimental results.\n\nQUESTIONS:\n(1) Figure 1 is a bit hard to understand as a teaser figure. It took me a while to figure out that the left parts refer to the “ five typical tasks and two more challenging tasks”, while the right parts are the subdomains of discussion…\n\n(2) Missing reference for saliency map on line 161\n\n(3) Are there any ablation studies on the effect of the number of prompting examples?\n\n(4) Regarding the new dataset: \n- What is the source of the website? \n- What kind of HTML elements are included? \n- What is the portion of each element? \n- How was the quality controlled (e.g., removing ads, removing unrelated banner, etc)? \n- How was the instructions for layout design collected? \n- Does the instruction creation process involve human effort or was it automatically generated?\n- Where is the data sheet for this new dataset?\n\n(5) Out of curiosity, may I ask why the authors did not conduct any human evaluation?\n\nLIMITATIONS:\nThe authors have included discussions of the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce LayoutPrompter, a novel approach for conditional graphic layout generation using\nlarge language models with minimal demonstrations.\n\nThe authors frame the graphic layout generation task as a sequence generation problem. They represent layouts as HTML\nfiles, leveraging the fact that large language models are pretrained on a corpus that frequently includes HTML. A unique\naspect of this approach is the use of a Dynamic Prompting strategy. In this strategy, the model retrieves exemplars that are\nmore closely related to each test sample based on a defined constraint similarity metric. This ensures that the in-context\nexamples used for prompting are highly relevant. Furthermore, a Layout Ranker is appended to the model, which selects\nthe optimal layout post-inference. Notably, the method does not require extensive training data and demonstrates\nadaptability across various layout domains.\n\nTo validate the efficacy of LayoutPrompter, the authors carry out experiments across different layout domains. The\nperformance of LayoutPrompter is benchmarked against various task-specific and task-agnostic baselines. Remarkably,\nLayoutPrompter yields competitive results even without domain-specific training or fine-tuning. It is worth noting that\nLayoutPrompter is particularly effective on a custom WebUI dataset, which is constructed to investigate a new text-to-layout\ngeneration task. This outperformance is more pronounced when compared to results on other layout domains and\nagainst other baselines.\n\nSTRENGTHS:\nOriginality: LayoutPrompter is innovative in using large language models for layout generation without fine-tuning. The\nadaptation of dynamically choosing better demos (Dynamic Prompting) and output verification (Layout Ranker) to this\nfield is a clever utilization of existing techniques.\n\nQuality: The method's solid integration of Sequence Generation, HTML representation, Dynamic Prompting, and Layout\nRanker underpins its robustness. Comprehensive benchmarks add credibility to the research.\n\nClarity: The paper is well-structured, with clear explanations of methodology and results, making it accessible and easy to\nfollow.\n\nSignificance: LayoutPrompter's ability to generalize without extensive training data is noteworthy, indicating the potential for\ndiverse applications and research in layout generation.\n\nWEAKNESSES:\nLimited Novelty in Techniques: The application of large language models for layout generation is noteworthy, but the utilization of\nHTML Representation, Dynamic Prompting, and Layout Ranker lack innovation. It would be interesting to see the exploration or\nintroduction of new methodologies tailored for this domain.\n\nInconclusive Ablation Study: The ablation study on the Dynamic Prompting and Layout Ranker does not convincingly demonstrate\nthe effectiveness of these two techniques.\n\nInsufficient Experimental Analysis: While the paper presents experimental results, it lacks in-depth explanations and analyses.\nElaborating on the implications of these results, and providing a more detailed interpretation, would add substantial value.\nIn-depth Analysis of Failures: The paper could provide a more detailed analysis of cases where LayoutPrompter fails or\nunderperforms. Understanding the limitations and areas where the model struggles would offer insights for future research and\nimprovements.\n\nWriting Clarity and Organization: The paper could improve in terms of clarity and organization. For instance, in discussing the\nmotivation for bridging the gap between existing methods and human designers, the text lacks coherence and the last two points seem\nto overlap. Clearer and more structured presentation of concepts would enhance the paper’s readability.\n\nQUESTIONS:\nAblation Study Details: Could the authors elaborate on the methodology used for the ablation study, and provide insights into why\nDynamic Prompting and Layout Ranker might not have shown significant effectiveness. Would considering different variations of\nthese techniques lead to different outcomes?\n\nDetailed Experimental Analysis: Could the authors provide a more detailed analysis of the experimental results? Specifically, why\nLayoutPrompter performs better on certain metrics compared to others. Moreover, what are the factors that contribute to its varied\nperformance across different domains?\n\nGeneralization Across Domains: The paper claims that the method can be generalized across domains. Can the authors provide\nadditional empirical evidence or examples to substantiate this claim? Specifically, how will the model perform in a zero-shot scheme or\ngiven demos in different domains?\n\nHandling of Failures and Underperformance: It would be helpful if the authors could elaborate on the specific cases where\nLayoutPrompter fails or underperforms. What are the common characteristics of these cases and are there any strategies that could be\nemployed to mitigate these issues?\n\nClarity and Organization of Writing: There are instances where the writing is not very clear or well-organized, as observed in the\nexplanation of closing the gap between existing methods and human designers. Could the authors clarify the intended message and\npossibly consider restructuring this section for better clarity?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper leverages large language models to solve the layout generation tasks. It shows a great performance when generalizing to other domains or other tasks. Since in-context learning is used, there is no need to train the model on a vast amount of data.

STRENGTHS:
1. The prompting method shows a good performance and does not need large amount of training data.
2. The new method can well generalize to new domains and new tasks without an additional model

WEAKNESSES:
1. The paper used text-davinci-003 as LLM, but it is also worthwhile to see how other LLMs perform with the proposed prompt-based method. Can these prompts perform well on other LLMs, including the in-house or open-sourced ones.
2. The in-context learning is not stable with LLMs. The sample prompt may have different results depending on the random seed and temperature parameters. The paper didn't report the variance of the results. 
3. In line 258, it is also necessary to mention the parameters for running LLM to repeat the experiments.

QUESTIONS:
See weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new conditional layout generation method using LLMs with few-shot learning. The method uses a pre-trained GPT-3 with two techniques: dynamic prompting and layout ranker. Dynamic prompting is to provide few-shot examples with similar constraints to the input query using ConsSim, and layout ranker is to rank the layout using weighted metrics. The authors show that this method can generate layouts that comply with different types of conditions across domains, and the qualitative and quantitative results show that the method is comparable and sometimes better than existing training-based, domain-specific methods on various metrics.\n\nSTRENGTHS:\n1. The proposed method addressed the problems of existing methods: reducing the training cost using few-shot learning and providing abilities across domains and tasks by properly designed input/output formats and pre-trained LLMs.\n\n2. In addition to generic layout conditions, the proposed method can be applied to two more challenging tasks: content-aware generation and text-to-layout. \n\n3. The two techniques, dynamic prompting and layout ranker, are simple and effective without extra fine-tuning or complex heuristics.\n\n4. The authors demonstrate the possibility of leveraging LLMs for design-related tasks.\n\nWEAKNESSES:\n1. It will be great to see more comparisons and discussions on content-aware layout generation. For example, some prior works, such as CanvasVAE [1] and CGL-GAN [2], are missing here. \n\n2. It would be great if the authors could examine if the retrieved examples are similar to the given test set layout. For instance, in RICO, multiple layouts might be extracted from the same app, so they might look very similar and provide a way for the model to cheat. \n\n3. Ablation studies can be improved by adding, for example, zero-shot results, results of different ways to represent the constraints, results of varying screen resolutions, results in groups with different numbers of elements, etc.  \n\n4. It would be great if the authors could conduct a user study, ideally on designers, since none of these metrics can perfectly capture human perceptions/judgments.\n\n[1] CanvasVAE: Learning to Generate Vector Graphic Documents, Kota Yamaguchi, ICCV 2021\n[2] Composition-aware Graphic Layout GAN for Visual-textual Presentation Designs, Zhou et al., IJCAI 2022\n\nQUESTIONS:\n1. The Gen-T and Gen-TS results in RICO in Fig 5 look very similar, and I wondered if the authors intentionally chose them for comparison. Otherwise, it seems impossible since the conditions are generic and have many possible solutions. \n\n2. In the ablation studies (Table 5), the Gen-T results w/o HTML achieve 0% constraint violations, and the proposed method performs the worst. Are there possible reasons? The other tasks do not have results w/o HTML, and I wondered if the same situation can be observed. \n\n3. It would be interesting to see the results in the refinement task with different perturbation levels.\n\n4. Have the authors tried fine-tuning LLMs on these tasks?\n\nLIMITATIONS:\nYes, the authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}