SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiawei Lin*
Xi'an Jiaotong University kylelin@stu.xjtu.edu.cn
Jiaqi Guo
Microsoft jiaqiguo@microsoft.com
Shizhao Sun
Microsoft shizsu@microsoft.com
Zijiang James Yang
Xi'an Jiaotong University zijiang@xjtu.edu.cn
Jian-Guang Lou
Microsoft  
jlou@microsoft.com
Microsoft dongmeiz@microsoft.com
Conditional graphic layout generation, which automatically maps user constraints to high-quality layouts, has attracted widespread attention today. Although recent works have achieved promising performance, the lack of versatility and data efficiency hinders their practical applications. In this work, we propose LayoutPrompter, which leverages large language models (LLMs) to address the above problems through in-context learning. LayoutPrompter is made up of three key components, namely input-output serialization, dynamic exemplar selection and layout ranking. Specifically, the input-output serialization component meticulously designs the input and output formats for each layout generation task. Dynamic exemplar selection is responsible for selecting the most helpful prompting exemplars for a given input. And a layout ranker is used to pick the highest quality layout from multiple outputs of LLMs. We conduct experiments on all existing layout generation tasks using four public datasets. Despite the simplicity of our approach, experimental results show that LayoutPrompter can compete with or even outperform state-of-the-art approaches on these tasks without any model training or fine-tuning. This demonstrates the effectiveness of this versatile and training-free approach. In addition, the ablation studies show that LayoutPrompter is significantly superior to the training-based baseline in a low-data regime, further indicating the data efficiency of LayoutPrompter. Our project is available here.
Layout, which consists of a set of well-arranged graphic elements, plays a critical role in graphic design. To alleviate the workload of designers and allow non-expert users to engage in the design process, numerous studies have delved into the automatic layout generation for diverse user needs [7, 15, 18, 19, 21, 22, 39] (i.e., layout constraints). Based on input layout constraints, existing conditional layout generation tasks can be categorized into the following groups: constraint-explicit layout generation (e.g., generating layouts conditioned on element types), content-aware layout generation, and text-to layout (see the left side of Figure 1 for constraint examples). Early works in this field [7, 19, 21, 22] primarily focus on individual tasks and develop task-specific model architectures and optimization methods. More recently, task-generic approaches [15, 12, 14] have emerged. Compared to task-specific methods, they achieve greater flexibility and controllability on more tasks, while maintaining the quality of the generated layouts.
Figure 1: LayoutPrompter is a versatile method for graphic layout generation, capable of solving various conditional layout generation tasks (as illustrated on the left side) across a range of layout domains (as illustrated on the right side) without any model training or fine-tuning.
Although state-of-the-art methods [15, 12, 14, 9, 24] have achieved promising results, they still suffer from some limitations that impede their applications in real-world scenarios. First, the previous approaches struggle to simultaneously cope with all the layout generation tasks depicted in Figure 1. They are typically tailored for specific tasks and cannot be applied to others. For instance, the state-of-the-art diffusion-based model LayoutDM [14] proposes to inject explicit layout constraints through masking or logit adjustment during inference, but it fails to do so for implicit or vague constraints, e.g., constraints expressed in natural language (i.e., text-to/layout). Consequently, distinct models need to be deployed for different tasks, leading to inconvenience. This motivates us to explore a more versatile approach for layout generation. Second, the existing methods are not data-efficient either. They usually necessitate extensive constraint-Layout pair data for model training. For example, LayoutFormer++ [15] relies on the publaynet dataset [40] with a size of  $300\mathrm{K}$  for training to generate aesthetically pleasing document layouts. However, collecting such large datasets for some low-resource layout domains (e.g., poster layouts) is prohibitively expensive. Besides, even if such a dataset is available, training is time-consuming and costly. Hence, there is a pressing need to develop a data-efficient layout generation method.
In this work, we consider leveraging the powerful pre-trained large language models (LLMs) to address the above problems. The intuition behind is as follows. First, recent research has shown the versatility of LLMs in various tasks [28, 13, 1, 38]. By carefully designing input-output formats, these tasks can be converted into sequence-to-sequence generation problems and effectively addressed by LLMs. This emerging trend inspires us to utilize LLMs to tackle all conditional layout generation tasks in a unified manner. Second, since the training corpus contains layout source code [28, 4] (e.g., HTML code and XML code), LLMs have acquired some layout-related knowledge during pre-training. For example, they may inherently possess the ability to align graphic elements and avoid unnecessary overlap between them, which is beneficial for producing high-quality and visually appealing layouts. Consequently, an LLMs-based approach holds promise to enhance data efficiency compared to existing models that are trained from scratch. Third, an additional advantage of LLMs lies in their remarkable in-context learning performance [3, 28, 36, 35, 13]. It means that instead of fine-tuning LLMs individually for each layout generation task, we can simply prompt them to perform the desired task with a few input-output demonstrations. This characteristic further allows LLMs to generate layouts in a training-free manner without any parameter updates.
To this end, we propose LayoutPrompter (see Figure 1). It formulates all conditional layout generation tasks as sequence-to-sequence transformation problems and leverages LLMs to tackle them through in-context learning. To unleash the full potential of LLMs for layout generation, two key issues need to be addressed. First, how to awaken the layout-related knowledge in LLMs for achieving decent performance? Second, how to facilitate LLMs understanding diverse user constraints and layout characteristics in distinct domains? LayoutPrompter tackles the two issues with the input-output serialization module and the dynamic exemplar selection module, respectively. I. Input-Output Localization. Since prevalent LLMs can only read token sequences, this module is responsible for representing user constraints and layouts as sequences so that LLMs can sufficiently exploit their
Methods	Versatile	Data-Efficient	Training-Free
LayoutTransformer [7], BLT [19], and so on [9, 24, 16, 30]	×	×	×
LayoutFormer++ [15], LayoutDM [14], LGDM [12]	partially	×	×
LayoutPrompter (ours)	✓	✓	✓
Table 1: A comparison between existing conditional layout generation methods and LayoutPrompter.
related knowledge. To represent input layout constraints as sequences, we borrow the successful experience of LayoutFormer++ [15], where they present two simple but effective principles (i.e., constraint representation and constraint combination) to serialize constraints. We experimentally find that the serialization scheme is also effective for LLMs. To represent layouts as sequences, our principle is to convert them into a format resembling what LLMs have encountered during pre-training, thereby leveraging the existing layout-related knowledge within LLMs. Specifically, we serialize the layout into the corresponding source code (e.g., HTML) to obtain the output sequence. II. Dynamic Exemplar Selection. This module is used to select prompting exemplars that have similar layout constraints to the test samples. In contrast to random exemplars, dynamic exemplars ensure that LLMs receive the most relevant context, so they can better comprehend the desired constraints and produce plausible layouts accordingly. To support this technique, we develop an evaluation suite to measure the constraint similarities between a given test sample and all candidate exemplars from the training set. Then, we select those with the highest similarity scores as prompting exemplars. In addition, we introduce a layout ranker to further improve LayoutPrompter's performance. Considering that LLMs can produce distinct outputs through sampling, we generate multiple layouts with the same input constraints, and use the ranker to select the highest-quality one as the final output.
We conduct extensive experiments on various tasks and layout domains to evaluate LayoutPrompter. Experimental results show that LayoutPrompter can tackle all existing conditional layout generation tasks, demonstrating its versatility. Despite without any model training or fine-tuning, LayoutPrompter is on par or even better than the state-of-the-art approaches. Besides, our ablation studies exhibit that LayoutPrompter can still achieve good performance when there is only a small set of candidate exemplars, indicating that it is superior to existing training-based methods in terms of data efficiency. In summary, LayoutPrompter is a versatile, data-efficient and training-free layout generation method.
Graphic Layout Generation. Automatic graphic layout generation is an emerging research topic in recent years. To meet diverse user requirements, existing methods have defined various layout generation tasks, including layout generation conditioned on element types [21, 19, 18], layout generation conditioned on element types and sizes [19], layout generation conditioned on element relationships [18, 21], layout completion [7, 23] and refinement [30]. In addition to these constraint-explicit tasks, some works consider more challenging but useful tasks, such as content-aware layout generation [39, 9] and text-to-query [11, 24]. Content-aware layout generation aims at arranging spatial space for pre-defined elements on a given canvas. The generated layouts not only need to be visually pleasing, but also avoid salient areas of the canvas. Text-to-query is to generate layouts according to human language descriptions.
Early works in this field primarily focus on an individual task and propose task-specific approaches based on Generative Adversarial Networks (GANs) [22, 9], Variational Autoencoders (VAEs) [21, 16] and Transformers [19, 7, 11, 24, 30]. Recently, some general approaches [15, 12, 14, 37] have appeared. LayoutFormer++ [15] proposes to represent various constraints as sequences and then leverages a Transformer [32] encoder-decoder architecture to generate layouts from constraint sequences. [12, 14] develop diffusion-based models for constraint-explicit layout generation. However, none of the existing methods can simultaneously handle all layout generation tasks. Furthermore, these methods are highly dependent on large amounts of training data, which hinders their practical applications. In this work, we introduce techniques such as dynamic exemplar selection, input-output serialization and layout ranking to effectively utilize LLMs to overcome the above limitations, making LayoutPrompter a versatile and data-efficient approach (see Table 1).
Large Language Models. Large language models (LLMs) with billions of parameters, such as GPT [3, 28], PaLM [6] and LLaMa [31], have demonstrated excellent few-shot performance on
Figure 2: An overview of LayoutPrompter. The complete prompt consists of a task-specific preamble,  $N$  in-context exemplars and a test input. The exemplars are dynamically retrieved from the training set according to the test input. Subsequently, the prompt is fed into an LLM to generate  $L$  distinct layouts. We employ a layout ranker to select the best one as the final output.
various natural language processing (NLP) tasks. Thanks to the emergent ability [33] brought by the scale of model and data, they largely outperform prior supervised approaches and even match human-level performance on some tasks, without any finetuning. The versatility and effectiveness of LLMs inspire us to develop a layout generation method based on them.
Recent studies show that the prompting strategy plays a crucial role in model performance. For example, chain-of-thought (CoT) prompting [34] is proposed to improve the reasoning ability of LLMs by incorporating intermediate reasoning steps in the exemplars. Least-to-most prompting [41, 17] (also known as decomposed prompting) is introduced to solve complex multi-step reasoning tasks. To enhance contextual knowledge, [26, 10] use a retrieval module to dynamically select in-context exemplars. They experimentally find that exemplars semantically similar to test samples can better unleash the model's knowledge. Specifically, they use a sentence encoder to convert model inputs to vector representations. Then, for each test sample, they retrieve the nearest neighbors in the encoded sentence embedding space to construct prompts. Motivated by them, we propose a similar prompting strategy in this work. Since the input of layout generation tasks is different from prior works, we introduce a customized evaluation suite to measure sample distances. Experimental results demonstrate its effectiveness in LayoutPrompter.
3 LayoutPrompter
In this section, we elaborate on LayoutPrompter, a versatile, data-efficient and training-free layout generation method built upon LLMs. Our main contribution lies in proposing a set of useful techniques for applying LLMs to layout generation. Specifically, to support sequence-to-sequence transformation and make maximum use of the design knowledge within LLMs, we carefully consider the serialization scheme that represents task inputs and outputs as sequences (Section 3.2). Moreover, to enhance the comprehension of user-specified layout constraints, we propose a dynamic exemplar selection module to retrieve the most relevant exemplars from the training set to perform in-context learning (Section 3.3). Besides, a layout ranker is designed to evaluate layout quality and rank multiple layouts generated under the same constraints, further improving model performance (Section 3.4).
Let's consider a conditional layout generation task. We denote its training set as  $\mathcal{D} = \{(x_j, y_j)\}_{j=1}^M$ . Here,  $(x_j, y_j)$  represents the  $j$ -th sample of  $\mathcal{D}$ , which is an (input constraint, output layout) pair, and  $M$  is the total number of samples. As illustrated in Figure 2, for a given test query  $x_{\mathrm{test}}$ , the in-context learning prompt  $\mathbf{P}$  is composed by sequentially concatenating a task-specific preamble  $R$ ,
$N$  exemplars and the query itself:
$$
\mathbf {P} = \left[ R; F _ {X} \left(x _ {k _ {1}}\right); F _ {Y} \left(y _ {k _ {1}}\right); \dots ; F _ {X} \left(x _ {k _ {N}}\right); F _ {Y} \left(y _ {k _ {N}}\right); F _ {X} \left(x _ {\text {t e s t}}\right) \right], \quad \left\{k _ {i} \right\} _ {i = 1} ^ {N} = G \left(x _ {\text {t e s t}}, \mathcal {D}\right). \tag {1}
$$
To be more specific, the preamble  $R$  provides the essential information about the target task, such as the task description, layout domain and canvas size.  $F_{X}(\cdot)$  and  $F_{Y}(\cdot)$  are serialization functions that transform task input  $x$  and output  $y$  into sequences, respectively.  $G(\cdot,\cdot)$  denotes an exemplar selection function, which retrieves the in-context exemplars from  $\mathcal{D}$  according to  $x_{\mathrm{test}}$ . The details of  $F_{X}, F_{Y}$  and  $G$  will be elaborated in the following sections.
Notably, when the number of exemplars  $N$  is set to 0, few-shot in-context learning degenerates to zero-shot learning, where LLMs predict the test output  $y_{\mathrm{test}}$  solely based on the preamble  $R$  and  $x_{\mathrm{test}}$ . In our experiments (see Section B.2 in Appendix), we find that additional exemplar guidance can help LLMs better comprehend the task and grasp the rough pattern of the required layouts. Hence, we opt for few-shot learning ( $N > 0$ ) instead of zero-shot learning ( $N = 0$ ) in this work.
3.2 Input-Output Compilation
To begin, we first establish some notations. For each element  $e$  that constitutes a layout, we describe it by its element type  $c$ , left coordinate  $l$ , top coordinate  $t$ , width  $w$  and height  $h$ , i.e.,  $e = (c, l, t, w, h)$ . Here,  $c$  is a categorical attribute. The other four are numerical geometric attributes, which will be discretized in the implementation (see Section A in Appendix).
Input Constraint serialization. For constraint-explicit layout generation, the input constraints are element-wise constraints on  $e$ . We serialize such constraints in the same way as LayoutFormer++ [15], where they represent each constraint as a sequence and then combine different constraints through concatenation. For example, if  $x$  specifies the element types and sizes,  $F_{X}(x)$  takes the form of  $F_{X}(x) = "c_{1}w_{1}h_{1}|c_{2}w_{2}h_{2}|..."$ . In this
Figure 3: An input canvas is converted into a saliency map.
work, we adopt these ready-made sequences for constraint-explicit layout generation tasks. Regarding content-aware layout generation, the image nature of the input canvas poses a unique challenge for serialization, i.e., enabling LLMs that can only read text to perceive image content. Inspired by DS-GAN [9], we recognize that the saliency map [8] can well capture the key content shape of a canvas while discarding other high-frequency, irrelevant details (see Figure 3). To facilitate serialization, we further convert it into a rectified saliency map  $m = (l_m, t_m, w_m, h_m)$  by detecting region boundaries with pixel values greater than a certain threshold. After preprocessing, the input canvas  $x$  can be represented in a format understandable by LLMs:  $F_X(x) =$  "Content Constraint: left  $l_m$ px, top  $t_m$ px, width  $w_m$ px, height  $h_m$ px". For the text-to-Layout task, where natural language descriptions are used to generate layouts, the constraint sequence is simply the input text itself.
Output Layout Compilation. For the output  $y$ , we propose to serialize it into the HTML format that LLMs are more familiar with and good at, rather than the plain sequence used in prior works [15, 7]. Following common HTML representation, we denote the complete output sequence as a concatenation of multiple HTML segments  $a$ :  $F_{Y}(y) = [a_{1};a_{2};\ldots]$ . Here, the  $i$ -th segment  $a_{i}$  represents the  $i$ -th graphic element  $e_{i}$  of  $y$ . It specifies the element attributes in the following format:
$$
<   \text {d i v} \quad \text {c l a s s} = ^ {\prime \prime} c _ {i} ^ {\prime \prime} \quad \text {s t y l e} = ^ {\prime \prime} \text {l e f t}: l _ {i} \mathrm {p x}; \text {t o p}: t _ {i} \mathrm {p x}; \text {w i d t h}: w _ {i} \mathrm {p x}; \text {h e i g h t}: h _ {i} \mathrm {p x} > <   / \text {d i v} >. \tag {2}
$$
Thanks to the powerful in-context learning ability of LLMs, the test output  $y_{\mathrm{test}}$  will be predicted in the same HTML format, making it easy to extract the required element attributes from the output. More input-output examples can be found in Section D of the supplementary material.
3.3 Dynamic Exemplar Selection
As mentioned above,  $G$  selects  $N$  in-context exemplars that have the most similar layout constraints to  $x_{\mathrm{test}}$  from  $\mathcal{D}$ . The selected exemplars are randomly shuffled and combined to construct  $\mathbf{P}$  (see Equation 1), thereby enhancing LLMs' understanding of various constraints. To achieve this, we design an evaluation suite  $s$  to measure the constraint similarity between the test query  $x_{\mathrm{test}}$  and each
Tasks	Methods	RICO	PubLayNet
mIoU ↑	FID ↓	Align. ↓	Overlap ↓	Vio. % ↓	mIoU ↑	FID ↓	Align. ↓	Overlap ↓	Vio. % ↓
Gen-T	BLT	0.216	25.633	0.150	0.983	-	0.140	38.684	0.036	0.196	-
LayoutFormer++	0.432	1.096	0.230	0.530	0.	0.348	8.411	0.020	0.008	0.
LayoutPrompter	0.429	3.233	0.109	0.505	0.64	0.382	3.022	0.037	0.047	0.50
Gen-TS	BLT	0.604	0.951	0.181	0.660	0.	0.428	7.914	0.021	0.419	0.
LayoutFormer++	0.620	0.757	0.202	0.542	0.	0.471	0.720	0.024	0.037	0.
LayoutPrompter	0.552	1.458	0.145	0.544	0.18	0.453	1.067	0.049	0.091	0.
Gen-R	CLG-LO	0.286	8.898	0.311	0.615	3.66	0.277	19.738	0.123	0.200	6.66
LayoutFormer++	0.424	5.972	0.332	0.537	11.84	0.353	4.954	0.025	0.076	3.9
LayoutPrompter	0.400	5.178	0.101	0.564	10.58	0.347	3.620	0.037	0.161	12.29
Completion	LayoutTransformer	0.363	6.679	0.194	0.478	-	0.077	14.769	0.019	0.0013	-
LayoutFormer++	0.732	4.574	0.077	0.487	-	0.471	10.251	0.020	0.0022	-
LayoutPrompter	0.667	7.318	0.084	0.428	-	0.476	2.132	0.023	0.017	-
Refinement	RUIT	0.811	0.107	0.133	0.483	-	0.781	0.061	0.029	0.020	-
LayoutFormer++	0.816	0.032	0.123	0.489	-	0.785	0.086	0.024	0.006	-
LayoutPrompter	0.745	0.978	0.159	0.478	-	0.647	0.278	0.072	0.048	-
Table 2: Quantitative comparison with baselines on constraint-explicit layout generation tasks.  $\uparrow$  indicates larger values are better,  $\downarrow$  indicates smaller values are better.
candidate exemplar  $(x_{j},y_{j})\in \mathcal{D}$ . Then,  $G$  can be further expressed as a Top-k selection function:
$$
G \left(x _ {\text {t e s t}}, \mathcal {D}\right) \triangleq \operatorname {T o p - k} \left(\bigcup_ {\left(x _ {j}, y _ {j}\right) \in \mathcal {D}} \left\{s \left(x _ {\text {t e s t}}, x _ {j}\right) \right\}, N\right). \tag {3}
$$
Since we divide existing layout generation tasks into three categories, each with distinct input constraints, their similarity measures have different representations. We'll elaborate below.
Constraint-Explicit Layout Generation. As constraint-explicit layout generation tasks only consider element-wise constraints, we define  $s(x_{\mathrm{test}}, x_j)$  using inter-element constraint similarities. Specifically, we construct a bipartite graph between  $x_{\mathrm{test}} = \{p_{\mathrm{test}}^u\}_{u=1}^U$  and  $x_j = \{p_j^v\}_{v=1}^V$ , where  $p$  denotes the element-wise constraint on  $e$ .  $U, V$  are the constraint numbers of  $x_{\mathrm{test}}, x_j$ . Then, the inter-element similarity  $W$  (i.e., the weight of bipartite graph) and the overall constraint similarity  $s$  are defined as:
$$
s \left(x _ {\text {t e s t}}, x _ {j}\right) \triangleq \frac {1}{\left| \mathbb {M} _ {\max } \right|} \sum_ {\left(p _ {\text {t e s t}} ^ {u}, p _ {j} ^ {v}\right) \in \mathbb {M} _ {\max }} W \left(p _ {\text {t e s t}} ^ {u}, p _ {j} ^ {v}\right), \quad W \left(p _ {\text {t e s t}} ^ {u}, p _ {j} ^ {v}\right) = \mathbb {1} \left(p _ {\text {t e s t}} ^ {u}, p _ {j} ^ {v}\right) 2 ^ {- \left\| \mathbf {g} _ {\text {t e s t}} ^ {u} - \mathbf {g} _ {j} ^ {v} \right\| _ {2}}. \tag {4}
$$
Here,  $\mathbb{1}$  is a 0-1 function equal to 1 if  $p_{\mathrm{test}}^u$  and  $p_j^v$  specify the same element type, and 0 otherwise. This ensures that constraint similarity is only considered between elements with the same type.  $\mathbf{g}_{\mathrm{test}}^u$  and  $\mathbf{g}_j^v$  are specified geometric attributes of  $p_{\mathrm{test}}^u$  and  $p_j^v$ . Given the edge weight  $W$  of the bipartite graph, we adopt Hungarian method [20] to obtain the maximum matching  $\mathbb{M}_{\max}$ . And  $s(x_{\mathrm{test}},x_j)$  is calculated as the average weight of matched edges (as shown in Equation 4).
Content-Aware Layout Generation. The constraint of content-aware layout generation is the input canvas. The similarity of two canvases  $x_{\mathrm{test}}$ ,  $x_{j}$  is defined as the IoU (Intersection over Union) of their rectified saliency maps (see Section 3.2)  $m_{\mathrm{test}}$ ,  $m_{j}$ :
$$
s \left(x _ {\text {t e s t}}, x _ {j}\right) \triangleq \operatorname {I o U} \left(m _ {\text {t e s t}}, m _ {j}\right) = \frac {\left| m _ {\text {t e s t}} \cap m _ {j} \right|}{\left| m _ {\text {t e s t}} \cup m _ {j} \right|}. \tag {5}
$$