LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Models
Authors: Zecheng Tang
Chenfei Wu
Juntao Li
Nan Duan
Venue: ICLR 2024 poster (2024)
Keywords: layout generation
large language model
code generation
Primary area: generative models
TL;DR: We propose LayoutNUWA, the first model that treats layout generation as a code generation task to enhance semantic information and harness the hidden layout expertise of large language models.

Abstract: Graphic layout generation, a growing research field, plays a significant role in user engagement and information perception. 
Existing methods primarily treat layout generation as a numerical optimization task, focusing on quantitative aspects while overlooking the semantic information of layout, such as the relationship between each layout element. 
In this paper, we propose LayoutNUWA, the first model that treats layout generation as a code generation task to enhance semantic information and harness the hidden layout expertise of large language models~(LLMs). 
Concretely, we develop a Code Instruct Tuning (CIT) approach comprising three interconnected modules: 1) the Code Initialization (CI) module quantifies the numerical conditions and initializes them as HTML code with strategically placed masks; 2) the Code Completion (CC) module employs the formatting knowledge of LLMs to fill in the masked portions within the HTML code; 3) the Code Rendering (CR) module transforms the completed code into the final layout output, ensuring a highly interpretable and transparent layout generation procedure that directly maps code to a visualized layout. We attain significant state-of-the-art performance (even over 50\% improvements compared to previous works) on multiple datasets, showcasing the strong capabilities of LayoutNUWA.

Published as a conference paper at ICLR 2024
LAYOUTNUWA: REVEALING THE HIDDEN LAYOUT EXPERTISE OF LARGE LANGUAGE MODELS
Zecheng Tangl.2 2. Chenfei Wu? Juntao Li'
Nan Duan
Soochow University
*Microsoft Research Asia
[ectangRstu., sjt]dsuus.edde en, aeteeu/aannteeelenccrreeeeee
Output Layout
Title
CODE
Code Rendering
wridth="189" height= "158""
Table
<rect data-cotegory=
title" xitle",x=""" x=" "="""""""""""""""""""""""""""""" w"""s""tath=""20 heigh"t="""""""""." 28"
(CR)
<rect data-cotegory=
table", x=="", yy=""" widthh"""""""""""""""""""""""""= width="8e" height="
<rect data-category-
x==x==" width- "88" height=
Text
</svg>
Code
Completion (CC)
Large Language Model
Inputs
Instruction: I want to generate layout in document style.
Please generate according to the categories I provide:
Text
Title
x
Initialization
Code
CODE
<svg width= height=
Table
<rect data-cotegory= title" x-dbt, widhh= <Mb, height=<M>/>> <rect dota-cotegory= table" x=<M>, y=cM> width=cH>, he <M>gh
width=109
(CI)
data-cotegorys text", y= y=
width=cMs widthh he neight=<M>/3
height=150
</svg>
Overview of LayoutNUWA. We propose Code Instruct Tuning (CIT) approach that consists of three modules: 1) the Code Initialization (CI) module quantifies the numerical conditions and initializes them HTML code with masks; 2) the Code Completion (CC) module utilizes the knowledge of large language models to complete the masked portions within the HTML code; 3) the Code Rendering (CR) module directly renders the completed code into the final graphic layout.
Figure 1:
ABSTRACT
Graphic layout generation plays significant role in user engagement and in- formation perception. Existing methods primarily treat layout generation as a numerical optimization task, focusing quantitative aspects while overlook- ing the semantic information of layout, such as the relationship between each layout element. In this paper, treats layout generation as a code generation task to enhance semantic informa- tion and harnesses the hidden layout expertise of large language models (LLMs). More concretely, we develop Code Instruct Tuning (CIT) approach compris- ing three interconnected modules: the Code Initialization (CI) module quan- tifies the numerical conditions and initializes them as HTML code with strategi- cally placed masks; 2) the Code Completion (CC) module employs the format- ting knowledge of LLMs to fill in the masked portions within the HTML code: 3) the Code Rendering (CR) module transforms the completed code into the fi- nal layout output, ensuring highly interpretable and transparent layout gener- ation procedure that directly maps code visualized layout. We attain sig- nificant ootte--r-the-rt performance (even over 50% improvements) multiple datasets, showcasing the strong capabilities of LayoutNUWA. Our code avail- able at pectps://gitnne htt.chhw..occcccccccocoootceetct http://hwwc..cccooooooooccccaaaa
propose LayoutNUWA, the first model that
Both authors contributed equally this research. During Zecheng's internship under the mentership of Chendei MSRA.
Corresponding

---

Published as a conference paper at ICLR 2024
1 INTRODUCTION
Graphic layout, which refers to the organization and positioning of design elements, sigmificantly influences the way users engage with and perceive the presented information (Lee al. 2020 As
a growing research field, layout generation (Li et al. 2019. Yang et al. 2020) aims to create diverse
and realistic layouts that streamline the design process and cater to various applications, such as user
interfaces (Deka et al. [0117; Jiang et al. (2022). indoor scenes (Di & Yu 2021 Feng ct al. 2023),
document layouts (Zhngg et al.
2019
Yamaguchi
2021), presentation slides (Fu al. 202 etc.
Current approaches @Jothh et al.
2019 ct al. 2019: Arroyo et al. 2021 Zhang et al. 2023a)
regard each element in the layout as numerical tuples (c,z, w. h), in which c indicates the ele- ment category, I and y represent coordinates, w and h correspond to width and height. For example, autoregressive-baeed methods (Yang et al., [2000 Jiang et al. 2022) view the tuple as a sequence and predict their values sequentially, while iifffusioo-basel methods (Chai et al.) [023] Inoue et al.) 2023) consider the tuple as a whole and predict their values through a denoising approach. Despite adopting different generative models, all of these methods fundamentally consider layout genera- tion as a numerical tuple optimization task. However, representing layouts as numerical tuples has its limitations, it primarily focuses capturing the quantitative aspects of the layout, such as positions and sizes, while lacking semantic information, e.g., the attribute of each numerical value, which may limit the model's ability capture more complex and rich layout information.
An insightful question emerges from the limitations of existing methods in layout generation: can integrate semantic information into the layout generation process to enrich the overall represen- tation and enhance the quality of the generated layouts? Addressing this question brings forth two major benefits: firstly, bolsters the understanding of relationships among various layout elements, and secondly, it enables us to tap into the semantic capabilities of LLMs (Tang et al., 2023) result- ing in more intricate and contextually relevant layouts for a wide range of applications (Jiang al.] 2022). Considering the inherent logical nature of layouts, which involve dependency relationships among layout elements, and the fact that each graphic layout can be represented with a fixed struc- ture sequence, code languages emerge numerical and semantic information while possessing a strong logical foundation (Chen et al. (022), which can thus bridge the gap between existing methods and the desired enriched representation.
promising alternative. Code languages can encompass
Based on the above observations, we propose LayoutNUWA, a groundbreaking model that revolu- tionizes the layout generation task by treating it as code generation task. Our innovative approach is designed to not only enhance the semantic information within layouts but also seamlessly leverage the expertise of L.LMs in the layout generation process. To achieve this, we design Code Instruct Tuning (CIT) approach comprising three interconnected modules: 1) firstly, the Code Initialization (CI) module quantifies the numerical conditions and initializes them as HTML code with strate- gically placed masks, paving the way for more meaningful and coherent layouts: 2) secondly, the Code Completion (CC) module employs the formatting knowledge of LLMs to fill in the masked portions within the HTML code, thereby harnessing the power of LLMs to improve the accuracy and consistency of the generated layouts; 3) lastly, the Code Rendering (CR) module transforms the completed code into the final layout output, ensuring highly interpretable and transparent layout generation procedure that directly maps code to a visualized layout.
Experiments across a variety of conditional layout generation tasks on three datasets, i.e., Rico (Deka) ct al. 2017). PubLayNet (Zhong et al., 2019) and Magazine (Zheng et al.) 2019). highlight the superiority of our method, in which LayoutNUWA can significantly outperform all the baselines and shows comparable results with the sascispecific models. Furthermore, LayoutNUW. can achieve at least 50% improvement in performance compared to the best baseline on the low-resource datasets, e.g., the Magazine dataset. In a nutshell, our contributions can be outlined as follows:
We introduce LayoutNUWA, the first model that treats the layout generation task as code generation task, effectively harnessing the hidden layout expertise of LLMss
We propose Code Instruct Tuning, which empowers the model to adhere to instructions and enriches the semantic information of layout, resulting in precise and standardized code.
We attain significant state-of-the -ort----rtttttttee performance on multiple datasets, showcasing the ro- bust capabilities of LayoutNUWA.
2

---

Published as a conference paper at ICLR 2024
2
RELATED WORK
2.1 LAYOUT GENERATION
Automatic layout generation, an important task for automatic graphical design for various scenarios
such as document layouts (Zheng et al. | Zhong et al. 2019] Yamaguchi 2021 Fu et al. 2022).
posters YYang et al. [016] Guo et al. |021] et al. 2023) and user interface (Deka al. 201 [ >
has
been recently extensively researched. Early approaches for layout generation involve embedding de-
sign rules into manually-defined energy functions OO'Donovan et al.) 2014 O' Donovan et al. 2015).
while other methods have explored generative models such GANs and VAEs for generating nu- merical graphic and scene layouts, including LayoutGAN (Li et al. 2019), LayoutVAE (Jyothi et al.
2019), LayoutGAN++ (Kikuchi et al.] 2021), NDN (Lee et al. (020)) and READ (Patil et al. 2020) Apart from them, transformer-based approaches utilize self-attention mechanisms to learn numeri- cal contextual relationships between elements and achieve layout completion based on partial layout
inputs (Yang et al. 2020; Kong et al. 2022] Feng et al.] 2023).. Recently, with the prevalence of
diffusion models, several works also adopted diffusion models tackle a broader range of condi-
tional layout generation (Chai et al. 2023 Inoue et al. 2023: Zhang et al., 2023a; Hui et al. 2023
Cheng et al., 2023). However, existing methods primarily treat layout generation
a numerical
optimization task, focusing on quantitative aspects while overlooking the semantic information of layout, such as the relationship between each layout element. Different from previous works, we convert the layout generation task into the code generation task to directly generate the layout in code language and thus utilize the rich knowledge from LLMs, which can significantly improve the FID by 50% in the Magazine dataset in 4.2
2.2 INSTRUCTION TUNING
Instruction tuning represents the process of fine-tuning LLMs on the instruction dataset in a super- vised fashion, which narrows the gap between the next-word prediction manner of LLMs and the users' objective of having LLMs adhere to human instructions (Zhang et al. 2023c). Early attempts on instruction tuning involve multi-task training with manually-writte wwwittten descriptions about differ- ent tasks (Mishra et al., 2021 Wei et al.) [202]] Sanh et al.) 2021; Xu et al., 2022 Muennighoff al. 2022 lyer et al. 2022) or automatically generated instructions (Wang et al. 2022 Gu et al.
2022 Zhhng et al. 2023b Honovich et al., (202aa b). Apart from controlling the LLMs through input instruction, Nye et al. (2021) show that LLM can handle more complex tasks by generating the intermediate steps and Wei et al. (2022) propose chain-of-thougle technique by enriching the
instruction with intermediate reasoning step descriptions, which endows LLMSS with better perfor- mance (Wang al., 2022; Zelikman al., 2022, Wu et al.) 2023 Xu et al. 2023). However, the
instruction tuning methods mentioned above are primarily intended for text generation tasks and not ideal for layout generation tasks, which involve numerical optimization. Thus, we propose code instruction tuning method that is specially designed for the layout generation task. Experiments in 3..1 indicate that the performance significantly drops if the code instruction tuning is not adopted.
METHODOLOGY
3.1
PROBLEM FORMULATION
The layout generation task aims to generate well-organized layout S = (s:) with N repre- senting the number of elements in the layout. Each element, 8; = (G,, (4, Ne, 24: wi, ht), consists of the following components: c, is the category, Iı, ys indicate the center location, and w;, h4 represent the width and height, respectively. In this study, we focus on the conditional layout generation task, wherein partial components in s; are masked with M, and the complete layout S should be predicted by model fo conditioned on the remaining components SiM:
S = (((S
(1)
Previous works (Joohhi et al., 2019 Yang et al. 2020 Inoue et al. 2023) regard each element s; as sequence of numerical values, e.g., (0. 10, 20, 25, 30), and train model to directly generate these values. However, this approach overlooks the semantic information of the components, thus limiting the model's understanding of the layout semanties. Based on this observation, we propose
3

---

Published as a conference paper at ICLR 2024
Golden Layout
Category +
Size
Position
<htel>
Condition
Input
RG, 20)
Math hright "10
2.. 10, 40.00 48)
21, 10, 90, B8)
TiBe
crect duto-cutagry" "Lable" -0, wirk s0 ig
Table
cruct duto-cutegory "extt" a-20, y-1 wiatd-10 ign---8
Cotteoryy acrieil Tear Tosta
</hody>
</opp
<ehtaal>
Completion
Category
Size Pocition
Condiction
Manerical
(C.A..y. tr.
Joint Loss
1y-10
Text
Large Language Model Code Instruct Tuning (Cm))
Table
X1
(1.m noy
Code Initialization (CI)
Code Completion (CC)
Instraction
Task Insoruction Code (Category Posstion)
Layout style Description; Foreat
Task befinition;
Jesest
(iiteL Structure)
Task Instruction code (Cacspory Size Pesition)
SuffIx: EasOcox heal:
JTTIL Structure
Toak instruction (complation)
generate lepcat deserent style.
<tady> Openteg Tay
Specify Background
Staseh paurrate sczording to renatrtig values prostee:
<s (ddth=(M)) heighc=(8)>
Layout Style
catels
Tap-
(hdlp
Eleeent xit
</>
(7at 2313 sacegptpn Stgggtp/// seok, pnol, vibitecle Jeigeteclo
</htab>
1o tast"
wite tnigniselt
Flawest
Code
cota-cnneger (ch, (2), pirh,, wisth-fer ((r), (M1//
</nnel
Figure 2: The training process of LayoutNUWA, which converts layout generation task to code generation task and utilizes code instruct tuning to leverage LLM's capability for layout generation.
problem definition, where convert the input Saas and output S into code language and view the layout generation task as a code generation task:
CODE(S) fo (CCODE((S
Eq. has the following 3 advantages compared with Eq.
By converting the numerical values into code language, the model can better capture the semantic relationships between different components of the layout.
Semantic Insights:
LLM Utilization: By using code language, the model can further leverage the knowledge Large Language Models (LLMs) and thus enhance the quality the generated layouts.
Model Scalability: The code language has stronger expressive capability compared to numerical values, which allows the addition of more attributes for layout elements.
3.2 CODE INSTRUCT TUNING
As shown in Fig. we propose Code Instruct Tuning (CIT) with three modules: (1) Code Ini- tialization module converts layout into masked code language with dynamic templates; (2) Code Completion module inputs the masked code to LLMs to generate complete code; (3) Code Render- ing module directly renders code to the final graphic layout. We illustrate these modules below.
3.2.1 CODE INITIALIZATION
Element Quantization We quantify the numerical values of i-th element position {x; Vi and size (w;, he) in the layout with Adaptive Quantization method IInoue et al., (023) that applies k-Means algorithm (MacQueen et al. 1967) to cluster the position and size information of each element, addressing the highly imbalanced distribution of these values, e.g., elements may overlap or cluster together. Different from the previous works (Chai al. 2023 Zhang et al. 2023a Inoue al. 2023), we use absolute position to represent the coordinates rather than relative positions. This aligns with code language and allows direct rendering of layouts without necessitating coordinate conversion, thereby preventing potential information loss. We maintain precision up to one decimal place and directly convert the clustered results into strings.
4

---

Published as a conference paper at ICLR 2024
Template Construction The overview of template construction is shown in Fig. We construct the templates based on the most common web page layout code, HTML, which contains a wealth of information and is easily accessed by LLMs during the pre-training process (Touvron et al. 2023 Rozière et al., 2023). Specifically, in HTMI code, each element is described with a tag that provides information about the content or the element structure. Since the elements in the layout regular squares, we chose the <rect> tag as the content tag to describe each element:
<rect data-cateegozy=(c]
(z)} y= (3:) width- (wy) height-(ha)
where is the element category in textual format and {a;, Vi. ws: hi) are the quantified position and size of the i-th element. Then, to combine all the elements into unified structure, we used an opening tag and closing tag to define the boundaries of each layout, which can be written as:
<html><<oode thtml><bod>><svg width-a (W) height=(H)>
</svg></body></htmm
where W and H the background width and height of the layout.
In order to facilitate better learning of layout in various domains and tasks and leverage the instruction-following capabilities of L.LMs, we design the following prompts:
want to generate layout in (Domain) style. Please generate the layout according to the (Task Condition) I provide:
where the {domain) and the (Task Condition) will vary according to different domains and tasks. For instance, for the RICO dataset, we set Domain as "mobile Ur", and for the layout completion task, we set Task Condition as "remaining values".
3.2.2 CODE COMPLETION
To construct the conditional input of the layout generation task, we utilize the mask tokens of LLMs to represent the masked values M and let the model predict the masked values within the HTML
code. Different from previous works (Chai et al. 2023 Zhang et al. 2023a Inoue et al. 2023) that applied the customized numerical vocabulary, doing so, we can leverage the knowledge of the numerical tokens inherited in the LLMs. Considering that almost all the LLMs follow auto-regressive generation manner that brings significant limitation to the layout generation task since the model should predict the same layout under different element orders. even if the layout doesn't have naturally defined order (Yang al., 2020). Thus, design self-consistency strategy that randomly permutes the order of the input elements in the layout within mini-batch. Meanwhile, in order to adapt LLMs different conditional layout generation tasks, have performed multi-task modeling on the same layout, utilizing various conditions and implementing a joint loss for these tasks. Given the permutation times and task numbers T,, the joint loss, denoted as L(-), for each layout S can be written as:
employ the LLM's token vocabulary directly. By
T N ΣΣΣ
L(S 0)
M (K)
(M)
(3)
where f the model parameters and sy denote the j-th element in the layout
3.2.3
CODE RENDERING
Most existing works require the extra conversion step to render the graphic layouts (Yang et al.) 2020 Chai et al. 2023 Zhang et al.) 2023a), e.g., converting the relative position to the absolute position, causing the information loss. Different from previous work, LayouNNUWA allows for immediate rendering as it generates the absolute position directly. Besides, considering the potential output issues such boundary overflow (Inoue al. 2023) and format errors, we employ regular expressions to remove mismatched formats and implement clipping operations for elements that exceed the background size.
5

---

Published as a conference paper at ICLR 2024
4 EXPERIMENT
4.1 EXPERIMENTAL SETTINGS
Datasets We evaluate the model performance on three widely used public datasets. RICO (Deka) et al. 2017) is user interface design dataset for mobile applications containing 25 element cat- egoniss and 66K+ UI layouts. PubLayNet (Zhong et al. 2019) consists of 360K+ layouts for documents with 5 element categories. Magazine (Zheng et al. (219)) is low-resource magazine layout dataset containing around 4K annotated layouts and 6 element categories. We follow Lay- outDM (Inoue et al.) 2023) view the original validation data as the testing set and pre-process all three datasets by discarding the layouts containing more than 25 elements as well as splitting the filtered data into the training and new validation sets by 95% and 5%.
Evaluation Metries We employ four metries to evaluate the generation results comprehensively, including Frechet Inception Distance (FID), Maximum Interaction over Union (mloU), Alignment (Align.), and Overlap. Among them, FID compares the distribution of generated and real layouts. Similar to the previous work (Inoue et al. 2023), we utilize an enhanced feature extraction model for layouts (Kikuchi et al.] 2021) to compute the FID score. We measure the conditional similarity between generated and real layouts using mioU, which is done by calculating the maximum IoU between bounding boxes of generated and real layouts with the same type set. Aligmment and Over- lap scores are calculated following the previous work (Li et al. 2019) to evaluate proper element alignment and overlapping in a generated layout, and is worth noting that we ignore normal over- laps, e.g., elements on top of the background, and discard the layouts that failed to generate. For reference, we show the evaluation results between the validation set and test set as Real data.
Tasks and Baselines We evaluate LayoutNUWA on three conditional layout generation task including the Category to Size and Position (C S+P) task, the Category and Size to Position (C+S
-+ P) task, and the Completion task. More concretely, the C -> S+P task requires the model to predict the position and size of the element based on its category. For the C+S - P task, the model
predicts the position of the element based on both its size and category. Finally, in the completion task, the element's size and position values are randomly masked up to 80%, and the model predicts the entire layout using the remaining values. We compare LayoutNU/A with six strong baselines, including LayoutTrans (Yang et al. (020). BLT (Kong et al., 2022) LayoutGAN++ (Kikuchi al. 2021. MaskGIT (Chang et al. (022). DiffusionLM (Li et al. (022) and LayoutDM (Inoue et al. For the above baselines, implement them with the official code directly.
2023
Implementation Details We implement LayouINUWA with two 7B LLMs: LLaMA2 (L2) Tou- vron et al. 2023) and CodeLLaMA (CL) (Rozière et al., 2023). We train LayoutNUWA with two settings: (1) Domain--Seeiiie
(DS) setting, where the model is trained on distinct datasets, and (2)
Domain-Agnostic (DA) setting, where the model is trained on all three datasets, including RICO, PubLayNet, and Magazine. The default configuration for LayouNNUWA utilizes Codel (CL)
and Domain-Agnostic (DA), i.e., LayoutNUWA-CI-IJ
We set permutation times K = 10 and
task numbers T' 3. For model training, we use DeepSpeed Library (Rajbhandari al. 2020 to run all experiments on 64 NVIDIA V100 GPUs. We apply Top-p sampling (Holizman et al. 2019) for inference, where p= 0.9 and the temperature is 0.6. We set the maximum generation length as 1024 for each sample to ensure the completeness of the layout code.
4.2 QUANTITATIVE EVALUATION
We report the model performance on three datasets: the Magazine dataset in Tab. [ RICO, and PubLayNet datasets in Tab. 2 For the Magazine dataset, LayoutNUWA demonstrates remark- able performance by significantly surpassing all baseline measures across all tasks. Moreover, it outperforms the strong baseline LayoutDM by more than 50% when assessed with the FID metric.
The significant improvements in Tab. are due to three aspects: 1) previous approaches generated numerical values, while LayoutNUWA generates code with labels, which greatly benefits the model
We also report the model performance three datasets under the unconditional generation setting in Appendix F for space limitation.

---

Published as a conference paper at ICLR 2024
Model
Layout Format
LLM
Domain
C. S
C+ P
Completion
mIOU
(i)
mIOU (t)
FID
mIOU
FID
LayoutTrans
Nomerical
Spocifie
0.116
36.207
0.153
33.931
0.228
25,804
BLT
Numerical
Specific
0,087
65.372
0.126
41.089
0.103
97.142
+
Numerical
Specific
16.952
0.293
MaskGIT
Numerical
Specilic
0.059
140.94
0.100
78.226
0,024
152.591
DiffusionI.M
Numerical
Specific
0.151
32.114
0,144
24.370
0.138
33,172
LayouiDM
Numerical
Specific
0.234
19:206
0.308
14,265
0.328
15,804
LayonNUWA-L.2-DS (ours)
Code
LLaMA2
Specifie
0.260
0.358
6.682
0,418
8.257
LayouNUWAA-22 lours)
Code
LLaMA2
Agnostic
0.293
9.632
0.394
7.238
0,113
8.734
LayonNUWA-CLL CL-DS (ours)
Code
LadeLLaMA
Specific
0.293
8.985
0.348
5.355
0,410
7,34
LayouLNUWA (ours)
CodeLLaMA
Agnostic
0.495
Real Duta
0.348
6.695
0.348
6.095
0.348
6.695
Table 1: Quantitative comparison on Magazine dataset, where the bold font denotes the best result and underline represents the second-best performance.
Models
RICO
PubLayNet
mloU (f)
Align. (-)
Overlap
FID (1)
mloU
Align. (-+)
Overlap (-+)
FID (1)
LayourTrans
0.219
01014
13.012
11.237
0.271
0.016
3,229
38,910
BLT
0.208
0.013
11.743
14.260
0.232
0.009
16.742
76,499
LayouttAN++
0.263
0.06
0.011
1.713
10.319
MisaGIT
0.267
0001
26,865
27 37440
0.320
0.044
1.857
16,98
Condition
DiffusiceLM
0.299
0.018
17.655
31.644
0.262
0.027
3,532
20.021
C S
LayouDM
0.275
0.010
3.576
0,310
0,010
0.024
7.915
LayorNUWA-11--25 (ours)
0.351
0.000
10.190
3,728
0.058
LayomNI/WA-12-DD (ours)
0.386
0011
10.214
3.101
0.324
0.011
0.072
6,800
COoUNUWA-CLDDS (ours)
0.377
0.099
10.263
3,705
0.3%
0.088
0,053
6.715
LayouNNUWA.com
0.445
Layou(Trans
0.311
0011
11,902
9.368
0.315
0.033
31 627
BLT
0.34]
0.008
13:170
4.487
0.356
0.006
5.169
$,831
LayouGGAN++
0.349
0.011
9.628
6.219
0.346
0.008
2,746
9.936
Mis&GIT
0.351
0.003
Condition
Diffusionl
0.278
11,884
3,990
C+S-P
LayouDM
0.391
0.009
12,072
2.288
0.381
0.010
2,041
4.175
LayouNUWA-L2-DDS (ouns)
04622
0.008
10.436
3.035
0.426
0.010
1,752
4. 4,105
Layou/NUWA-12--A. (ouns)
0.464
LayouNUWA-CL CL-DDS (ours)
0.169
0.007
9,856
2.984
0.166
0.009
L610
4,012
Lyyo (ours)
0.564
0.007
7,968
2,700
0.483
0.002
0.108
3.697
LayourTrans
0.561
0.008
10.080
3,733
0.439
0.012
2,053
8 6.689
BLT
0.471
0.007
53.658
121.110
0.157
0.002
109, 483
155.157
Mas&GIT
0.537
0.024
9.211
33-463
0.349
0.011
4,768
12.013
DillisionMM
0.218
0.021
K.681
22.220
0.332
0.012
4,436
16.576
Completion
LayouDMM
0.580
7,570
LayouNUWA-L2 L2-DS (ours)
0.610
0.009
7,239
8,875
0.407
0.010
1.337
7.337
LayouNUWA-L22 L2-DA (ours)
0.621
0.007
10.157
8.724
0.477
0.012
1,383
7.169
LayouNUWA CL-DS (ours)
0.641
0.007
7.529
8.734
0.473
0.012
1.311
7.233
LayouEUWA (ours)
Real Data
0438
0.044
8.706
6,250
0.691
0.001
0.039
1,850
Table 2: Quantitative comparison on the RICO and PubLayNet Datasets. For Align. and Overlap metrics, the closer to the real data, the better performance is (indicated by -t).
by utilizing the semantic information of layout attributes such width, height, position, and cate- gory; 2) none of the previous methods used LLMss. However, we have introduced LLMSs for the first time, which has resulted in significant performance enhancements, i.e., performance has improved from 19.206 to 9.741. Furthermore, when we use CodeLLaMA, which is tuned on code language, the performance improves even further 8.985; 3) since different domains require distinct layout manner. How-
formats, early numerical-based methods could only be trained in domain- ever, LayoutNUWA is based on code structure, which can be trained in a domain- aggossic manner, allowing for complementary among data from various domains, thus further improving FID to 8. 7.1..
We have also conducted extensive experiments on two other datasets: RICO and PubLayNet, as shown in Tab. 2 The LayoutNUWA notably surpasses all baseline methods in the majority of tasks. Although it does not achieve the best performance in two specific tasks, still secures at least the second-highest performance in those instances. This shows the strong generalization of the LayoutNUWA. It is worth mentioning that our model also achieves closer Align. and Overlap scores to the Real Data compared to the baselines. Although previous work has suggested that refinement and discriminator processes can contribute to improving the Align. and Overlap (Inoue et al. 2023 Li et. 2019) scores, our method attains better results without employing these steps.

---

Published as conference paper at ICLR 2024
[Ours] LayoutNaiwA
[Ours] LayoucNUWA
Category
Layouttrans
LayourtI
Real Design
Category
Layouttrem
LayouttM
Maseoit
Design
x10
X10
(- S+p
List
List
X4
Test X12
Title X7
FAgure X1
Title X2
List
C+
Figure x3
Text
X-3
List x1
Text X:14
Test K15
Title
Titll X2
Completion
Title x1
Figure
Tablu x1
Figure 3: Samples generated by LayoutNUWA the PubLayNe Net dataset.
Task
Models
Tuning Method
mloU (f)
Align. (-+)
Overlap (-+)
FID (L)
Fail (L)
LayoutNUWA-L2-DD
CTT
0.260
0,021
2.898
9.741
0.000 %
Condition
wlo template
Instruct Tuning (DS)
0.124
0.049
3.221
16.324
1.020
C-S+P
template
Instruct Tuning (DA)
0,000
wlo template&knnttuec
Numerical Tuning
0.126
0.053
3.581
17.982
3,571
LayoutNU MnyouNNUWA-L2-D UWA-L2-DS
CIT
0,358
0,020
2,483
4,682
0 %
Condition
template
Instruct Tuning (DS)
0.182
0.021
2.673
12.432
0.000
C+S
w/o template
Instruct Tuning (DA)
0.000
w/o template&knsttuct
Numerical Tuning
0.189
0.024
2.892
14,326
0.000 %
LayoutNU WWALL2DD
CIT
0.418
0.020
2,309
7.257
0.253 %
template
Instruct Tuning (DS)
0,206
0.017
Completion
w/o template
Instruct Tuning (DA)
6.633 %
w/o templatekinstruct
Numerical Tuning
0.214
0.020
3.003
16.243
6.122
Real Data
0.348
0.016
1.521
6.695
Table Comparison among different tuning methods, where "Fail" the failure ratio of generation.
4.3 QUALITATIVE EVALUATION
We render the generated layout code with the Code Rendering (CR) method, and Fig. 3 shows the sampled rendering results the PubLayNet dataset. By comparing with other baselines, we can observe that the layouts generated by LayouNUUA exhibit excellent element alignment, and the proportion of overlap between elements is minimal. Additionally, our results are the most consistent with the Real Design data, i.c., the size and position of the generated element are essentially consis- tent with the real design, indicating that by treating the layout generation task as a code generation task, NuyA has successfully learned the distribution of document layouts, thus result in more precise and realistic layouts. More generated cases can be referred to Fig. [1 in the appendix.
5 ABLATION STUDY
We investigate the effectiveness of the CIT tuning method in Sec. 5.1 and compare the impact of different output formats and fine-tuning Sec. 5.2. More concretely, we set the LayoutNUWA-LE DS model the basic setting and conduct the ablation studies the Magazine dataset.
8

---

Published as a conference paper at ICLR 2024
Layout Format
Task
Model
mIoU (1)
Align. (-)
Overlap (-)
FID (J)
Fail (L)
Condition
LayouNUWA-N LayouNUWAA-L2-DS
Numerical
0.000
0.000
0.867
78.030 %
C + P
Code
0.260
0.021
2.898
9.741
0,000 %
Condition
LayoutNUWA-N LayouNUWA-LL--DS
Numerical
0.000
0.000
24.959
349.231
21.717
C S P
Code
0.358
0.020
2.483
4.682
0.000 %
Completion
LayouINUWA-N LayouNUWW--L--DS
Numerical
0.000
16.602
29,293
Code
0,418
0.020
2.309
7.257
0.253 %
Real Data
0.348
0.016
1.521
6.695
Table 4: Comparison among different output formats.
5.1
EFFECT OF TUNING METHOSS
We progressively reduce the modules in CIT and fine-tune the model using the corresponding con- structed data. Specifically, we first exclude the code template and directly convert the element information into ordered sequence S with task instruction before it, i.e., the instruction tuning method. Then, we further remove the task instruction and directly fine-tune the model using data from different tasks separately, i.e., the numerical tuning method. As shown in Tab. we can ob- serve that the model performance has declined significantly without the code template, and it can only work in the DS setting since the model can simply generate repetitive and out-of-order results that are inconsistent with the element sequence in the DA setting. Furthermore, the numerical tuning method can only support the DS setting as there is no task instruction for the model to distinguish between different tasks, and the model performance is far inferior compared to those of the CIT as such an approach overlooks the rich semantic information among the elements and can not calibrate the prior code knowledge of LLMs.
5.2 EFFECT OF OUTPUT FORMAT AND FINETUNING
We compared the effects of the model out- put in code format and numerical format. For the numerical output format, we de- signed a Code Infilling task, which in- volves making the LLM predict only the masked values rather than predicting the entire code sequence. As shown in Tab, 4 we can find that generating in numerical format will increase the failure ratio model generations, e.g., the model will generate repetitive results, and significantly decrease the model performance. This is because the layout generated by the conditional layout generation task should be logical, while only predicting the masked parts can lead to discrete values that lack logic. Besides, Due to the influence of the autoregressive manner, where the content generated the next step depends on the previous history, this phenomenon may result in a higher failure probability of model generation when predicting layouts with more masked values. We also conduct a comparison between LayoutNUWA and GPT- 4 (Bubeck et al.] (023). Specifically, we allow GPT-4 to perform inference by constructing the input using the CIT method. Tab. showw code instruct tuning for LLM necessary, as using LLM in zero-shot manner leads to high fail rate (100% fail rate of L.LMA22 and around 30% for GPT-4).
C- S+P
C+S P
Completion
Model
(
(4)
Fail (4)
LLaMAZ Shol)-
100.0 %
100.0 %
100.0
aadelIaMA (Zero-sho))
100.0 %
100.0 %
100.0 %
GPT-4 (Zero-Shot)
34.2
28.8
28.5 %
LayouINUWA
0.0 %
0.0 %
0.3
Table 5: Comparison with LLMs.
CONCLUSION
In this paper, we propose LayoutNUWA, a groundbreaking approach that treats layout generation as code generation task, effectively enriching the semantic information of layouts and leveraging the hidden expertise of LLMs. Extensive experiments on multiple datasets have demonstrated the superiority of our method. This research has the potential to revolutionize the field of layout genera- tion and pave the way for further exploration and development of semantic-ane aware layout generation approaches in various applications.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This research done in alignment with Microsoft's 's responsible Al principles.
REFERENCES
Diego Martin Arroyo, Janis Postels, and Federico Tombari. Variational transformer networks for layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13642-13652, 2021.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Harly experiments with gpt-4. arXiv preprint arXiv:2303. 12712, 2023.
Shang Chai, Liansheng Zhuang, and Fengying Yan. Layoutdm: Transformer- diffusion model for layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18349-18358, 2023.
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315 11325, 2022.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12585 2022.
Chin-Yi Cheng, Forrest Huang, Gang Li, and Yang Li. Play: Parametricalty conditioned layout generation using latent diffusion. arXiv preprint arXiv:2301. 15529 2023.
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibsehman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design ap- plications. In Proceedings of the 30th annual ACM symposium on user interface software and technology, pp. 845-854, 2017.
Xinhan Di and Pengqian Yu. Multi-agent reinforcement learning of 3d furniture layout simulation in indoor graphics scenes. arXiv preprint arXiv:2102.09137, 2021.
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuchai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and genera- tion with large language models. arXiv preprint arXiv:2305. 15393, 2023.
Tsu-Jui Fu, William Yang Wang, Daniel McDuff, and Yale Song. Doc2ppt: automatic presentation slides generation from scientific documents. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 634-642, 2022.
Yuxian Gu, Pei Ke, Xiaoyan Zhu, and Minlie Huang. Learning instructions with unlabeled data for zero-shot cross-task generalization. arXiv preprint arXiv:2210.09175. 2022.
Shunan Guo, Zhuochen Jin, Fuling Sun, Jingwen Li, Zhaorui Li, Yang Shi, and Nan Cao. Vinci: an intelligent graphic design system for generating advertising posters. Proceedings of the 2021 CHI conference human factors in computing systems, pp. 1-17, 2021.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv: /904. /904.99944. 2019.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689, 2022a.
Or Honovich, Uri Shaham, Samuel Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205. 2022b.
Mude Hui, Zhizheng Zhang, Xiaoyi Zhang, Wenxuan Xie, Yuwang Wang, and Yan Lu. Unifying layout generation with a decoupled diffusion model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1942-1951, 2023.
10

---

Published as a conference paper at ICLR 2024
Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10167-10176, 2023.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint urXiv:2212.1201 12017, 2022.
Zhaoyun Jiang, Shizhao Sun, Jihua Zhu, Jian-Guang Lou, and Dongmei Zhang. Coarse-to-fing generative modeling for graphic layouts. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 1096-1 103, 2022.
Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochas- tic scene layout generation from label set. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 9895-9904, 2019.
Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Constrained graphic layout generation via latent optimization. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 88-96, 2021.
Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, and Irfan Essa. Blt: bidirectional layout transformer for controllable layout generation. European Conference on Computer Vision, pp. 474-490. Springer, 2022.
Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong B Le, Haifeng Gong, Ming-Hsuan Yang, and Weilong Yang. Neural design network: Graphic layout generation with constraints. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 491-500 Springer, 2020.
Fengheng Li, An Liu, Wei Feng, Honghe Zhu, Yaoyu Li, Zheng Zhang, Jingjing Lv, Xin Zhu, Junjie Shen, Zhangang Lin, et al. Relation-aware diffusion model for controllable poster layout generation. arXiv preprint arXiv: 2.0.02.300.00086, 2023.
Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Generating graphic layouts with wireframe discriminators. arXiv preprint 1911 09991.07777,.. 2019.
Xiang Li. John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion- Im improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328 4343, 2022.
Quanyu Long, Tianze Luo, Wenya Wang, and Sinno Jialin Pan. Domain confused contrastive learn- ing for unsupervised domain adaptation. arXiv preprint arXiv:2207.0456: 2022.
James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pp. 281-297. Oakland, CA, USA, 1967.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXhv:2104.08773, 2021.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf. et al. Crosslingual gen- cralization through multitask finetuning, arXiv preprint arXh::2211.01786 2022.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2//2.001 14, 2021.
Peter O' O'Donovan, Aseem Agarwala, and Aaron Hertzmann. Designscape: Design with interactive layout suggestions. Proceedings of the 33rd annual ACM conference on human factors in computing systems, pp. 1221-1224, 2015.
11

---

Published as a conference paper at ICLR 2024
Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. Learning layouts for single-pagegrane pagegraphic designs. IEEE transactions on visualization and computer graphics, (118)))2200-2133, 2014.
Akshay Gadi Patil, Omri Ben- Eliezer, Or Perel, and Hadar Averbuch- Read: Recursive au- toencoders for document layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pp. 544-545, 2020.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 2tt.:22308..29555 12950, 2023.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021.
Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song- Cong-Chum Zhu, Yitao Liang, and Muhan Zhang. Large language models are in-context semantic reasoners rather than symbolic reasoners. arXiv preprini arXiv:2305.4483 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.099 09288, 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. colfsconsi improves chain of thought reasoning in language models. arXiv preprint 220.v 2022.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint 1XXiv:109.00652. 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.119033 2022.
Lijun Wu, Xu Tan, Di He, Fei Tian, Tao Qin, Jianhuang Lai, and Tie- Yan Liu. Beyond error propagation in neural machine translation: Characteristics of language also matter. arXiv preprint arXiv:1809.0 0000000 2018.
Yang Wu, Yanyan Zhao, Zhongyang Li, Bing Qin, and Kai Xiong. Improving cross-task generaliza- tion with step-by-step instructions. arXiv preprint (xxiv:2305004299 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.122444 12244 2023.
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. Ze- roprompt: Scaling gromet-based pretraining to 1,000 tasks improves zero-shot generalization. arXiv preprint arXiv:2201. 006910, 2022.
Kota Yamaguchi. Canvasvae: Learning to generate vector graphic documents. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5481-5489, 2021.
Cheng-Fu Yang, Wan-Cyuan Fan, Fu-En Yang, and Yu- Yu-Chiang Frank Wang. Layoutransffrmmer:
Relation-aware scene layout generation. 2020.
Xuyong Yang. Tao Mei, Ying-Qing Xu, Yong Rui, and Shipeng Li. Automatic generation of visual- textual presentation layout. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 12(2):1-22, 2016.
12

---

Published as a conference paper at ICLR 2024
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances Neural Information Processing Systems, 35: -114466-55488, 2022.
Junyi Zhang, Jiaqi Guo, Shizhao Sun, Jian-Guang Lou, and Dongmei Zhang. Layoutdiffusion: Improving graphic layout generation by discrete diffusion probabilistic models. arXiv preprint arXiv:2303. 1589, 2023a.
Ruohong Zhang, Yau-Shian Wang, and Yiming Yang. Generation driven contrastive self-training for zero-shot text classification with instruction-tuned gpt. arXiv preprint arXiv:2304. 1872, 2023b.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang. Jiwei Li, Runyi Hu, Tianwi Zhang. Fei Wu, al. Instruction tuning for large language models: A survey. arXiv preprint Xrv::23308 10792, 2023c.
Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau. Content-aware generative modeling of graphic design layouts. ACM Transactions Graphics (TOG), 38(4):1-15, 2019.
Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for docu- ment layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1015-1022. IEFE, 2019.
13

---

Published as a conference paper at ICLR 2024
A
LIMITATIONS
Since LayoutNUWA employs the autoregressive (AR) L.LMs as the backbone, our method naturally inherits the shortcomings of the AR models:
The generation speed slower than the non-autoregressive models (Chang al. 2022
It suffers from the error propagation problem (Wu al. 2018 i.e., as shown in Fig. especially when training is insufficient, where the content generated later in the sequence may be negatively affected by the errors the content generated earlier.
Additionally, model may suffer from the domain confusion issue Long et al. (2022) to some extent under the Dgnaii-Agnostic setting as illustrated in Appendix F In our future work, we will address these challenges and make improvements to generate better graphic layouts.
Code Generation Process (w/o Error Propagation)
Title
<hinl>
<body>
Code
Table
<svg width-"198 height="1500
Rendering
<rect dato-cutegory- "title", xit1)"",x=11 xx15,,--16 wialhhe76, height
data-category- "tahll" y-1,, width--8, 48/
<rect date-cutegorye "Lexttteoorr""ex'""" x=18,y=98, 9id i-httBe/3
(CR)
Text
</svg>
c/body>
</html>
Title
Code Generation Process (w/ Enror Propagation)
Code
Rendering
<htnn>>
<body>
<svg width="18e" hes8""
(CR)
catt-category--- "title" y- w wrightisgnn 58,
[Renain to generste.
.]
Too high Title Element
> Lower Table Element's Position
Code Generation Process (w/ Error Propagation)
Title
<hinl> <body>
Code
Rendering
<svg width="188 height= "150">
<rect cate-category- "title" x=15, y-18, wisth-7e, hetth/o,,helhhe height se. cate-cattgory--o
y- widtt width--0, hright 99
(CR)
Table
[Renain to generate. ..]
No space for Text Element > Overlapping of Elements
Code Generation Process (w/ Error Propagation)
Title
<body>
Code
width height="159"
Rendaring
crect data category- y-18, -i 55.
Text
<rect date-category- "table" x=18, widthwoo. height 38.
category- "toxt" y-48,y-45, height-58/
(CR)
Table
</svg>
</body>
</htnl>
Figure 4: One possible case illustrating error propagation, where the top group shows an ideal model generation scenario, and the bottom group shows the error propagation process.
14

---

Published as a conference paper at ICLR 2024
B COMPARISON WITH GPT-4
We utilize the GPT-4 model with the commercial API and strictly follow the usage policy 2 report the detailed performance of the GPT-4 model in Tab. 6 and show several rendered graphic layouts in Fig. TO We can observe that the content generated by GPT-4 in the zero-shot setting primarily follows the layout design rule, which further confirms the potential capability of LLMs in generating layouts when guided by the CIT approach. However, when compared to LayoutNUWA, there are several issues with the results generated by GPT-4: the distribution of elements is uneven, with elements tending to be concentrated in certain areas, such as the left side of the canvas; 2) the element sizes are inconsistent, for instance, in some graphic layouts, there might be one two large elements, which results in the high scores of the mIOU and Overlap metrics for some tasks; 3) there is a significant discrepancy between the data distribution of generated content and the real data.
We
Task
Model
mIOU (J)
Align. (-)
Overlap (-)
FID (L)
Fail (L)
Condition
GPT-4 (ZeooSShot)
0.264
0.006
0.165
34.184 %
C
LayouNNUW--L2-D
0.260
0.021
2.898
9.741
0.000
Condition
GPT-4 (Zero-Shot)
0.330
0.011
1.149
28.788 %
C+S
LayouNNUWA-22-DS
0.358
0.020
2.483
4.682
0.000
GPT-4 (Zero Shot)
0.362
0.044
0.728
28.535 %
Completion
LayouNNUWA-22-DS
0.418
0.020
2.309
7.257
0.253
Real Data
0.348
0.016
1.521
6.695
Table 6: Detailed performance of GPT4 on the Magazine dataset. is worth noting that due to the significant difference between the results generated by GPT-4 and the real data, the FID score cannot be calculated.
C HUMAN EVALUATION
We conduct the human evaluation for the model performance on the RICO and PubLayNet datasets. Specifically, We compare LayoutNUWA with other strong baselines, including LayoutDM (In- ouue et al., 2023) and
(Yang et al. 2020), and randomly sample 25 graphic lay- outs generated from each model. We invite the annotators choose which model performs better according to two evaluation settings: 1) quality evaluation based on the detail depiction, overlapping degree, and layout rationality in each layout; 2) diversity evaluation based on the diversity of the element arrangement in each layout. We hire 10 annotators to give their preferences, and the results are shown in Fig. 5(a) and Fig. [56) We can observe that layoutNUWA significantly outperforms the other two strong baselines, i.e., LayouDDM and Layout Transformer, in terms of both generation quality and generation diversity. More generated cases can be referred to Fig. T0 (Magazine dataset)
and Fig. II (RICO and PubLayNet datasets).
LayoulDM
LayoutThena
Layo
LaysuloMt
Laysuttrans
0.56
0.5
0.48
0.5
0,50
9.12
0.3
0.24
0.2
6:16
0.1
9.68
Quality
Diverdity
Quality
Divershty
(a) Human evaluation on the RICO dataset.
(b) Human evaluation on the PubLayNet dataset.
15

---

Published as a conference paper at ICLR 2024
D SAMPLED TRAINING-TESTING PAIRS WITH SIMILAR LAYOUT DISTRIBUTION
Considering that the qualitative generation results of LayoutNUWA look very similar to the actual design (Fig. 3) in order to further verify the fairness and rationality of our experiment, and to prove that the model is not overfitting some specific training data, we used DocSim to select content from the training dataset that is similar to the Layout distribution of the that the layoul distribution between the test data and some of the training data has some similarities. For example, the PublayNet dataset, the Tite Element appears at the top of the overall layout, but it is not entirely consistent.
dataset. We can observe
Testing
Training
Training
Training Training
Magazine
RICO
PublayNet
Figure Sampled Training- Teettig--TTnngggg pairs by DocSim metric.
E MODEL PERFORMANCE UNDER UNCONDITIONAL SETTING
For unconditional generation, models generate 1,200 samples with the random seed. For LayouNUWA-CL-DSS randomly provide the empty templates without any prior information. e.g. category name, positions, eic However, since LayoutUWA-CLL-DA is trained under the domain-agnoste setting, we must provide with the domain prompt, e.g.. "I want generate lay- out in mobile UI style" for the RICO dataset. We report Align. and FID score of generation results Tab. 7 where we can observe that LayouNNUWA-CL-I scores and also achieve the best/comparable performance in Align. Score. As for the LayoutNUWA. CL-DA model, although the performance not as good as the CL-DS setting, the results are still better than the baseline (some indicators comparable to the bascline). We hypothesize that this may be due to potential domain confusion, i.c., the model has learned common features from previ- datasets, which may have negative effect specific domain. Stronger prior information is required to guide the model to generate layouts that are more consistent with a certain domain, such element categories,
can obtain the best results on FID
For qualitative evaluation, we render the generation results in Fig. 9 We can observe that the graphic layouts generated LayoutNUWA tion is closer to that of real data. This can be attributed to the given template which defines the output format of the model and the utilization of the prior code knowledge from LLMs.
compact, with lower failure rate, and their distribu-
TThis process is similar the fixed length provided by the previous works.
16

---

Published as a conference paper at ICLR 2024
RICO
PybLayNet
Magazine
Models
Align. (-)
FID (4)
Align. (-)
FID (4)
Align. (-))
FID (U)
TayourTrans
0.008
12.286
0.015
33.416
0.043
32.083
MaskGIT
0.003
60.724
0.004
28.836
0.003
157,560
DiffusionLM
0.019
23,997
0.016
18.720
0.043
37.951
LayouDDM
0.010
6.456
0.011
13.437
0.037
59.507
LayouNUWWA-LL-DS (ours)
0.008
5.674
0.007
8,914
0.092
24.108
LayonNUWAACI--DA (ours)
0.014
6.932
0.013
9.208
0.073
28.930
Real Data
0.004
6.250
0.001
1.850
1.693
6.695
Table 7: Model Performance on three testing sets under the Unconditional Generation setting.
(Dun) LayaGNNUMA
LayouNNUWA
Layouttrans
Layoutori
MaskkIT
Real Design
Layouttrans
LayoutoM
MaskGIT
Real
LayousNuw
Layouttrans
LayoutOM
MaskGI
Real Denign
Figure 7: PubLayNet Dataset
Figure 8: RICO Dataset
Figure 9: Magazine Dataset
Figure 9: Cases generated by LayoutNUWA under the Unconditional Generation setting. We
suggest zooming in on the monitor for better viewing.
F MODEL PERFORMANCE UNDER MIXED DOMAIN SETTING
Considering the impressiveness of the domain aamaii- model, we also take into account the issue of domain confusion (Long et al.) 2022), that is, the data from one domain may affect another domain. Here, we design a toy experiment with the mixed domain setting, that is, applying the domain conditions of Magazine and RICO separately prefixes well as the code template behind using data from another domain. For example, we can utilize the below instruction to induce model generation under the Magazine RICO mixed domain:
want generate layout magazine style. Please generate the layout according to the Task Condition) provide: {RICO code template)
As shown in Tab, g we can observe that when given more prior layout information, such as the S P setting, the model is less affected by domain confusion the model tends to "resist" such
17

---

Published as a conference paper at ICLR 2024
domain confusion, Le., The performance of LayoutNUWA is much when domain confusion setting is applied compared to when it is not used. However, when there a lack of layout prior conditions, such as in the C + S P Completion setting, the model is greatly affected by the domain confusion as such issue simply causes minor disturbances to the results, i.e., the model "yields to" the domain confusion under such circumstance. On one hand, this toy experiment reflects the significant impact of prior knowledge of code templates brought to the model, which can make LLM resist such domain confusion. On the other hand, the knowledge of the code template may have overwhelmed some of the information provided by the domain condition, which can lead to the generated content being weakly associated with the domain condition when there is insufficient code template information.
In future work, especially when adapting the L.MSs to multiple layout distributions simultaneously, more attention needs to be paid to how to make models distinguish different domains by designing stronger constraints.
RICO -> Magazine
Magazine
RICO
Tasks
Align. (-+
FID
Align. (+)
FID (L)
C+ -
0.001
8.027
Domain-Confusion
0.472
6.755
0.007
2.870
C S w/o Domain-Confusion
0.196
10.623
0.014
4.569
0.359
8.791
0.004
2.524
Completion
0.217
9.283
0.013
9.756
w/o DommiinCoonfssoo
0.416
7.572
0.007
7.542
Real Data
1.693
6.695
0.004
6.250
Table 8: Model Performance under the Mixed Domain setting, where A + B indicates using A domain condition plus the B layout template. "_" denotes the failed generation.
G MORE TRAINING DETAILS
We deployed and trained the model based the open-source LLM training framework LLaMA-X and HuggingFacq We optimized the training using DeepSpeed Zero3 technology (Rajbhandari al. 2020 For the DS settings, we the learning rate 5e-5, For the DA settings, we the learning rale to 5e-6 to prevent model explosion. We trained on 128 V100 units until the loss on the development converged, at which point we stopped the training. We will open-source all the training details of this article after the anonymization period.
'huts://gttub.cco http:://gghtuu..coo///cccrrrrrrt wwwp//AeeheheeeeeeAcccaaa -ama-
hhtps:/hhuggingface..v
18

---

Published as a conference paper at ICLR 2024
GPT-4
LayoutNUWA
c>S+P
GPT-4
LayoutNUWA
Completion
Golden
LayoutNUWA
c+
Golden
Figure 10: Comparison of rendered graphic layouts between GPT4 and LayoutNUWA on the Mag-
azine dataset.
19

---

Published as conference paper at ICLR 2024
(Durs) LayoutNUWA
(Ours) LayoutNUWA
Catagory
LayoutTrans
MaskGIT
LayoutTrans
LayoutDM
MaskGIT
Real
Text
Inage
Inage x 10
Icon X
Icon X
C S+P
Neb x
Text x
Text 10
Inage x
Button
Buttton x
Ican
Indic
Switch
MaskiGIT
Real
Category
Layouttrans
LayoutDM
MaskSIT
Text
x
Inage x
Icon
C+S-1 A
Toolbar
RICO
Multi x
Text
Text
Inage
Image
Icon
Icon
Button
Buttoon
List
Input
wab View
Card x
Draseer X
Adver x
Adven x
Toolbar
Layouttrans
LayoutDM
MaskGIT
LayoutNUWA
LayoutDM
MaskGIT
Real Design
Text 10
Inage
Image 11
Icon
Completion
Button
Adver
Test X
Teat
Image
Button
Input 3
Draser
x
LayoutNUWA
LayoutTrans
LayouIDM
MaskGIT
Real Design
Category
LayoutNUWA
LayoutTrans
LayoutDM
MaskGTT
Real Design
Title
Title
x
List
List
S+
Figure
Figure
Text x 12
Title
x
List
Figure
Category
Design
Category
Text x 11
Text
Title
PubLayNet
Title
List
C+S
Figure
Table
Text 11
Title x
Title x
Table
Category
LayoutNUWA
LayoutTrans
LayourDM
MaskGIT
Real Design
Category
LayoutNUWA
LayoutTrans
LayoutDM
MaskGrT
Real Design
Text
Title
Title x
Completion
Text x
Text
Title x
Figure
Table x
Figure 11: Cases generated by LayoutNUWA on the RICO and PubLayNet dataset.
20


Reviews:
[{"review_id": "BanQHaWJRQ", "reviewer": "Reviewer_7ZDJ", "summary": "This paper introduces a novel method to instruction-tune LLMs for layout generation task. Utilizing pre-trained LLMs and formulated as a code-generation task, the proposed method can build upon the semantic knowledge about geometries and code in the LLMs and achieve SoTA performance for conditional layout generation tasks. This paper further explores both domain-specific and domain-agnostic settings, demonstrating that a single LLM instruction-tuned in multiple domains can achieve SoTA performance in multiple domains.", "questions": "- Regarding the potential diversity and overfitting, I wonder if the author(s) can show whether the model can generate diverse layouts during the sampling process, given the same input conditions. Moreover, I wonder if the 'real design' in the Figures (e.g., Fig 5) are taken from the training or validation set? It could indicate overfitting given the high similarity between real designs and the generated designs in LayoutNUWA, especially given the LLMs  are significantly larger in size over prior work.\n\n- Have the author(s) also explored unconditional generation? It would be good to understand the capability of the models to generate diverse layouts unconditionally, following the point above.\n\n- While the domain-agnostic model is impressive, I wonder if the author(s) have observed domain confusion? Such as the model generating a magazine layout (when prompted to generate a mobile layout (from RICO).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Achieved SoTA in Layout Generation tasks in multiple domains with a single LLM. \n- Extensive experimentation with multiple baselines, model variants, tasks and datasets.\n- Enabled LLMs to generate layouts, which opens up possible research avenues for complex language-based interactions in the layout generation space.", "weaknesses": "- *Limited Architectural Novelty from Fine-tuning LLMs*\n- *Constrained Tasks and Conditions*: While the evaluated tasks are established and important for fairly comparing against existing work, I believe the paper can be significantly improved by exploring more complex, language-based interactions, given that LLMs are used to perform the layout generation task for the first time. It would significantly improve the contribution if the author(s) can explore (could be preliminary) whether LLMs can fuse its natural language / common task knowledge with layout generation, such as having a conversation with users about their layout generation constraints, or specifying the use-cases of the layouts.\n- *Potential limited diversity and overfitting*: The qualitative generation results appear very similar to the real design, and it is unclear whether the model can generate diverse layouts given a single condition, or if the model has overfitted to a certain specific layout. I suggest the author(s) to include the most similar training example to the generation (by DocSim) in a figure in the appendix. Please also see questions below for addressing my concerns.\n\nBecause of these concerns, I am currently on the fence about accepting this paper, but I am willing to raise my score if these concerns are addressed adequately in the rebuttals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SnfVIE7dzy", "reviewer": "Reviewer_dDf9", "summary": "This paper tackles the task of graphic layout generation. Previous works usually treat this task as a numerical optimization problem, whereas this paper proposes to leverage the knowledge of LLMs for layout generation. Specifically, the authors propose a Code Instruct Tuning framework with three components: Code Initialization, Code Completion, and Code Rendering. The proposed method achieves strong performance compared with current SOTAs on three datasets, RICO, PubLayNet, and Magazine.", "questions": "The authors mention that LayoutNUWA suffers from the error propagation problem. Some visualizations/examples might be useful to illustrate this point.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well organized and easy to read.\n\n- Utilizing LLMs for layout generation seems a good way to tackle this graphic layout generation task. The code template also supports three different variants for code completion task.\n\n- The paper provides strong results compared with baseline models, as well as several ablation studies including LLaMA2/CodeLLaMA, Domain-Specific/Domain-Agnostic settings, and code/numerical output formats.", "weaknesses": "- This paper focus on graphic layout generation. Even though LLMs are not used for this specific task, layout generation via LLMs seems not novel [1,2,3]. Therefore, the novelty of this paper is not very strong. \n\n\n\n[1] Cho, Jaemin, Abhay Zala, and Mohit Bansal. \"Visual Programming for Text-to-Image Generation and Evaluation.\" arXiv preprint arXiv:2305.15328 (2023)\n\n[2] Feng, Weixi, et al. \"LayoutGPT: Compositional Visual Planning and Generation with Large Language Models.\" arXiv preprint arXiv:2305.15393 (2023)\n\n[3] Lian, Long, et al. \"LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models.\" arXiv preprint arXiv:2305.13655 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aSUC7oyEuy", "reviewer": "Reviewer_qsQV", "summary": "This paper introduces a new approach for the layout generation task. The proposed method treats layout design as an HTML code formulation and harnesses the power of a large language model to fill in the masked attributes. The proposed method operates in three primary phases: it begins by transforming the given conditions into an initialized HTML code. Subsequently, the language model is deployed to predict the final output including the masked attributes. Finally, this output is translated back into a layout design. The experiments on three datasets, (RICO, PubLay, and Magazine) show the improvement over the state-of-the-art methods in different settings.", "questions": "As I understand, the authors also pass the layout size as the initialization part. However, this is not given in for example LayoutDM to the best of my knowledge and can be a strong signal to the model. Can you clarify on this part?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper presents is easy to follow. The proposed approach to utilizing the capabilities of LLMs for layout design, particularly through the formulation of HTML codes, is novel to the best of my knowledge. Moreover, the proposed technique surpasses current state-of-the-art methods, especially when tested on the Magazine dataset.", "weaknesses": "A significant limitations that in my view is its inability to incorporate non-categorical data, such as image embeddings. This constraint considerably narrows the method's applicability in real-world situations, especially when users input attributes that aren't readily translatable for language models.\n\nThe presentation could benefit from further refinement. Specific implementation details are missing, including the number of training iterations, batch size, learning rate, etc. Furthermore, Figure 3's organization makes it challenging to visually compare the proposed method against both the original design and LayoutDM as the state-of-the-art method.\n\nGiven that prior techniques were optimized for domain-specific settings, the enhancements seen in the \"LayoutNUWA-*-DS\" variants don't come across as particularly robust, as evidenced by metrics like the FID in the RICO dataset. This limitation, coupled with the inherently slower processing speed of LLMs compared to other methods, may limit the method's broader application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qe1HyJsWUo", "reviewer": "Reviewer_8sLn", "summary": "The paper tackles conditional layout generation by formulating it as code generation and leveraging the prior knowledge of large language models (LLMs) to improve the performance of layout generation.\n\nThe main technical contribution is a code instruct tuning (CIT) method specialized in conditional layout generation, which consists of three components: 1) a code initialization module to convert input conditions into a masked HTML code; 2) a code completion module to predict the masked values using LLM; 3) a code rendering module to render the completed code into the final layout.", "questions": "1. In Eq.(3), what does M_j^t denote? What is the definition of L on the right hand side of the equation?\n\n2. From the results in Table 1 and 2, the performance improvement on RICO and PubLayNet is not as significant as on the Magazine dataset. What are possible reasons behind this?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "1. The idea of leveraging LLMs for conditions layout generation is very interesting, which has not been explored previously. The attempt of this paper on bridging LLMs and layout generation is inspiring, which provides a new perspective to layout modeling.\n\n2. The performance improvement on the Magazine dataset is impressive.", "weaknesses": "1. The amount of technical contribution is limited. While the paper claims that a novel CTI approach is contributed, the technical innovation of the CTI is not significant since it only involves a simple solution to converting layout data to a LLMs-readable format (template generation), and some small modifications to adapt LLMs to layout generation (such as element order permutation and multi-task modeling). Therefore, the technical contribution that the paper can make to the ICLR community may not be significant. \n\n2. The experiment setup is questionable. First, the proposed method, in theory, can support both conditional and unconditional layout generation (e.g., by setting the task instruction in Sec. 3.2.1 properly and tuning the LLM to generate the entire code instead of just the masked values). However, only the experiments on conditional generation are presented now, and there is no evaluation on unconditional generation that most of the previous works primarily aim for. Thus, a (potentially) key capability of the proposed method (also a problem interesting to the layout modeling community) is ignored in the paper. Second, LayoutGAN++ is only trained for the “C -> S + P” task. It is unclear how it is adapted to solve the other two conditional generation tasks in the experiments. Third, LayoutDM is only trained for unconditional generation, which is then adapted to conditional generation without any fine-tuning or re-training. In contrast, the proposed method is especially fine-tuned for various conditional generation tasks. As a result, direct comparison between LayoutDM and the proposed method under such a setting is unfair.\n\n3. The performance on RICO and PubLayNet is unsatisfactory. As shown in Table 2, the advantage of the proposed method over SOTA methods is marginal on average. This is somewhat below expectation given the fact that the proposed method is based on large pre-trained models and has a chance to observe more training data (under the DA setting).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)