SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(1) background image, image, text, text button*2, text 2 center canvas, text button 3 center canvas, text 2 smaller background image 0
(2) image, text*4, text button, text 4 bottom image 0, text 3 larger text 2, text button 5 bottom text 4
(3) icon, list item*6, text*7, toolbar, list item 2 top canvas, list item 4 center canvas, list item 6 center canvas, list item 1 bottom icon 0, list item 2 larger icon 0, list item 4 larger icon 0, text 7 right icon 0, toolbar 14 top list item 1, list item 3 equal list item 2, list item 5 bottom list item 2, text 7 top list item 2, text 13 smaller list item 2, text 7 smaller list item 3
(4) input*2, text*3, text button*4, text button 7 bottom canvas, text 2 equal input 0, text 4 top input 0, text button 6 smaller input 0, text 3 smaller input 1, text button 7 smaller input 1, text button 7 bottom input 1, text button 6 larger text 3, text button 6 bottom text 4
(5) image, text*3, text button*3, text 2 top canvas, text 3 bottom image 0, text 3 bottom text 2, text button 5 bottom text 2, text button 5 top text 3
(6) icon*3, image, input*2, text button*3, toolbar, input 5 center canvas, text button 6 center canvas, icon 2 equal icon 1, text button 6 larger icon 1, text button 8 larger icon 1, input 4 bottom image 3, text button 8 smaller image 3, text button 7 bottom input 4, text button 8 larger input 5, text button 8 bottom input 5, text button 8 larger text button 6
(1)
Figure 11: Qualitative results of Gen-R on RICO and PubLayNet. The element relationship constraints are in the table.
(a) RICO
(3)
(1) table, text*4, text 4 bottom text 1, text 4 smaller text 3
(2) figure, text*3, text 2 smaller figure 0, text 2 bottom figure 0
(3) figure, text*5, text 1 center canvas, text 5 bottom canvas, text 5 bottom text 2
(4) text*13, title*2, title 13 top canvas, text 4 bottom text 0, text 2 bottom text 1, text 4 bottom text 1, text 6 smaller text 1, text 9 larger text 1, title 13 top text 1, title 14 smaller text 1, title 13 smaller text 2, text 8 larger text 3, text 9 bottom text 3, text 6 bottom text 4, text 8 bottom text 4, text 10 larger text 4, text 9 bottom text 6, text 12 larger text 6, title 13 larger text 6, text 10 larger text 8, text 12 smaller text 9, text 12 smaller text 10, title 13 smaller text 10, title 14 top text 10, text 12 smaller text 11
(5) figure, text*5, title, text 5 smaller figure 0, text 5 bottom figure 0, text 5 smaller text 2, title 6 bottom text 3, title 6 center text 5
(6) figure, list*2, text, text 3 bottom figure 0, list 2 right list 1
C.4 Layout Completion
(7)
Figure 12: Qualitative results of completion on RICO and PubLayNet.
(9)
(b) PubLayNet
C.5 Layout Refinement
(a) RICO
(2)
(4)
Figure 13: Qualitative results of refinement on RICO and PubLayNet. Note that each group has two layouts. The left one is the noisy layout, and the right one is the refined layout.
(5)
(b) PubLayNet
C.6 Content-Aware Layout Generation
(7)
Figure 14: Qualitative results of content-aware layout generation on PosterLayout.
C.7 Text-to-Layout
Textual Descriptions
(1) A page to file an insurance claim. There should be two text boxes. One is for the title 'Need to file an insurance claim' and the other is a short description. Two buttons are necessary to click to file or manage the claim.
(2) A footer page showing the homepage of a company. There should be five entries of text links. And the text links should be unfolded. Users can get access to different info through these links.
(3) A page to introduce an online editor. The page should contain an image, and the slogan "Online Infographics, Diagrams, & Charts Maker" to have a brief introduction, and a further description of the specific service of the editor. Then a button is needed for the user to click to learn info.
(4) A page for introducing the home tours. The page should have a title, "home tours". Also, there are four groups and each group has an image and a title.
(5) A page for introducing the service provided by a website. The page should have three groups of information sets: "Web Development" "Social Media", and "Link Building". Each group should include a small image, a title, a further introduction, and a link "READ MORE" for more info by clicking it.
(6) A header page for navigation of an online shop. The page should have an announcement to explain the product is out of stock, a logo, and two links "HOME" and "BUY DITTO HANGERS" for the user to return to the homepage and view products to shop. Also, two buttons for search and shopping cart are needed.
Figure 15: Qualitative results of text-to layout on WebUI. The textual descriptions are in the table.
D Prompt Examples
Here, we show a prompt example for each layout generation task, including Gen-T (Table 11), Gen-TS (Table 12), Gen-R (Table 13), completion (Table 14), refinement (Table 15), content-aware layout generation (Table 16), and text-to-query (Table 17).
Please generate a layout based on the given information.
Task Description: generation conditioned on given element types
Layout Domain: android layout
Canvas Size: canvas width is 90px, canvas height is 160px
Exemplar 1
Element Type Constraint: image | image | image | image | text | text | text button
<div class="image" style="left:15px; top:42px; width:51px; height:82px"></div>
<div class="image" style="left:52px; top:77px; width:22px; height:50px"></div>
<div class="image" style="left:54px; top:96px; width:18px; height:14px"></div>
<div class="image" style="left:0px; top:143px; width:90px; height:1px"></div>
<div class="text" style="left:34px; top:15px; width:21px; height:7px"></div>
<div class="text" style="left:22px; top:22px; width:44px; height:9px"></div>
<div class="text button" style="left:2px; top:147px; width:41px; height:10px"></div>
</body
Exemplar 2
Element Type Constraint: image | image | image | image | pager indicator | text | text | text button | text button | text button
<div class="image" style="left:0px; top:5px; width:90px; height:93px"></div>
<div class="image" style="left:30px; top:8px; width:29px; height:10px"></div>
<div class="image" style="left:38px; top:86px; width:12px; height:12px"></div>
<div class="image" style="left:32px; top:86px; width:24px; height:12px"></div>
<div class="pager indicator" style="left:0px; top:119px; width:90px; height:5px"></div>
<div class="text" style="left:0px; top:98px; width:90px; height:10px"></div>
<div class="text" style="left:0px; top:109px; width:90px; height:9px"></div>
<div class="text button" style="left:0px; top:124px; width:45px; height:10px"></div>
<div class="text button" style="left:45px; top:124px; width:45px; height:10px"></div>
<div class="text button" style="left:0px; top:138px; width:90px; height:11px"></div>
Element Type Constraint: icon | image | image | text | text | text button | toolbar
(Generated by LLMs)
Table 11: A prompt example of Gen-T on RICO.
Task Description: generation conditioned on given element types and sizes
Element Type and Size Constraint: icon 12 12 | image 0 0 | input 81 10 | input 81 10 | text 81 4 |
text 224 | text button 8510 | text button 104 | toolbar 9012
<div class="icon" style="left:0px; top:5px; width:12px; height:12px"></div>  
<div class="image" style="left:12px; top:11px; width:0px; height:0px"></div>  
<div class="input" style="left:4px; top:40px; width:81px; height:10px"></div>  
<div class="input" style="left:4px; top:28px; width:81px; height:10px"></div>  
<div class="text" style="left:4px; top:23px; width:81px; height:4px"></div>  
<div class="text" style="left:13px; top:9px; width:22px; height:4px"></div>  
<div class="text button" style="left:2px; top:62px; width:85px; height:10px"></div>  
<div class="text button" style="left:75px; top:43px; width:10px; height:4px"></div>  
<div class="toolbar" style="left:0px; top:5px; width:90px; height:12px"></div>  
</body>  
</html>
Element Type and Size Constraint: card 86 41 | icon 12 12 | input 64 12 | input 78 12 | input 78 9 | input 61 9 |
text 15 5 | text button 25 7 | text button 13 7 | text button 62 3 | toolbar 90 12
<div class="card" style="left:1px; top:19px; width:86px; height:41px"></div>  
<div class="icon" style="left:0px; top:5px; width:12px; height:12px"></div>  
<div class="input" style="left:5px; top:36px; width:64px; height:12px"></div>  
<div class="input" style="left:5px; top:23px; width:78px; height:12px"></div>  
<div class="input" style="left:5px; top:23px; width:78px; height:9px"></div>  
<div class="input" style="left:5px; top:36px; width:61px; height:9px"></div>  
<div class="text" style="left:15px; top:8px; width:15px; height:5px"></div>  
<div class="text button" style="left:60px; top:51px; width:25px; height:7px"></div>  
<div class="text button" style="left:70px; top:38px; width:13px; height:7px"></div>  
<div class="text button" style="left:13px; top:62px; width:62px; height:3px"></div>  
<div class="toolbar" style="left:0px; top:5px; width:90px; height:12px"></div>  
</body>  
</html>
Element Type and Size Constraint: icon 12 12 | input 83 9 | input 83 9 | text 83 8 | text button 19 9 |
text button 775 | toolbar 90 12
Table 12: A prompt example of Gen-TS on RICO.
Task Description: generation conditioned on given element relationships
Element Type Constraint: image | image | image | text | text | text | text | text button | toolbar
Element Relationship Constraint: text 5 bottom canvas | image 1 larger image 0 | text 3 larger image 0 | text 5 larger image 0 | toolbar 8 larger image 0 | image 2 equal image 1 | text 4 smaller image 2 | text 6 smaller image 2 | toolbar 8 top text 4
<div class="image" style="left:0px; top:7px; width:7px; height:7px"></div>
<div class="image" style="left:31px; top:33px; width:28px; height:29px"></div>
<div class="image" style="left:30px; top:101px; width:28px; height:29px"></div>
<div class="text" style="left:8px; top:8px; width:28px; height:5px"></div>
<div class="text" style="left:24px; top:66px; width:40px; height:5px"></div>
<div class="text" style="left:18px; top:133px; width:52px; height:5px"></div>
<div class="text" style="left:18px; top:140px; width:51px; height:7px"></div>
<div class="text button" style="left:75px; top:5px; width:14px; height:11px"></div>
<div class="toolbar" style="left:0px; top:5px; width:90px; height:11px"></div>
Element Type Constraint: text | text | text | text | text button
Element Relationship Constraint: text 3 bottom text 0 | text 2 equal text 1
<div class="text" style="left:0px; top:7px; width:90px; height:5px"></div>
<div class="text" style="left:3px; top:19px; width:83px; height:30px"></div>
<div class="text" style="left:3px; top:57px; width:83px; height:30px"></div>
<div class="text" style="left:3px; top:95px; width:83px; height:52px">
<div class="text button" style="left:0px; top:148px; width:90px; height:11px"></div>
Element Type Constraint: icon | image | text | text | text | text | text button | text button
Element Relationship Constraint: text 3 top canvas | text 5 top canvas | text 2 right icon 0 | text button 6 bottom icon 0 |
text 3 bottom image 1 | text button 7 bottom text 4
Table 13: A prompt example of Gen-R on RICO.
Task Description: layout completion
Partial Layout: image 2154740
<html>
<body>
<div class="image" style="left:21px; top:5px; width:47px; height:40px"></div>
<div class="text button" style="left:2px; top:53px; width:84px; height:15px"></div>
<div class="image" style="left:7px; top:74px; width:9px; height:5px"></div>
<div class="text" style="left:19px; top:74px; width:67px; height:5px"></div>
<div class="text button" style="left:2px; top:85px; width:84px; height:14px"></div>
<div class="text button" style="left:1px; top:104px; width:86px; height:12px"></div>
<div class="text button" style="left:1px; top:136px; width:86px; height:11px"></div>
</body>
</html>
Partial Layout: image 1755611
<html>
<body>
<div class="image" style="left:17px; top:5px; width:56px; height:11px"></div>
<div class="image" style="left:0px; top:17px; width:90px; height:48px"></div>
<div class="text" style="left:2px; top:65px; width:86px; height:48px"></div>
<div class="image" style="left:0px; top:108px; width:90px; height:5px"></div>
<div class="pager indicator" style="left:38px; top:114px; width:12px; height:8px"></div>
<div class="text button" style="left:3px; top:124px; width:82px; height:13px"></div>
<div class="text button" style="left:62px; top:137px; width:17px; height:10px"></div>
<div class="text" style="left:10px; top:140px; width:51px; height:6px"></div>
</body>
</html>
Partial Layout: image 12 10 65 32
Table 14: A prompt example of layout completion on RICO.
Task Description: layout refinement
Noise Layout: advertisement 11 18 70 11 | icon 76 5 11 11 | icon 0 6 12 10 | image 16 8 13 11 | text 30 3 2 1 5 |
text 29 11 23 4 | toolbar 0 5 88 16 | web view 9 16 69 12 | web view 11 17 70 12 | web view 0 20 90 140
<div class="advertisement" style="left:10px; top:18px; width:70px; height:11px"></div>
<div class="icon" style="left:77px; top:6px; width:12px; height:11px"></div>
<div class="icon" style="left:0px; top:5px; width:12px; height:13px"></div>
<div class="image" style="left:15px; top:6px; width:14px; height:11px"></div>
<div class="text" style="left:30px; top:6px; width:21px; height:6px"></div>
<div class="text" style="left:30px; top:12px; width:23px; height:5px"></div>
<div class="toolbar" style="left:0px; top:5px; width:90px; height:13px"></div>
<div class="web view" style="left:10px; top:18px; width:70px; height:11px"></div>
<div class="web view" style="left:0px; top:18px; width:90px; height:141px"></div>
Noise Layout: advertisement 048911 | background image 0489145 | icon 41767 | icon 111946 | image 1855 |
image 0 13 20 10 | text 35 8 18 5 | text button 80 6 7 3 | text button 16 14 64 8 | text button 82 14 9 7 |
text button 10 29 68 11 | text button 0 39 88 12 | web view 10 2 69 12 | web view 9 6 69 10
<div class="advertisement" style="left:0px; top:5px; width:90px; height:10px"></div>
<div class="background image" style="left:0px; top:5px; width:90px; height:144px"></div>
<div class="icon" style="left:5px; top:19px; width:4px; height:4px"></div>
<div class="icon" style="left:11px; top:19px; width:4px; height:4px"></div>
<div class="image" style="left:2px; top:7px; width:5px; height:5px"></div>
<div class="image" style="left:0px; top:16px; width:21px; height:7px"></div>
<div class="text" style="left:35px; top:7px; width:18px; height:5px"></div>
<div class="text button" style="left:81px; top:8px; width:6px; height:5px"></div>
<div class="text button" style="left:16px; top:16px; width:63px; height:10px"></div>
<div class="text button" style="left:81px; top:16px; width:8px; height:7px"></div>
<div class="text button" style="left:11px; top:27px; width:68px; height:10px"></div>
<div class="text button" style="left:0px; top:41px; width:90px; height:11px"></div>
<div class="web view" style="left:10px; top:5px; width:70px; height:10px"></div>
Noise Layout: icon 6851012 | icon 15912 | icon 8051213 | text 147562 | toolbar 059010 |
web view 0 18 90 130 | web view 0 19 90 130
Table 15: A prompt example of layout refinement on RICO.
Task Description: content-aware layout generation
Layout Domain: poster layout
Canvas Size: canvas width is 102px, canvas height is 150px
Content Constraint: left 25px, top 25px, width 30px, height 12px
Element Type Constraint: logo | text | underlay | text | text
<div class="logo" style="left:34px; top:14px; width:66px; height:23px"></div>
<div class="text" style="left:10px; top:25px; width:94px; height:36px"></div>
<div class="underlay" style="left:18px; top:37px; width:85px; height:48px"></div>
<div class="text" style="left:36px; top:40px; width:64px; height:45px"></div>
<div class="text" style="left:28px; top:48px; width:74px; height:53px"></div>
Content Constraint: left 23px, top 60px, width 56px, height 69px
Element Type Constraint: logo | text | underlay | text
<div class="logo" style="left:35px; top:0px; width:66px; height:10px"></div>
<div class="text" style="left:15px; top:22px; width:86px; height:33px"></div>
<div class="underlay" style="left:29px; top:37px; width:73px; height:49px"></div>
<div class="text" style="left:35px; top:40px; width:67px; height:47px"></div>
Content Constraint: left 26px, top 62px, width 50px, height 60px
Element Type Constraint: logo | text | text | text | underlay | text
Table 16: A prompt example of content-aware layout generation on PosterLayout.
Task Description: text-to layout
Layout Domain: web layout
Canvas Size: canvas width is 120px, canvas height is 120px
Text: A header page for the company Fashionably Latellc. On the page, there should include a name of the company, three navigation links "Home" "Our Fashionably Items" and "Return/Refund Policy" for the user to click to return to the homepage, view items of the company, and know the return or refund policy. Besides, it is necessary to have three icons for login, search, and shopping cart.
<html>
<body>
<div class="icon" style="left:101px; top:3px; width:2px; height:2px"></div>
<div class="icon" style="left:106px; top:3px; width:2px; height:2px"></div>
<div class="icon" style="left:110px; top:2px; width:5px; height:5px"></div>
<div class="link" style="left:35px; top:3px; width:4px; height:2px"></div>
<div class="link" style="left:42px; top:3px; width:17px; height:2px"></div>
<div class="link" style="left:62px; top:3px; width:16px; height:2px"></div>
<div class="title" style="left:5px; top:3px; width:24px; height:3px"></div>
</body>
</html>
Text: A header page for a website Png AAA. There should have a logo on the left, and three links "LOG IN" "SIGN UP", and "IMPORT", so the user can click them to log in, create an account, and upload something. <html> <body> <div class="link" style="left:88px; top:2px; width:5px; height:2px"></div> <div class="link" style="left:97px; top:2px; width:6px; height:2px"></div> <div class="link" style="left:110px; top:2px; width:6px; height:2px"></div> <div class="logo" style="left:2px; top:1px; width:15px; height:4px"></div> </body> </html>
Text: A header page of the website "comment". On the page, there should include a logo of the website. Five links ("Latest", "Top100", "About", "Register", and "Login") a button "Create" and an icon are on the page. (Generated by LLMs)
Table 17: A prompt example of text-to layout on WebUI.