SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training layout-aware LM. To obtain layout-aware LM, we use Vicuna 13B [16], a public state-of-the-art chatbot language model finetuned from LLaMA [51]. We use parameter-efficient finetuning with LoRA [52] to preserve the original knowledge of LM and save memory during training and inference. We collect the text layout pair annotations from training sets of three public datasets: Flickr30K entities [17], MS COCO instances 2014 [18], and PaintSkills [19], totaling 1.2M examples. We use randomly selected 2,000 examples for the validation set and use the rest for training. For the object/count generation step, we set the maximum counts for a single object class as 7. We train
Table 8: VPEval accuracy and FID of VPGen (Vicuna 13B + GLIGEN) and its closest baseline Stable Diffusion v1.4. F30: Flickr30K Entities, C: COCO, P: PaintSkills.
Model	VPEval Skill-based Acc. (%) ↑	VPEval Open-ended Acc. (%) ↑	FID (COCO 30K) ↓
Stable Diffusion v1.4	37.7	70.6	16.5
VPGEN (F30)	43.9	71.0	15.9
VPGEN (F30+C+P)	51.0	68.3	20.1
Table 9: Training Vicuna 13B with real v.s. synthetic data. F30: Flickr30K Entities, P: PaintSkills.
Model	VPEval skill-based Acc. (%) ↑	VPEval open-ended Acc. (%) ↑
VPGEN (F30; real)	43.9	71.0
VPGEN (P; synthetic)	46.0	65.6
Vicuna 13B with per-gpu batch size 96 (= 24 batch x 4 gradient accumulation). When training Vicuna 13B with Flickr30K+COCO+PaintSkills dataset, we train the model for 2 epochs. When training only with Flickr30K dataset, we train the model for 6 epochs, to rough match the training time. Training takes 26 hours with 4 A6000 GPUs (each 48GB).
GLIGEN inference details. We use Huggingface Diffusers [69] implementation of GLIGEN  $[20]^2$ . Following the default configuration, we use gligenscheduled_sampling_beta = 0.3, num_inference_steps = 50, and fp16 precision during inference.
Inference time. On a single A6000 GPU, VPGEN takes around 6s (2s for Vicuna 13B and 4s for GLIGEN) to generate an image.
E.2 VPEVAL Details
VPEVAL prompts and evaluation programs. Our skill-based benchmark has an almost uniform distribution of objects and relations among the text prompts. This ensures that generation models cannot achieve high scores by performing highly on a few common objects, relations, and counts, etc. We create skill-specific prompts by composing templates with lists of objects (from 80 COCO objects), counts (1-4), relations (2D: above, below, left, right; 3D: front, behind), scale (smaller, bigger, same), and text (31 unique words). Our evaluation can be extended to any number of objects/counts/relations/etc. In total, our skill-based benchmark has 400/1000/1000/1000/403 prompts for the object/count/spatial/size/text skills, respectively. In Table 11, we show templates and evaluation programs used for each skill.
Captioning/VQA/CLIP baseline details. For captioning, we generate a caption with BLIP-2 [56] and calculate text similarity metrics between the caption and the original text prompt. For VQA, we give the BLIP-2 (Flan-T5 XL) the image, a question, and a yes/no answer choice, with a prompt template "Question: {question} Choices: yes, no Answer:". The {question} is a formatted version of the text prompt (e.g., "a photo of a dog"  $\rightarrow$  "is there a dog in the image?"). For the spatial and scale skill, we ask three questions: if object A is present (e.g., "is there a dog in the image?") if object B is present (e.g., "is there a cat in the image?"), and if the relationship between them is true (e.g., "is the cat behind the dog?"). We mark each image as correct if the VQA model outputs 'yes' for all three questions. The first two questions prevent accidental false positives if an
Table 10: Step-wise error analysis of VPGen (Vicuna+GLIGEN). For the Spatial skill, 'front' and 'behind' splits are skipped in this table since our Vicuna 13B does not generate depth information.
Skills	Vicuna 13B	GLIGEN
Object Recall (%)↑	Object Count (%)↑	Layout Accuracy (%)↑	Image Accuracy (%)↑
Object	99.2	98.8	99.2	96.8
Count	99.1	98.8	98.8	72.2
Spatial	98.3	98.2	87.5	63.3
Scale	92.8	92.6	38.2	26.3
$^{2}$ https://github.com/gligen/diffusers
Error in Layout Generation (Vicuna 13B)
(a) Generating unspecified objects
a cat jumping in the air to get onto a table
A vehicle composed of two wheels held in a frame one behind the other, propelled by pedals and steered with handlebars attached to the front wheel.
Figure 12: Text-to-Layout generation error examples. The Vicuna 13B sometimes (a) generates objects that are not specified in prompts (e.g., 'chair' in the 1st example) or (b) misses objects that are mentioned in prompts (e.g., 'building' in the 5th example).
A traffic light and a signpost at a crossroads intersection near a waterway.
(b) Missing some objects
a library filled with kids reading books
Error in Layout-to-Image Generation (GLIGEN)
(a) Wrong objects
A photo of a suitcase and bench; bench is left to suitcase
Figure 13: Layout-to-Image generation error examples. The GLIGEN sometimes (a) generates wrong objects (e.g., 'suitcase', instead of 'bench', in the 1st example), (b) misses some objects (e.g., 'umbrella' in the 3rd example), or (c) misses object details (e.g., 'closed oven' instead of 'oven with the door open' in the 5th example).
(b) Missing objects
An umbrella on top of a spoon
A car and a truck sitting a red light
(c) Wrong object details
A man standing by an oven with the door open
object is missing. For CLIP (ViT-B/32) [32], we encode the generated image and text prompts, and take the cosine similarity between the visual and text features as the final score.
Inference time. On a single A6000 GPU, each VPEVAL module typically takes less than 1s to run.
Program generation API cost. Using GPT-3.5 Turbo<sup>3</sup> as the program generator, VPEVAL takes about $0.007 (less than 1 cent) to generate programs from a prompt (4K input tokens, 0.5K output tokens).
F FIFA Prompt Element Analysis Details
To better understand the characteristics of the TIFA prompts [36] that we use in open-ended evaluation, we randomly sample 20 prompts and label whether each prompt has any elements that require an understanding of spatial layouts, as mentioned in the main paper Sec. 5.3. Each prompt has multiple elements about objects, attributes, spatial layouts, etc. For example, "a zebra to the right of a fire hydrant" has the elements: 'zebra', 'right', and 'fire hydrant'. From the 20 prompts, there are 81 total elements, and only 11/81 (13.6%) are relevant to object layouts (e.g., 'right'). The remaining 70 elements are all related to objects or attributes (e.g., "a white cat with black ears and markings" has the elements 'cat', 'white cat', 'ears', 'black ears', 'markings', and 'black markings').
<sup>3</sup>https://openai.com/pricing
(a) A realistic photo of a Pomeranian dressed up like a 1980s professional wrestler with neon green and neon orange face paint and bright green wrestling tights with bright orange boots
(b) a circle with  $3\mathrm{cm}$  radius and a circle with  $6\mathrm{cm}$  radius
Figure 14: Images generated by VPGEN (Vicuna+GLIGEN) from challenging, non-canonical prompts.
Figure 15: Examples of our VPEVAL evaluation programs and module outputs. Each module covers different parts of the text prompts (see Appendix D for discussion).
G Limitation and Broader Impacts
Limitations. Since our LMs were primarily trained with English-heavy datasets, they might not well understand prompts written in non-English languages. Likewise, as our generation/evaluation modules were primarily trained with natural images, they might not well handle images in different domains (e.g., fine-arts, medical images, etc). Note that our framework supports easy extensions so that users can update existing modules or add new ones with better domain-specific knowledge.
Generating evaluation programs using LLMs can be costly both in terms of computational resources and price (API cost). Thus, we release the evaluation programs so that users do not have to re-generate the evaluation programs. In addition, we will release a public LM (finetuned for evaluation program generation using ChatGPT outputs) that can run on local machines.
Broader Impacts. While our interpretable VPGEN framework can be beneficial to many applications (e.g., user-controlled image generation/manipulation and data augmentation), it could also be used for potentially harmful cases (e.g., creating false information or misleading images), like other image generation frameworks.
Table 11: The prompt templates and evaluation programs for skill-based evaluation. We also insert grammar tokens like  $< \mathrm{a} >$  that are replaced with "an" or "a" to maintain correct grammar.
Skill	Object	Count	Spatial	Scale	Text Rendering
Prompt	<objA><a><objA>
a photo of <a><objA>
an image of <a><objA>
a picture of <a><objA>	<NI><objA><a>
a photo of <NI><objA><a>
a picture of <NI><objA><a>
an image of <NI><objA><a><NIEN><objA><a>
<NIEN><objA><a><NIEN><objA><a>
a photo of <NIEN><objA><a><NIEN><objA><a><NIEN><objA><a>
a picture of <NIEN><objA><a><NIEN><objA><a><NIEN><objA><a>
an image of <NIEN><objA><a><NIEN><objA><a><NIEN><objA><a>	<NI><objB><a><NI><objB><a><NI><objB><a><NI><scale>	<NI><objB><a><NI><objB><a><NI><scale>	a sign that reads ' <text>
a book cover that reads ' <text>
a poster that reads ' <text'
a sign that says ' <text'
a book cover that says ' <text'
a poster that says ' <text'
a storefront with ' <text' written on it
a storefront with ' <text' written
a storefront with ' <text' displayed
a piece of paper that says ' <text'
a piece of paper that reads ' <text'
a piece of paper that says ' <text' on it
a piece of paper that reads ' <text' on it
Evaluation Program | objectEval(img, <objA>)	countEval(img, <objA>, <N>)	spatialEval(img, <objA>, <objB>, <reb>)	scaleEval(img, <objA>, <objB>, <scale>)	textEval(img, <text>)
H License
We will make our code and models publicly accessible. We use standard licenses from the community and provide the following links to the licenses for the datasets, codes, and models that we used in this paper. For further information, please refer to the links.
FastChat (Vicuna): Apache License 2.0
GLIGEN: MIT
Grounding DINO: Apache License 2.0
DPT:MIT
LAVIS (BLIP-2): BSD 3-Clause
TIFA: Apache License 2.0
minDALL-E: Apache License 2.0, CC-BY-NC-SA 4.0
DALL-E mini (DALL-E Mega): Apache License 2.0, MIT
EasyOCR: Apache License 2.0
OpenAI API (ChatGPT): MIT
Diffusers: Apache License 2.0
Transformers: Apache License 2.0