Visual Programming for Step-by-Step Text-to-Image Generation and Evaluation
Authors: Jaemin Cho
Abhay Zala
Mohit Bansal
Venue: NeurIPS 2023 poster (2023)
Keywords: text-to-image generation; visual programming; text-to-image evaluation; step-by-step generation; interpretability; explainability

Abstract: As large language models have demonstrated impressive performance in many domains, recent works have adopted language models (LMs) as controllers of visual modules for vision-and-language tasks. While existing work focuses on equipping LMs with visual understanding, we propose two novel interpretable/explainable visual programming frameworks for text-to-image (T2I) generation and evaluation. First, we introduce VPGen, an interpretable step-by-step T2I generation framework that decomposes T2I generation into three steps: object/count generation, layout generation, and image generation. We employ an LM to handle the first two steps (object/count generation and layout generation), by finetuning it on text-layout pairs. Our step-by-step T2I generation framework provides stronger spatial control than end-to-end models, the dominant approach for this task. Furthermore, we leverage the world knowledge of pretrained LMs, overcoming the limitation of previous layout-guided T2I works that can only handle predefined object classes. We demonstrate that our VPGen has improved control in counts/spatial relations/scales of objects than state-of-the-art T2I generation models. Second, we introduce VPEval, an interpretable and explainable evaluation framework for T2I generation based on visual programming. Unlike previous T2I evaluations with a single scoring model that is accurate in some skills but unreliable in others, VPEval produces evaluation programs that invoke a set of visual modules that are experts in different skills, and also provides visual+textual explanations of the evaluation results. Our analysis shows that VPEval provides a more human-correlated evaluation for skill-specific and open-ended prompts than widely used single model-based evaluation. We hope that our work encourages future progress on interpretable/explainable generation and evaluation for T2I models.

Visual Programming for Text-to-Image Generation and Evaluation
Jaemin Cho
Abhay Zala UNC Chapel Hill
Mohit Bansal
(jmincho, aszala, mbansal)@css unc.edu https://vp-t2:: get.ig.tttbbiii.
Abstract
As large language models have demonstrated impressive performance in many do- mains, recent works have adopted language models (LMs) as controllers of visual modules for dasion-and-languaal tasks. While existing work focuses on equipping LMs with visual understanding, we propose two novel interpretableeexplainabe visual programming frameworks for -ext- (T2I) generation and evaluation. First, we introduce VPGEN, an interpretable step-by-step T21 generation frame- work that decomposes T2I generation into three steps: object/count generation, layout generation, and image generation. We employ an LM to handle the first two steps (object/count generation and layout generation), by finetuning it on text- layout pairs. Our step-by-step T2I generation framework provides stronger spatial control than end-to-end models, the dominant approach for this task. Furthermore, we leverage the world knowledge of pretrained LMs, overcoming the limitation previous layout-guided T2I works that can only handle predefined object classes. We demonstrate that our VPGEN has improved control counts/spatial rela- tions/scales of objects than state-of-the ttthhe-rtt T2I generation models. Second, introduce VPEVAL, an interpretable and explainable evaluation framework for T2I generation based visual programming. Unlike previous T21 evaluations with a single scoring model that is accurate some skills but unreliable others, VPEAAL produces evaluation programs that invoke set of visual modules that are experts in different skills, and also provides visual+textual explanations of the evaluation results. Our analysis shows that VPEVAL provides more human- correlated evaluation for skill-specific and open-ended prompts than widely used single model- mosel-based evaluation. We hope that our work encourages future progress on interprttableeexllinnbb. generation and evaluation for T2I models.
Introduction
Large language models (LLMs) have shown remarkable performance on many natural language pro- cessing tasks, such question answering, summarization, and story generation [1; 2; 3; 4; 5; 6; 7; 8]. Recent works have shown that LLMs can also tackle certain iision-and-language tasks such as visual question answering and visual grounding, by generating visual programs that can control external vi- sual modules and combine their outputs to get the final response [9; 10; 11; 12; 13; 14; 15). However, no prior works have combined LLMs and different visual modules for the challenging text-to-image (T2I) generation task. Our work proposes two novel interpretable/xxplaimbbb visual programming (VP) frameworks combining LLMs and visual modules for T2I generation and evaluation.
First, we introduce VPGEN (Sec. 3), a new step-by-step T2I generation framework that decomposes the T2I task into three steps (object/count generation, layout generation, and image generation), where each step is implemented as module and executed an interpretable generation program, as shown
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) VPGen: Step-by-Step T2I Generation
(b) VPEval: Explainable T2I Evaluation
"two Pikachus table"
"three dogs inage"
Visual Program
Visual Program
aaporiipttoourrggptoofooossgpppp prorpt), prompt)
counttval(ing "dog", (exg"))
Generation Modules
Evaluation Modules
Object/ Count
Generation
Layout Generation
Image Generation
Object Detect
OCR
Spottal
Scale
Count
VQA
Generation Program
Generated Image
Evaluation Program
>>> soantEvall ling, "dag", *eigh)
>>> proept "buo Pikachus on a table"
>>> atgpeaunt prtdejcaurtspppeepp "toote"
"""tkaccu":
Visual+ Textual Explanation of Evaluation results
There "day" objects,
IC"ythacnu", (d, C'ptbaccn", (x3,95, (2,32) ("bable", (x1,)4,22,200
>>9 ing LaglentL proeps)
Illustration of the proposed visual programming frameworks for text-to-image (T2I) generation and evaluation. In (a) VPGEN, first generate list of objects, then object positions, and finally an image, by executing three modules step-by-step. In (b) VPEVAL, we use evaluation programs with mixture of evaluation modules that handle different skills and provide visual+textual explanation of evaluation results.
Figure I:
in Fig. 1 (a). We employ Vicuna [16]. powerful publicly available LLM, to handle the first two steps (object/count generation and layout generation), by finetuning it on text-layout pairs from multiple datasets [17; 18; 19], resulting in improved layout control for T2I generation. For the last image generation step, we use off-the-shelf pnyout-to-image generation models, e.g., GLIGEN [20]. Our generation framework provides more interpretable spatial control than the widely used end-to-end T2I models [21; 22; 23; 24J. Furthermore, VPGEN leverages the world knowledge of pretrained LMs to understand unseen objects, overcoming the limitation of previous layout-guided T21 works that can only handle predefined object classes [25; 26; 28:. In our analysis, we find that our VPGEN (Vicun++GLIGEN) could generate images more accurately following the text description (especially about object counts, spatial relations, and object sizes) than strong T2I generation baselines, such as Stable Diffusion [21] (see See. 5.2).
Next, we introduce VPEVAL (Sec. 4), a new interpretabeeeeplainabbee T2I evaluation framework based on evaluation programs that invoke diverse visual modules to evaluate different T2I skills [19] (see Sec. 4.1), and also provide visual+textual explanations of the evaluation results. Previous T2I
evaluation works have focused on measuring visual quality [29; 30] and image-text alignment with single visual module based on text retrieval [31], CLIP cosine similarity [32; 33], captioning [25], object detection [34; 19; 35], and visual question answering (VQA) [36; 37J. However, we cannot interpret the reasoning behind the scoring; i.e., why CLIP assigns higher score to an image-text pair than another. In addition, these modules are good at measuring some skills, but not reliable in other skills (e.g.. VQA/CLIP models are not good at counting objects or accurately parsing text rendered in images). In VPEVAL, we break the evaluation process into mixture of visual evaluation modules, resulting in an interpretable evaluation program. The evaluation modules are experts in different skills and provide visual+textual (i.e., multimodal) result/error explanations, as shown in Fig. 1 (b). We evaluate both multimodal LM and ceffussion-basedd T21 models with two types of prompts: (1) skill-based prompts (See. 4.3) and (2) open-ended prompts (Sec. 4.4). Skill-based prompts evaluate a single skill per image, whereas open-ended prompts evaluate multiple skills per image. We adapt large language model (GPT-3.5-Turbo [38]) to dynamically generate an evaluation program for cach open-ended prompt, without requiring finetuning on expensive evaluation program annotations.
In our skill-based prompt experiments, while all T2I models faced challenges in the Count, Spatial, Scale, and Text Rendering skills, our VPGEN shows higher scores in the Count, Spatial, and Scale skills, demonstrating its strong layout control. We also show a fine-graine fine-grained sub-split analysis on where our VPGEN gives an improvement over the baseline T2I models. In our open-ended prompt experiments, our VPGEN achieves competitive results to the T2I baselines. Many of these open- ended prompts tend to focus more on objects and attributes, which, as shown by the skill-based evaluation, T21 models perform quite well in; however, when precise layouts and spatial relationships needed, our VPGEN framework performs better (Sec. 5.3). In our error analysis, also find

---

that GLIGEN sometimes fails to properly generate images even when Vicuna generates the correct layouts, indicating that as better layout-to-image models become available, our VPGEN framework will achieve better results (Sec. 5.3). We also provide an analysis of the generated evaluation programs from VPEVAL, which proved to be highly accurate and comprehensive covering elements from the prompts (Sec. 5.4). For both types of prompts, our VPEVAL method shows a higher correlation to human evaluation than existing single model-based evaluation methods (Sec. 5.4).
Our contributions can be summarized as follows: (I) VPGEN (Sec. 3), a new step-by-step T21 generation framework that decomposes the T2I task into three steps (object/count generation, layout generation, and image generation), where each step is implemented as a module and executed in an interpretable generation program; (2) VPEVAL (Sec. 4), new interpretable/explaaabbe T2I evaluation framework based on evaluation programs that execute diverse visual modules to evaluate different T2I skills and provide ttextll explanations of the evaluation results; (3) comprehensive analysis of different T2I models, which demonstrates the strong layout control of VPGEN and high human correlation of VPEVAL (Sec. 5). We will release T2I model-generated images, VPEVAL programs, and a public LM (finetuned for evaluation program generation using ChatGPT outputs). We hope our research fosters future work on interpretableexxplainbb generation and evaluation for T2I tasks.
2 Related Works
Text-to-image generation models. In the T2I generation task, models generate images from text. Early deep learning models used the Generative Adversarial Networks (GAN) [39] framework
for this task [40; 41; 42; 31]. More recently, multimodal language models [43; 22] and diffusion
models [44; 45; 21; 46] have gained popularity. Recent advances in multimodal language models such as Parti [47] and MUSE [48], and diffusion models like Stable Diffusion [21], UnCLIP [23],
and Imagen [24], have demonstrated a high level of photorealism in zero-shot image generation.
Bridging text-to-image generation with layouts. One line of research decomposes the T2I genera- tion task into two stages: text-to-layout generation and layout-to-image generation [25; 26; 27; 28]. However, the previous approaches focus on a set of predefined object classes by training a new layout predictor module from scratch and therefore cannot place new objects unseen during training. In contrast, our VPGEN uses an LM to handle layout generation by generating jijject//countspoosttiin in text, allowing flexible adaptation of pretrained LMs that can understand diverse region descriptions.
Language models with visual modules. Although large language models (LLMs) have shown a broad range of commonsense knowledge, most of them trained only text corpus and cannot understand image inputs to tackle lannnuuae- (VL. tasks. Thus, recent works explore tackling VL tasks by solving sub-tasks with external visual modules and combining their outputs to obtain the final response [9; 10; 11; 12; 13; 14; 15]. The visual sub-tasks include describing images as text, finding image regions relevant to the text, editing images with text guidance, and obtaining answers from a VQA model. However, existing work focuses on converting visual inputs into text format so that LLMs can understand them. Our work is the first work using visual programming for interpretable and explainable T2I generation and evaluation.
Evaluation of text-to-image generation models. The text-to-image community has commonly used two types of automated evaluation metrics: visual quality and image-text alignment. For visual quality, Inception Score (IS) [29] and Fréchet Inception Distance (FID) [30] have been widely used. For image-text alignment, previous work used a single model to calculate an alignment score for image-text pair, based text retrieval [31], cosine similarity [33], captioning [25], object detection [34; 19; 35], and visual question answering (VQA) [36; 37]. In this work, we propose the first T2I evaluation framework VPEVAL, based on interpretable and explainable evaluation programs which execute a diverse set of visual modules (e.g.. object detection, OCR, depth estimation, object counting). Our VPEVAL evaluation programs provide visual+textual explanations of the evaluation result and demonstrate a high correlation with human judgments.
3 VPGEN: Visual Programming for Step-by-Step Text-to-Image Generation
We propose VPGEN, novel visual programming framework for interpretable step-by-step text-to- image (T2I) generation. As illustrated in Fig, 2, we decompose the text-to-image generation task into
3

---

two Pikachus table
Object/Count Generation
Layout Generation
Image Generation
deteraiie abjects thele cuets to traw at Lmage,
detensloo conrdteates dbcects. Caption: two Pikactes as Latte 28facts:
plazete (6,,1,22,58) pokects (dl
(6)
LM
plaahe (3) tabda (4)
LM
pilorts (aturt pilacts dayl) (ntanaa
L2I
Interpretable step-by-step text-to-image generation with VPGEN. VPGEN decomposes T2I generation into three steps: (1) object/count generation, (2) layout generation, and (3) image generation, and executes the three modules step-by-step.
Figure 2:
three steps: (1) object/count generation, (2) layout generation, and (3) image generation. In contrast to previous T2I generation works that use an intermediate layout prediction module [25; 26; 27; 28]. VPGEN represents all layouts (object description, object counts, and bounding boxes) in text, and employs an LM to handle the first two steps: (1) object/count generation and (2) layout generation. This makes it easy to adapt the knowledge of pretrained LMs and enables generating layouts of objects that are unseen during text-to-layour training (e.g., 'pikachu"). For layout representation, we choose the bounding box format because of its efficiency; bounding boxes generally require fewer tokens than other formats."
Two-step layout generation with LM. Fig. 2 illustrates how our LM generates layouts in two steps: (1) object/count generation and (2) layout generation. For the first step, we represent the scene by enumerating objects and their counts, such as "'oj1 (# of obj1) obj2 (# of obj2)". For the second step. following previous LM-based object detection works [49; 50], we normalize 'xyxy' format bounding box coordinates into [0, 1] and quantize them into 100 bins; single object is represented as 'obj (xmin, ymin, xmax, ymax)", where each coordinate within {0. (0....99))
Training layout-aware LM. To obtain the layout-aware LM, we use Vicuna 13B [16], public state-of-the-ard language model finetuned from LLaMA [51]. We use parameter-efficient finetuning with LoRA [52] to preserve the original knowledge of the LM and save memory during training and inference. We collect text-layout pair annotations from training sets of three public datasets: Flickr30K entities [17], MS COCO instances 2014 [18], and PaintSkills [19], totaling 1. 1.2M examples.
See appendix for more training details.
Layout-to-Image Generation. We use a recent layout-to- to-image generation model GLIGEN [20] for the final step - image generation. The toyout-to-image model takes a list of regions (bounding boxes and text descriptions) as well as the original text prompt to generate an image.
4 VPEVAL: Visual Programming for Explainable Evaluation of Text-to-Image Generation
VPEVAL is novel interpreaabe/eexllinablle evaluation framework for T2I generation models, based on visual programming. Unlike existing T2I evaluation methods that compute image-text alignment scores with an end-to-end model, our evaluation provides an interpretable program and vieual+ explanations for the evaluation results, as shown in Figs. 3 and 5. We propose types of evaluation prompts: (1) skill-based evaluation and (2) open-ended evaluation. In skill-based evaluation, we define five image generation skills and use set of skill-specifi prompts and evaluation programs, as illustrated in Fig. In open-ended evaluation, we use diverse set of prompts that require multiple image generation skills. We adopt language model dynamically generate evaluation program for each text prompt, as shown in Fig. 5. In the following, we describe evaluation skills (Sec. 4.1), visual evaluation modules (Sec. 4.2), kiill-bsedd evaluation with visual programs (Sec. 4.3), and open-ended evaluation with visual program generator LM (Sec. 4.4).
For example, xyry-format bounding box can be represented with 4 tokens, while 64x64 segmentation map requires 4096 tokens.

---

Skill-based Interpretable Evaluation Program Evaluation Results Visual Textual Explanation
V
Incorrect X
Object
Proept: photo dog"
"dog" ooject
Program:
objectEvoL((ngg "dog""
Count
Proept: "I dogs"
There "dog" abjects.
There are "Mang objects,
Programs countEvol (1ng, "dng", "end"))
Spatial
C'sporer, "pottee plant") found.
(0e12, ebg2) slanet)
plant"
Program: spottalEvel "spoon, plant, front")
with "iprot" 223 founs.
Scale
("laptap" soll with Thi higgee (arex] (ea)) fauns.
Proept:
"a laptop higger than sports
"sgorts ball" object found.
Program: ScoLcEvaL(150, "Laptop, sports balt, Bigger")
Text Rendering
Proept: "a poster that reads "shop"" (110, "shop"")
"Shop" text Founs,
SSDPP
Program:
Figure 3: Illustration of skill-based evaluations in VPEVAL (Sec. 4.3). Left: Given text prompts that require different image generation skills, our interpretable evaluation programs evaluate images by executing relevant visual modules. Right: Our evaluation programs output binary scores and provide visual (bounding boxes detected objects/text) + textual explanations the evaluation results.
4.1 Evaluation Skills
Inspired by skill-based T2I analysis of PaintSkills [19], our VPEVAL measures five image generation skills:
Object, Count, Spatial, Scale, and Text Rendering. Powered by evaluation programs with expert visual modules, our VPEVAL supports zero-shot evaluation of images (no finetuning of T2I models required), detecting regions with free-form text prompts, new 3D spatial relations (front, behind), and new scale comparison and text rendering skills, which were not supported in PaintSkills [19]. Fig. 3, we illustrate the evaluation process for each skill.
Object. Given a prompt with an object (e.g., "a photo of a dog"), a T2I model should generate an image with that object present.
Count. Given a prompt with a certain number of an object (e.g., "3 dogs"), a T2I model should generate an image containing the specified number of objects.
Spatial. Given a prompt with two objects and a spatial relationship between them (e.g.. "a spoon is in front of potted plant"), " a T2I model should generate an image that contains both objects with the correct spatial relations.
Scale. Given prompt with two objects and a relative scale between them (e.g., "a laptop that is bigger than sports ball"), T2I model should generate an image that contains both objects, and each object should be of the correct relative scale.
Text Rendering, Given prompt with certain text to display (e.g., "a poster that reads "shop""), T2I model should generate an image that properly renders the text.
4.2 Visual Evaluation Modules
To measure the skills described above in Sec. 4.1, we use eight expert visual evaluation modules specialized for different tasks. The modules provide visual+ explanations with their score. Visual explanations are generated by rendering bounding boxes on the images and textual explanations generated through text templates, shown in Fig. 3 and 5. We provide the Python pseudocode implementation for each module Fig.
Module definitions. objDet detects objects in image based on referring expression text and returns them (+ their 2D bounding boxes and depth), using Grounding DINO [53] and DPT [54]. oer detects all text in image and returns them (+ their 2D bounding boxes), using EasyOCR [55].
5

---

Object Detect
objoet( mny des objs, depth(ingg det 06
detocttinns 005, boxe3
def ocrting):
#ind_toxttiin det tects
def spatialevalinng, ob33 tast, dbj1 text, relation):
Object Eval
objecteval
Spatial Eval
objects = dejout T0bs1 text,ob52 text"?
tanget pnelation as "right":
objects oojoet (ldect_text) les(okjetts)
return any(abjects(1) sbjerts(B).x)
def scaleEval(ing,
relation):
Count Evail
court (ings objerr_ nexty count):
Scale
objects obf0et( "obsi tortittrrt,otll cort,obsR_tevt"" target, relation -- "biggor";
oejects (ojDet( objoct_ retumt Jon(objeres) targec,
return any(objecta(1)).rd objecca(e)
Text Eval
text iiny, target__ees)
VQA Eval
def voevol(ing, question, answer, cholces, target, answor): answer vo8_nodei(inn, guestion, answor_cholces)
toxts och(ing)
Figure 4: Python pseudocode implementation of visual modules used in VPEVAL.
vqa answers multiple-choice question using BLIP-2 (Flan-T5 XL) [56]. It handle phrases that cannot be covered only with objDet or ocr (e-g., pose estimation, action recognition, object attributes). objectEval evaluates if object in image using obj countEval evaluates an object occurs in an image a certain number of times using objDet and an equation (e.g., "==3", "<5"). spatialEval evaluates if two objects have a certain spatial relationship with each other. For six relations (above, below, left, right, front, and behind), it compares bounding box/depth values using objDet. For other relations, uses vqa. scaleEval evaluates if two objects have certain scale relationship with each other using obj For three scale relations (smaller, bigger, and same) it compares bounding box areas by using objDet. For other relations, uses vqa. textEval evaluates if a given text is present in an image using ocr.
4.3 Skill-based Evaluation with Visual Programs
For skill-based evaluation, we create text prompts with various skill-specific templates that are used for image generation and evaluation with our programs. See appendix for the details of prompt creation. Fig. 3, we illustrate our skill-based evaluation in VPEVAL. Given text prompts that require different image generation skills, our evaluation programs measure image-text alignment scores by calling the relevant visual modules. Unlike existing T2I evaluation methods, our evaluation programs provide visual+textual explanations of the evaluation results.
4.4 Open-ended Evaluation with Visual Program Generator LM
Although our evaluation with skill-specific prompts covers five important and diverse image genera- tion skills, user-written prompts can sometimes be even more complex and need multiple evaluation criteria (e.g., a mix of our skills in Sec. 4. and other skills like attribute detection). For example, evaluating images generated with the prompt 'A woman dressed for wedding is showing water- melon slice woman on a scooter." involves multiple aspects, such as two women (count skill), 'a woman scooter' (spatial skill), 'dressed for wedding" (attribute detection skill), etc. To handle such open-ended prompts, we extend the VPEVAL setup with evaluation programs that can many visual modules together (whereas single-skill prompts be evaluated with program of 1-2 modules). We generate open-ended prompt evaluation programs with an LLM, then the evaluation programs output the average score and the visual+textual explanations from their visual modules. The program generation involves choosing which prompt elements to evaluate and which modules to evaluate those elements (see Fig. 5).
Open-ended prompts. For open-ended evaluation, we use 160 prompts of TIFA v1.0 human judg- ment dataset [36] (we refer to these prompts 'TIFA160'). The dataset consists of (1) text prompts from COCO [18], PaintSkills [19], DrawBench [24], and Partiprompts [47], (2) images generated by five baseline models (minDALL-E [57]. VQ-Diffusion [58], Stable Diffusion v1.1/v1.5/v2.1 [21]),
and (3) human judgment scores on the images (on 1-5 Likert scale).
Generating evaluation programs via in-context learning. As annotation of evaluation programs with open-ended prompts can expensive, we use ChatGPT (GPT-3.5-Turbo) [38] to generate evaluation programs via in-context learning. To guide ChatGPT, we adapt the 12 in-context prompts from TIFA [36]. We show ChatGPT the list of visual modules and example text prompts and programs, then ask the model to generate program given new prompt, illustrated in Fig, 5, For

---

Open-ended Interpretable Evaluation Program
Evaluation Results Visual Textual Explanation
Glvan decrappier, gererate that verifies The lsege destripttion connect.
Example text prompt
Incorrect
no "ran" obfect Found.
2" Incoontert eoomles
Description: poodig 34272a jecket
Exomple evalustioo
ddjectiool: "man")s
who 6an wan,aoon, (oir) "Nor")
aearsa?
Correct
"evergreen" object Found.
text promet
bescripttion alite stope cowers backgraend, walle the and one wosdy underdevclepes evergrecs toe foreground.
Incorrect
X
there "evergreen" objects,
dbjecttoal
sbjectioal [Inage. (evergren
ChatGPT
countEval Lmngn, 13372
"what the foreground?" grassy slope.
ugat ugactnngg, "utat is the diregraasd?" 'grassy
Correct
sloce, beach, Fauld, Poreaet, "araaay alspe
Figure 5: Evaluation of open-ended prompts in VPEVAL (Sec. 4.4). Left: We generate evaluation programs with ChatGPT via in-context learning. Right: Our evaluation programs consist of multiple modules evaluating different elements from a text prompt. We output the final evaluation score by averaging the outputs of each evaluation module.
Table VPEVAL scores of T21 generation models on skill-based prompts (see Sec. 5.2 for analysis).
F30: Flickr30K Entities, C: COCO, P: PuintSkills.
VPEVAL Skill Score (%) f
Model
Object
Count
Spatial
Scale
Text Rendering
Average
Stable Diffusion vl.4
97.3
47.4
22.9
11.9
8.9
37.7
Stable Diffusion v2.1
96.5
53.9
31.3
14.3
6.9
40.6
Karlo
95.0
59.5
24.0
16.4
8.9
40.8
minDALL-E
79.8
29.3
7.0
6.2
0.0
24.4
DALL-E Mega
94.0
45.6
17.0
8.5
0.0
33.0
VPGEN (F30)
96.8
55.0
39.0
23.3
5.2
43.9
VPGEN (F30+C+P)
96.8
72.2
56.1
26.3
3.7
51.0
reproducible and accessible evaluation, we release the evaluation programs so that VPEVAL users do not have generate the programs themselves. We will also release a public LM (finetuned for evaluation program generation using ChatGPT outputs) that can run on local machines.
Experiments and Results
In this section, we introduce the baseline T2I models we evaluate (Sec. 5.1), compare different T2I models with skill-based (Sec. 5.2) and open-ended prompts (Sec. 5.3), and show the human- correlation of VPEVAL. (Sec. 5.4).
5.1 Evaluated Models
We evaluate our VPGEN (Vicun++GLIGEN) and five popular and publicly available T2I models,
covering both diffusion models (Stable Diffusion v1.4/v2.1 [21] and Karlo [59]), and multimodal autoregressive language models (minDALL-E [57] and DALL-E Mega [60]). Stable Diffusion v1.4 the most comparable baseline to VPGEN, because the GLIGEN [20] in VPGEN uses frozen Stable Diffusion v1.4 with a few newly inserted adapter parameters for spatial control.
5.2 Evaluation on Skill-based Prompts
Diffusion models outperform multimodal LMs. In Table 1, show the VPEVAL skill accuracies for each model. The diffusion models (Stable Diffusion, Karlo, and our VPGEN) show higher overall accuracy than the multimodal LMs (minDALL-E and DALL-B Mega).
Count/Spatia//Scall/Teek Rendering skills are challenging. Overall, the five baseline T21 models achieve high scores (above 93% except for minDALL-E) in Object skill; i.e., they are good at
7

---

generating high-qualing single object. However, the models score low accuracies in the other skills, indicating that these skills (Count, Spatial, Scale, and Text Rendering) are still challenging with recent T21 models.
Step-by-step generation improves challenging skills. The bottom row of Table 1 shows that VPGEN achieves high accuracy in Object skill and strongly outperforms other baselines in Count (+12.7% than Karlo), Spatial (+24.8% than Stable Diffusion v2. and Scale (+9.9% than Karlo) skills. This result demonstrates that our step-by-step generation method is effective in aligning image layout with text prompts, while also still having an interpretable generation program. All models achieve low scores on the Text Rendering skill, including our VPGEN. A potential reason why our VPGEN model does not improve Text Rendering score might be because the text-to-layout training datasets of VPGEN (Flickr30K Entities [17], COCO [18], and PaintSkills [19]) contain very few images/captions about text rendering, opening room for future improvements (e.g., finetuning on a dataset with more images that focus on text rendering and including text rendering module as part
of our generation framework).
Fine-grained analysis in CountSpattial/Salee skills. To better understand the high performance of our VPGEN in Count, Spatial, and Scale skills, we perform detailed analysis of these skills with fine-grained splits. In Fig. 6, we compare our VPGEN (Vicuna+GLIGEN and its closest baseline Stable Diffusion v1.4 on the three skills. Overall: VPGEN achieves better performance than Stable Diffusion on every split on all three skills. Count: While both models show difficulties with counting larger numbers, our VPGEN model achieves better performance with 50+ accuracy on all four numbers. Spatial: Our VPGEN model performs better on all six spatial relations. VPGEN shows higher accuracy on 2D relations (leftrrigh/hellow/aovee) than on 3D depth relations (front/behind), while the Stable Diffusion is better at 3D relations than 2D relations. Scale: VPGEN generates more accurate layouts with two objects of different sizes (bigger/smaller), than layouts with two objects of similar sizes (same).
Count
Spatial
Scale
VPGen (Ours)
0.5
0.0
left
right
below
above
front
bohind
bigger
smaller
saimo
Figure 6: VPEVAL score comparison our VPGEN (Vicun++GLIGEN) and Stable Diffusion vl.4 on fine-grained splits in CountSpatiial/Saale skills.
5.3 Evaluation on Open-ended Prompts
Table 2:
VPEVAL scores on open-
ended prompts (see Sec. 5.3), GLIGEN
Table 2 shows the VPEVAL score on the open-ended TIFA160 prompts. We caleulate the score by averaging accuracy from the modules (see Fig. 5 for an example). The overall score trend of the baseline models (diffusion models > multimodal LMs) is consistent with the skill- based prompts. Unlike the trend in skill-based evalua- tion (Sec. 5.2), our VPGEN (VicUN++GLIGEN) methods achieve similar performance the Stable Diffusion base- line, while also providing interpretable generation steps. We perform analysis of the open-ended prompts used for evaluation. In these prompts, object descriptions and attributes are the dominating prompt element (86.4% of elements), whereas descriptions of spatial layouts, only account for 13.6% of elements. See appendix for more details. For prompts from PaintSkills [19] where spatial layouts are more important, we find that our VPGEN scores much higher than Stable Diffusion v1.4 (71.0 v.s. 63.5) and Stable Diffusion v 1 (71.0 v.s. 68.4 Note that GLIGEN used in VPGEN is based on Stable Diffusion vl.4, making the closest baseline, and hence future versions of GLIGEN based on Stable Diffusion v2.1 or other stronger layout-to-image models will improve our results further. Also, we find that GLIGEN sometimes fails to properly generate images even when VPGEN generates the correct layouts (see following paragraph).
in VPGEN has Stable Diffusion v1.4
backbone. F30: Flickr30K Entities, C:
COCO, P: PaintSkills.
Model
Score (%) f
Stable Diffusion v1.4
70.6
Stable Diffusion v2.1
72.0
Karlo
70.0
minDALL-E
47.5
DALL-E Mega
67.2
VPGEN (F30)
71.0
VPGEN (F30+C+P)
68.3
8

---

Object
Count "on imaga-ee of 4 4bottes"
Spatial "A tran the oightol cow
Scale gineil that bisger than cup
Text dgnn that night
modules (instead of Grounding "an image of tennis roclet
NE
Stable Diffusion v1.4
VPGen (Ours)
COCO
DrawBench
PaintSkills
PartiPrompt
or Ming top the devk on laphnp..
1A rebra the right ofa tim hyftord"
photoo bear and diving trbte diving tab beinw bear"
"a drop-top sports coming wounda bead the read"
Stable Diffusion v1.4
VPGen (Ours)
Figure 7: Images generated by Stable Diffusion v1.4 and our VPGEN (Vicuna 13B+GLIGEN) for skill-based (top) and open-ended (bottom) prompts.
VPGEN sources of error: layout/image generation. Since our VPGEN separates layout generation and image generation, we study the errors caused by each step. For this, we manually analyze the images generated from TIFA 160 prompts, and label (1) whether the generated layouts align with the text prompts and (2) whether the final images align with the text prompts/layouts GLIGEN sometimes fails to properly generate images even when Vicuna 13B generates the correct layouts, indicating that when better luyout-to-image models become available, our VPGEN framework will achieve higher results. We include more details in the appendix.
Qualitative Examples. In Fig. 7, we provide example images generated by our VPGEN and Stable Diffusion v1.4 (the closest baseline) with various skill-based (top) and open-ended (bottom) prompts. See appendix for more qualitative examples with skill- selll--ae//ope-eeddee prompts, prompts with unseen objects, counting 4 objects, and error analysis.
5.4 Human Evaluation of VPEVAL
To measure alignments with human judgments and our VPEVAL, we compare the human- correlatioss of VPEVAL. and other metrics both skill-based prompts and open-ended prompts.
Human correlation VPEVAL on skill-based prompts. We two expert annotators to evaluate 20 images, from each of the five baseline models, for each of the five skills with binary scoring (total 20 x x = 500 images). The inter-annotator agreements were measured with Cohen's [61] = 0.85 and Krippendorff's
[62] 0.85, indicating 'ncar- tncar-perfect" (K 0.8 agreement [63; 64; 62]. We compare our visual programs with captioning (with metrics BLEU [65], ROUGE [66], METEOR [67], and SPICE [68]). VQA, and CLIP (ViT-B/32) based evaluation. We use BLIP-2 Flan- T5 XL, the ofate-of-the-ar public model for image captioning and VQA.
0.8)
In Table 3, our VPEVAL shows a higher overall correlation with human judgments (66.6) than single module-based evaluations (CLIP, Captioning, and VQA). Regarding per-skill correlations, VPEVAL shows especially strong human correlations in Count and Text Rendering. As the BLIP-2 VQA module also shows strong correlation on Object/Spatial/SSaal,, we also experiment with VPEVAL.: using BLIP-2 VQA for (ojeccEval//ppttiallvla////aaal

---

Table 3: Human correlation study of skill-based evaluation. We measure Spearman's correlation
between human judgment and different automated metries on the skill-based prompts (Sec. 4.3).
VPEVAL using BLIP-2 VQA for /abpceEiwl/sptiilllvaaaalaalllle
modules.
Human-metric correlation (Spearman's 's p)
Eval Metric
Object
Count
Spatial
Scale
Text
Overall
CLIP Cosine similarity (ViT-B/32)
35.2
38.6
35.4
13.7
40.0
20.4
BLIP-2 Captioning BLEU
11.9
31.4
26.3
24.0
23.6
-3.4
BLIP-2 Captioning ROUGE
15.7
26.5
28.0
12.2
28.3
11.9
BLIP-2 Captioning -ETTOO
33.7
20.7
40.5
25.1
26.6
29.3
BLIP-2 Captioning SPICE
56.1
20.9
40.6
27.3
18.6
28.1
BLIP-2 VQA
63.7
63.1
38.9
26.1
31.3
65.0
VPEVAL VPEVAL
34.5
63.8
48.9
29.4
85.7
73.5
63.7
63.8
51.2
29.5
85.7
79.0
DINO and DPT), which increases human correlation scores. Note that our object detection modules visually explain the evaluation results, as shown in Fig. 3, and we can have an even higher correlation when we have access to stronger future object detection models.
Human correlation of VPEVAL on open-ended prompts. We generate visual programs with our program generation LM on TIFA160 prompts [36]. The dataset consists of 800 images (160 prompts x 5 T2I models) and human judgments (1-5 Likert scale) along with other automatic metrics (BLIP-2 captioning, CLIP cosine similarity, and TIFA with BLIP-2) on the images. Table 4 shows our VPEVAL achieves a better human correlation with TIFA (BLIP-2), and our ! version achieves an even higher correlation. The results indicate that using various interpretable modules specialized in different skills complements each other and improves human correlation, while also providing visual+tcxtual explanations.
Table 4: Human correlation
on open-ended evaluation with Spearman's p.
Metries
p (f)
BLIP-2 Captioning
BLEU-4
18.3
ROUGE-I.
32.9
METEOR
34.0
SPICE
32.8
Cosine-simitarity CLIP (VTT-B/32)
33.2
LM VQA module TIFA (BLIP-2)
Human analysis on the generated programs. Lastly, we also measure the faithfulness of the generated evaluation programs. For this, we sample TIFA160 prompts and analyze the evaluation programs by (1) how well the modules cover elements in the prompt; (2) how accurate the module outputs are when run. We find that programs generated by our VPEVAL have very high coverage over the prompt elements and high per-module accuracy compared human judgment. We include more details in the appendix.
55.9
LM multiple modules VPEVAL (Ours)
56.9
VPEVAL (Ours)
60.3
6 Conclusion
We propose two novel visual programming frameworks for interpreaablefxxllanaalee T2I generation and evaluation: VPGEN and VPEVAL. VPGEN a step-by-step T2I generation framework that decomposes the T21 task into three steps (object/count generation, layout generation, and image generation), leveraging an LLM for the first two steps, and layout-to-image generation models for the last image generation step. VPGEN generates images more accurately following the text descriptions (especially about object counts, spatial relations, and object sizes) than strong T2I baselines, while still having interpretable generation program. VPEVAL is T2I evaluation framework that uses interpretable evaluation programs with diverse visual modules that are experts in different skills to measure various T2I skills and provide visual+textull explanations of evaluation results. In analysis, VPEVAL presents higher correlation with human judgments than single modell-bsee evaluations, both skill-specific and open-ended prompts. We hope our work encourages future progress on interpretableexxplainall generation and evaluation for T2I models.
Limitations & Broader Impacts. See Appendix for limitations and broader impacts discussion.
10

---

Acknowledgement
We thank the reviewers for their valuable comments and feedback. This work was supported by NSF-AI Engage Institute DRL21 1263, ONR N00014-23-1-2556, and DARPA ECOLE Program No. HR001 12390060. The views, opinions, and/or findings contained in this article are those of the authors and not of the funding agency.
ARO 19F121002200, DARPA MCS N66001-19-2-4000
References
[1] Jacob Devlin, Ming-Wii Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL, oct 2019.
[2] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer, and Pre-training Bart. BART: Denoising saaeence-t-Sequenn Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL, 2020.
[3] Tom B. Brown. Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry. Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretehen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Matcusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In NeurIPS, 2020.
[4] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with Unified Text-to-Text Transformer. JMLR, 21:1-67, 2019.
[5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari. Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Olcksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstent Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways, 2022.
[6] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc Le.. Finetumed Language Models Are Zero-Shot Learners. In ICLR, 2022.
[7] OpenAL. GPT-4 Technical Report, 2023.
[8] Hyung Won Chung, Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Yunxuan Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling Instruction- Instruction-Finetunee Language Models, 2022.
[9] Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An Empirical Study of GPT-3 for Few-Shot Knowledge- Based VQA. In AAAI, 2022.
[10] Yushi Hu, Hang Hua, Zhengyuan Yang. Weijia Shi, Noah A. Smith, and Jiebo Luo. PrompiCap: Prompt- Guided Task-Aware Image Captioning, 2022.
[11] Tanmay Gupta and Aniruddha Kembhavi. Visual Programming: Compositional visual reasoning without training. In CVPR, nov 2023.
[12] Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual Inference via Python Execution for Reasoning, 2023.
[13] Chenfei Wu, Shengming Yin. Weizhen Qi, Xiaodong Wang. Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models, 2023.
11

---

[14] Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action, 2023.
[15] Yaobo Liang. Chenfei Wu, Ting Song. Wenshan Wu, Yan Xia. Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, Yun Wang. Linjun Shou, Ming Gong, and Nan Duan. TaskMatrix.lI Completing Tasks by Connecting Foundation Models with Millions APIs, 2023.
[16] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[17] Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. ICV,, 123(1):74-93, 2017.
[18] Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Ohjects in Context. In ECCV, 2014.
[19] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In ICCV, 2023.
[20] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: Open-Set Grounded Text-to-lmage Generation. In CVPR, 2023.
[21] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent difffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674-10685, 2021.
[22] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Sher Shot Text- toxt-to-Ine Im-emaee Generation. In ICML, 2021.
[23] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents, 2022.
[24] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes. Tim Salimans. Jonathan Ho, David Flect, and Mohammad Noruuzi. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. In NeurIPS, 2022.
[25] Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lec. Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis. In CVPR, 2018.
[26] Fuwen Tan, Song Feng, and Vicente Ordonez. Text2Seene: Generating Compositional Scenes From Textual Descriptions. In CVPR, 2019.
[27] Wenho Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei I Lyu, and Jianfeng Gao. Object-Driven Text- Im-Ie Synthesis via Adversarial Training. In CVPR, pages 12166-12174, 2019.
[28] Jiadong Liang, Wenjic Pei, and Feng Lu. Layout-Bridging Bridging Text-to-Image Synthesis, 2022.
[29] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved Techniques for Training GANs. In NIPS, 2016.
[30] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by Two Time-Scale Update Rule Converge to Local Nash Equilibrium. In NIPS, 2017.
[31] Tao Xu, Pengehuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. AunGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks. In CVPR, 2018
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger. Ilya Sutskever. Jong Wook, Kim Chris, Hallacy Aditya, Ramesh Gabriel, Goh Sandhini, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger. and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICML, 2021.
[33] Yupan Huang, Hongwei Xue, Bei Liu, and Yutong Lu. Unifying Multimodal Transformer for Bi-directional Image and Text Generation. In ACM MM. Association for Computing Machinery, 2021.
12

---

[34] Tobias Hinz. Stefan Heinrich, and Stefan Wermier. Semantic Object Accuracy for Generative -maaee Synthesis. IEEE Transactions on Puttern Analysis and Machine Intelligence, pages 1-1, 2020.
[35] Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Frie Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking Spatial Relationships in Text-to-lmage Generation, 2022.
[36]
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In ICCV, 2023.
[37] Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. arXiv preprint arXiv:2305. 2023.
[38] OpenAI. Chatgpt. https://openaa: com
2023.
[39] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. In NIPS, 2014.
[40] Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating Images from Captions with Attention. In ICLR, 2016.
[41] Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. Learning What and Where to Draw. NIPS, 2016.
[42] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bemt Schiele, and Honglak Lee. Generative adversarial text image synthesis. In ICMII, 2016.
[43] Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannanch Hajishirzi, and Aniruddha Kembhavi. X-LXMERT: Paint, Caption and Answer Questions with Multi- Modal Transformers. In EMNLP, 2020.
[44] Jascha Dohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In ICML, 2015.
[45] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In NeurlPS, pages 1-25, 2020.
[46] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever. and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text- Guided Diffusion Models. In ICML, 2022.
[47] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. Transactions on Machine Learning Research, 2022.
[48] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsua Yang. Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-To-Image Generation via Masked Generative Transformers, 2023.
[49] Ting Chen. Saurabh Saxena, Lala Li, David J. Fleet, and Geoffrey Hinton. Pix2seq: A Language Modeling Framework for Object Detection. In ICLR, 2022.
[50] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling. In ECCV, 2022.
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, 2023.
[52] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, pages 1-26, 2022.
[53] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023.
[54] René Ranitl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. 2021 IEEE/CVF International Conference Computer Vision (ICCV), pages 12168, 2021.
13

---

[55] Jaided AL. Easyoer: Ready-to-use ocr with 80+ supported languages and all popular writing scripts including latin, chinese, arabic, devanagari, cyrillic and etc.. 2023.
[56] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language- Image Pre- training with Frozen Image Encoders and Large Language Models, 2023.
[57] Sachoon Kim, Sanghun Cho, Chiheon Kim, Doyup Lee, and Woonhyuk Back. mindall-e conceptual captions. httpp:///gtthnn thubbb.ttttu...... wwt.cooo//aaaaa cco.m//aaaaabrraannnnnnaa-L----.
2021.
[58] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector Quantized Diffusion Model for Text-to-Image Synthesis. In CVPR, 2022.
[59] Donghoon Lee, Jiseob Kim, Jisu Choi, Jongmin Kim, Minwoo Byeon, Woonhyuk Back, and Sachoon Kim. Karlo-v1.O.alpha .laphh coyo- 100m and cc15m. https:/ http///gtttuub//e/// thub. com/Kaakaobraiinkarl.., 2022.
[60] Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phuc Le Khac, Luke Melas, and Ritobrata Ghosh. Dall-e mini, 7 2021.
[61] Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46, 1960.
[62] Klaus Krippendorff. Content Analysis: An Introduction to Its Methodology, Sage Publications, 4th edition, 2018.
[63] Mary I. McHugh. Interrater reliability: the kappa statistic. Biochem. Med. (Zagreb), 22(3))276-282, 2012.
[64] Julius Sim and Chris C Wright. The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy, 85(3):257 268, 03 2005.
[65] Kishore Papineni, Salim Roukos, Todd Ward, and Wj Wei-jing Zhu. BLEU: a Method for Automatic Evaluation of Machine Translation. In ACL, 2002.
[66]
Chin- Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In ACL Workshop, 2004.
[67] Satanjeev Banerjee and Alon Lavie. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In ACL Workshop, 2005.
[68] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: Semantic Propositional Image Caption Evaluation. In ECCV, 2016.
[69] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. and Thomas Wolf. Diffusers: State-of-ther art diffusion models. https://githu: com/ (tt.eaarcce/ffuuerss.. 2022.
14

---

Appendix
In this appendix, we provide qualitative examples (Appendix A), additional VPGEN analysis (Appendix B), VPGEN error analysis (Appendix C), VPEVAL error analysis (Appendix D), VP- GEN/VPEVAL implementation details (Appendix E), TIFA prompt element analysis details (Ap- pendix F), limitations and broader impacts (Appendix G), and license information (Appendix H).
A Qualitative Examples
In Fig. 8 and Fig. 9, we provide example images generated by our VPGEN and Stable Diffusion v1.4 (the closest baseline of VPGEN) with various skill-based and open-ended prompts, respectively. In Fig. 10, we show VPGEN generation examples of placing objects that are unseen during the text-to-layout finetuning of Vicuna 13B.
Object
Count
Spatial "a truck la behind motortycle
Scale remote that than
Text "a sign that "sarking"
"zebra"
"3
PAKKINNG IOU PANKEP PAVKGKNG
Stable Diffusion v1.4
PAING
VPGen (Ours)
Object picture of bus"
Count
Spatial "a zebra to the left of broycle"
Scale that bigger then bicyclle
Text
poster that Trmeppsiing
Stable Diffusion v1.4
TRSaSSS
VPGen (Ours)
Figure 8: Images generated by Stable Diffusion v1.4 and our VPGEN (Vicuna 13B+GLIGEN) for skill-based prompts.
B
Additional VPGEN Analysis
In the following, provide additional analysis of VPGEN, including counting prompts, bounding box quantization, using GPT-3.5-Turbo as layout generation LM, and visual quality metric (FID).
Counting prompts: More than 4 objects & Unspecified number of objects. In addition to the Count skill results with counts of to in the main paper Fig. 6, we provide additional results of VPGEN with counts of up to 7 in Table 5 and show generation examples in Fig. 11 (a) (d). As expected, the trend continues from Fig. 6, where higher counts are more challenging. We also experiment with prompts without specific numbers. We find that VPGEN can successfully generate layouts/images even when the prompts not have explicit counts. As shown in Fig. 11 (e), our VPGEN generates two Pikachus from the prompt "Pikachus on the table".
15

---

COCO
DrawBench "A car playing digital
PalntSkills
PartiPrompt "the skyline City
"Black and white photograph of bed with laptop it."
"a photo bear and bus: bus above bear"
Stable Diffusion v1.4
VPGen (Ours)
Figure 9: Images generated by Stable Diffusion v1.4 and our VPGEN (Vicuna 13B+GLIGEN) for open-ended prompts.
pikachus
the
Link
Figure 10: Images generated by VPGEN (Vicuna 13B+GLIGEN) from prompts including objects (e.g.. 'Pikachu', 'Link', 'master sword') that are not included in text-to-layou annotations (Flickr30K Entities, COCO, PaintSkills) for Vieuna fine-tuning.
Bounding box quantization. As mentioned the main paper, we use normalized and quan- tized coordinates with 100 bins represent bounding boxes, following LM-based object detection works [49; 50]. In Table 6, we compare 100 and 1000 bins for the bounding boxes. Increasing the granularity of the bounding box to 1000 does not increase the accuracy in both skill-based and open-ended prompts, suggesting that the current 100-bin quantization does not hurt the accuracy of object placements.
GPT-3.5- Turbo v.s. Vicuna 13B. In our initial experiment, we tested GPT-3.5-Ture to generate spatial layouts by showing in-context examples and found that the generated layouts were often inac- curate not meaningful. For example, GPT-3.5-Turbe often generates list of bounding boxes that the same size as the entire image (e.g., [object
00,0,5 99 ,9) object3 (0,0, 2,00, .999 ])) We conjecture that this is because their training corpus might not include many bounding boxes. Then, we collected text-layout annotations and trained a language model to generate spatial layouts from text prompts.
(0,0,99, ,9)99), object
For quantitative comparison, we implement VPGEN with GPT-3.5-turbe with 36 in- context examples that cover different skills and compare it with Vicuna 13B trained on Blib::Ok=<CCCO+Pinnnkii As shown in Table Vicuna 13B based VPGEN shows higher skill-based and open-ended VPEval accuracies than TFr-3.5-Turbo based VPGEN.
Visual quality metric (FID). In Table 8, we compare our VPGEN (Vieuna13B + GLIGEN) to its backbone Stable Diffusion v1.4, in FID [30] (30K images of COCO val 2014 split) and VPEval Accuracy. Both VPGEN checkpoints show better based accuracy than Stable Diffusion v1.4 while achieving comparable open-ended accuracy. and FID. In FID (lower the better), we find VPGEN (Flickr30k) Stable Diffusion v1.4 < VPGEN Piickr30s=CCCO+Paiittttt that bit of increase (but still reasonably good) in the FID of the Fickkook-cocOoPiimssill..
We think
16

---

Count >= 4
Without exact count in prompt
M M
(a) 4 dogs
(b) 5 dogs
(c) dogs
(d) dogs
(e) Pikachus the table
Figure 11: Images generated by VPGEN with prompts requiring counting skills. (a) (d) show the generation examples with count 2 4. (e) shows generation from a prompt that does include exact count ('pikachus").
Table 5: VPEVAL Count skill accuracy of VPGEN with different number of objects.
VPEval Count Acc. (%)
4
7
VPGEN 85.3
80.1
65.6
55.2
48.0
35.3 28.1
checkpoint is because the layouts of PaintSkills are different from those of natural scenes (COCO and 3lickr30k).
Training with real v.s. synthetic images. Our layout generation module (Vicuna 13B) in VP. GEN is trained on mix of both 3D simulator-based synthetic (PaintSkills) and real-world images (MSCOCO/Fliccr3KK)
conduct an ablation study of training only with real v.s. synthetic images. We train the Vicuna only PaintSkills and only Flickr30K and compare their results on both skill- based and open-ended prompts in Table 9. For skill-based prompts, PaintSkills training shows slightly higher average accuracy (46.0) than Flickr30K training (43.9). For open-ended prompts, PaintSkills training gets a VPEval score of (65.6), lower than Flickr30K training (71.0.. This indicates that training synthetic data inject knowledge of specific skills such counting and spatial relation understanding, but it is hard to cover diverse objects and attributes only with synthetic data.
C VPGEN Error Analysis
Layout v.s. Image generation skill-based prompts. We analyze the performance of each step the VPGEN pipeline to determine how much error propagated with the following four metrics.
1. Object Recall: The ratio of whether correct objects are included in the generated layouts.
2. Object Count: The ratio of the correct number of objects is included the generated layouts.
3. Layout Accuracy: VPEval accuracy of the generated layouts (using the generated layouts as the object detection results).
4. Image Accuracy: VPEval accuracy of final image by GLIGEN (in the main Table 1).
As shown in Table 10, the layout accuracy is much higher than image accuracy, especially for Count/Spatial skills. This indicates that the major error source the image rendering step (with GLIGEN), and using more accurate layout-to- image generation model in the future would improve accuracy of our VPGEN pipeline.
Human evaluation on open-ended prompts. As mentioned in the main paper Sec. 5.3, we ask two expert human annotators to evaluate 50 randomly sampled prompts from TIFA160 [36], with the following two metrics: (1) layout accuracy: if the generated layouts align with the text prompts; (2) image accuracy: if the generated images align with the text prompts. The human annotators show high inter-annotator agreements for both layout accuracy (Cohen's 's 0.73, Krippendorff's = 0.73) and image accuracy (Cohen's x = 0.87, Krippendorff's a 0.87). The Vicuna 13B's layout accuracy is (92%) higher than GLIGEN's image correctness (65%). The result is consistent with the skill-based prompts error analysis result (Table 10) and suggests that a better layout-to-image generation model could further improve our VPGEN framework.
17

---

Table 6: Bounding box quantization: 1000 v.s. 100 bins.
Bins
VPÉval skill-based Acc. (%)
VPEval open-ended Acc. (%b
68.2
100 (default)
51.0
68.3
Table 7: GPT-3.5-Turbo u.s.
Vicuna 13B in VPGEN.
Model
VPEval skill-hased Acc. (%)
VPEval open-ended Acc. (56
GPT-3.5-Turbo (36 examples) GLIGEN Vicuna 13B GLIGEN
40.7
65.5
51.0
68.3
Common error categories. We show some common categories of errors from the text-to-layout generation step (with Vicuna 13B) and from the layout-to-image generation step (with GLIGEN), respectively. As shown in Fig. 12, the Vicuna 13B sometimes (a) generates objects that are not specified in prompts (e.g., 'chair' in the 1st example) or (b) misses objects that are mentioned in prompts (e.g., "building" in the 5th example). generates wrong objects (e.g-. 'suitcase', instead of 'bench', in the 1st example), (b) misses some objects (e.g., 'umbrella' in the 3rd example), or (c) misses object details (e.g., 'closed oven' instead of 'oven with the door open' in the 5th example).
shown in Fig. 13, the GLIGEN sometimes (a)
Challenging, non-canonical prompts. In Fig. 14, we show generation examples with challenging. non-canonical prompts (a) "A realistic photo of Pomeranian dressed up like 1980s professional wrestler with neon green and neon orange face paint and bright green wrestling tights with bright orange boots" (from DrawBench [24]) and (b) "a circle with cm radius and a circle with cm radius". Our VPGEN understands the important parts of the prompts (e.g., generating 'realistic', 'Pomeranian dog', 'bright orange boots' and generating layouts of two circles in different sizes), but misses some aspects (e.g., the bigger circle is not twice the size of the smaller circle). This is probably because the training prompts (COCO, PaintSkills, Flickr30k Entities) do not include many prompts written in these styles. We believe that scaling datasets with various sources can further improve VPGEN.
D VPEVAL Error Analysis
coverage and accuracy. For the evaluation program analysis (main paper Sec. 5.4), we experiment with two expert annotators using 50 randomly sampled TIFA160. We ask two expert annotators to analyze the evaluation programs in two criteria: (1) how well the evaluation program covers the content from the text prompt; (2) how accurately each evaluation module evaluates the images. To make it easier for the annotators, for (2) module accuracy analysis, we only ask them to check the output of three randomly sampled evaluation modules. We find that, on average, our evaluation programs cover 94% of the prompt elements, and our modules are accurate 83% the time. This shows that our VPEVAL evaluation programs are effective in selecting what to evaluate and are highly accurate while also providing interpretability and explainability. In Fig. 15, we visualize our evaluation programs that cover different parts of the prompts and evaluation modules that provide accurate outputs and visual+textual explanations.
Evaluation program analysis:
E VPGENVPEVAL Implementation Details
E.1 VPGEN Details
Training layout-aware LM. To obtain layout-aware LM, we use Vicuna 13B [16], a public sfate-of- the-art chatbot language model finetuned from LLaMA [51]. We use parameter-efficient finetuning with LoRA [52] to preserve the original knowledge of LM and save memory during training and inference. We collect the text-layout pair annotations from training sets of three public datasets:
Flickr30K entities [17]. MS COCO instances 2014 [18], and PaintSkills [19], totaling 1.2M examples.
We use randomly selected 2,000 examples for the validation set and use the rest for training. For
the object/count generation step, set the maximum counts for single object class as 7. We train
18

---

Table 8: VPEval accuracy and FID of VPGen (Vicuna 13B + GLIGEN) and its closest bascline Stable Diffusion v1.4. F30: Flickr30K Entities, C: COCO, P: PaintSkills.
Model
VPEval Skill- based Acc. (%)
VPEval Open-ended (%)
(COCO 30K)
Stable Diffusion v1.4
37.7
70.6
16.5
VPGEN (F30)
43.9
71.0
15.9
VPGEN (F30+C+P)
51.0
68.3
20.1
Table 9: Training Vicuna 13B with real v.s. synthetic data. F30: Flickr30K Entities, P: PuintSkills.
Model
VPEval skill- Acc. (%) f
VPEval open- Ace. (%)
VPGEN (F30; real) VPGEN (P: synthetic)
43.9
71.0
46.0
65.6
Vicuna 13B with per-gpu batch size 96 (= 24 batch x 4 gradient accumulation). When training Vicuna 13B with Flick30K=COCC+PaannnS
dataset, we train the model for 2 epochs. When training only with Flickr30K dataset, we train the model for 6 epochs, to rough match the training time. Training takes 26 hours with 4 A6000 GPUs (each 48GB).
GLIGEN inference details. We use Huggingface Diffusers [69] implementation of GLI- GEN [20]². Following the default configuration, 0.3, num inf ereeecreteepeeeeeeeeeeeeeeeeeeee = 50, and fp16 precision during inference.
aeligen_schedin wawpslmmmlllngggga
Inference time. On single A6000 GPU, VPGEN takes around 6s (2s for Vicuna 13B and 4s for GLIGEN) to generate an image.
E.2 VPEVAL Details
VPEVAL prompts and evaluation programs. Our skill-based benchmark has almost uniform distribution of objects and relations among the text prompts. This ensures that generation models cannot achieve high scores by performing highly on few common objects, relations, and counts, etc. We create skill-specific prompts by composing templates with lists of objects (from 80 COCO objects), counts (I-4), relations (2D: above, below, left, right; 3D: front, behind), scale (smaller, bigger, same), and text (31 unique words). Our evaluation can be extended to any number of objetts/countt/eelaiinnseee In total, our skill-based benchmark has 400/1000// for the objjcc/coun//spatillizzzzzte
prompts skills, respectively. In Table 1, we show templates and evaluation programs used for each skill.
Captioning/VQA/CCL.IP baseline details. For captioning, we generate a caption with BLIP-2 [56] and calculate text similarity metrics between the caption and the original text prompt. For VQA, give the BLIP-2 (FlanTT5 XL) the image, a question, and a yes/no answer choice, with a prompt template "Question: {question] Choices: yes, formatted version of the text prompt (e.g., "a photo of a dog" + "is there dog in the image?"). For the spatial and scale skill,
Answer:". The {question} is
ask three questions: if object A is present (e.g., "is there dog in the image?") if object B is present (e.g., "is there a cat in the image?"), and if the relationship between them is true (e.g.. "is the cat behind the dog?"". We mark each image correct if the VQA model outputs 'yes' for all three questions. The first two questions prevent accidental false positives if an
/ /tttps://github.. com com/gligen/tiffuser.
Table 10: Step-wise error analysis of VPGen (Vieuna+GLIGEN) For the Spatial skill, 'front' and 'behind' splits are skipped in this table since our Vicuna 13B does not generate depth information.
Vicuna 13B
GLIGEN
Skills
Object Recall (%)
Object Count (%) 1
Layout Accuracy (%)
Image Accuracy (%)
Object
99.2
98.8
99.2
96.8
Count
99.1
98.8
98.8
72.2
Spatial
98.3
98.2
87.5
63.3
Scale
92.8
92.6
38.2
26.3
19

---

Error in Layout Generation (Vicuna 13B)
(a) Generating unspecified objects
(b) Missing some objects
vetricle wheels frome one behind the other, propatted by pedies steered attached
traffic light signpost at crossroads intersection
library filled with kids reading books
dirt
ento table
road inxt to building..
tront wheel.
Text-to-Layout generation error examples. The Vicuna 13B sometimes (a) generates objects that are not specified in prompts (e.g., 'chair' in the 1st example) or (b) misses objects that mentioned in prompts (e.g., 'building" in the 5th example).
Figure 12:
Error in Layout-to-lmage Generation (GLIGEN)
(a) Wrong objects
(b) Missing objects
(c) Wrong object details
and bench: bench left to suitcass
assortment fruits vegetable wel together
umbrella
sitting red
oven with the dooor
Figure 13: Layout-to-Image generation error examples. The GLIGEN sometimes (a) generates wrong objects (e.g., 'suitcase', instead of 'bench', in the 1st example), (b) misses some objects (e.g., 'umbrella" in the 3rd example), or (c) misses object details (e.g., 'closed oven' instead of "oven with the door open' in the 5th example).
object is missing. For CLIP (ViT-B/32) [32]. we encode the generated image and text prompts, and take the cosine similarity between the visual and text features as the final score.
Inference time. On single A6000 GPU, each VPEVAL module typically takes less than to run.
Program generation API cost. Using GPT-3.5 Turbo the program generator, VPEVAL takes about $0.007 (less than cent) to generate programs from prompt (4K input tokens, 0.5K output tokens).
F TIFA Prompt Element Analysis Details
To better understand the characteristics of the TIFA prompts [36] that we use in open-ended evaluation, we randomly sample 20 prompts and label whether each prompt has any elements that require an understanding of spatial layouts, as mentioned in the main paper Sec. 5.3. Each prompt has multiple elements about objects, attributes, spatial layouts, etc. For example, "a zebra to the right fire hydrant." has the elements: 'zebra', 'right', and 'fire hydrant'. From the 20 prompts, there 81 total elements, and only 11/81 (13.6%) are relevant to object layouts (e.g., 'right'). The remaining 70 elements all related to objects attributes (e.g., "a white cat with black ears and markings" has the elements 'cat', 'white cat', 'ears', 'black ears', 'markings', and 'black markings").
nnttps://ppeaa. wwt.:/iiiiicommmiiininn...
20

---

retlatic Pomeranian 1080s professionnal wesetler with and face paint bright wesetling tights with bright boots
and circle with cm radius
Figure 14: Images generated by VPGEN (Vicun++GLIGEN) from challenging, non-canonical prompts.
Evaluation Program
Evaluation Results & Visual Textual Explanation
zebra the right of fire hydrant.
objectEvalling. "sebra");
Incorrect X
"sobra" object found.
hydranc.
objectivallmng hydrent"];
Cornect F
"fire hydrant" object found.
zebra the right hydrant.
spe "sobra,fire lydrarc/ighht
Incorrect
no "sebra" object found.
aebra to the right of fine hydrennt
vealima, "what animal to the right of the fire bytheent
Incorect
Q: what animal to the right the fire hydrant? zebra
white with black objectEvalimg, "cat"); markings
Correct
"cat" object found.
enrs and maricings gs
vea/ing, "whit cat?",
Correct E
white cot with black ears and markinngs
veoling. "does the cat have Black ears?",
Incorrect X
Q: does the cat have black cars? A: no
white cot with black and markings
veoling. "does the cat have black markings?
Incorrette
Q: does the black markings?
Figure 15: Examples of our VPEVAL evaluation programs and module outputs. Each module covers different parts of the text prompts (see Appendix D for discussion).
G Limitation and Broader Impacts
Limitations. Since our LMs were primarily trained with English-heavy datasets, they might not well understand prompts written in non-English languages. Likewise, as generation/evaluation modules were primarily trained with natural images, they might not well handle images in different domains (e.g., fine-arts, medical images, etc). Note that our framework supports easy extensions so that users can update existing modules or add new ones with better domain-specific knowledge.
Generating evaluation programs using LLMs can be costly both in terms of computational resources and price (API cost). Thus, we release the evaluation programs so that users not have to re- re-generate the evaluation programs. In addition, will release public LM (finetuned for evaluation program generation using ChatGPT outputs) that can run on local machines.
Broader Impacts. While our interpretable VPGEN framework can beneficial many applica- tions (e.g., user-controlled image ceneration/mmniiulaatio and data augmentation). it could also be used for potentially harmful cases (e.g., creating false information or misleading images), like other image generation frameworks.
21

---

Table 11: The prompt templates and evaluation programs for kisee-baedd evaluation. We also insert grammar tokens like <a> that are replaced with "an" or "a" maintain correct grammar.
Spatel
Rraleing
adgs Surrnal "ceass"
pare Fat "Sentt"
seygo plotiill cayy
look ooe tha "deett"
20 uagargg
taslivede vegger
lieetier "Cons" urilee
aabous of il El- vetuangg
ssaetiree 'dentt cmptiee
sptec pipe "eett""
sanccl my
oajto]
H
License
We will make our code and models publicly accessible. We use standard licenses from the community and provide the following links to the licenses for the datasets, codes, and models that used in this paper. For further information, please refer to the links.
FastChat (Vicuna): Apache License 2.0
GLIGEN: MIT
Grounding DINO: Apache License 2.0
DPT: MIT
LAVIS (BLIP-2): BSD 3-Clause
TIFA: Apache License 2.0
minDALL-E: Apache License 2.0, CC-BY-NC-SA 4.0
DALL-E mini (DALL-E Mega): Apache License 2.0, MIT
EasyOCR: Apache License 2.0
OpenAl API (ChatGPT): MIT
Diffusers: Apache License 2.0
Transformers: Apache License 2.0
PyTorch: BSD-style
22


Reviews:
[{"review_id": "2D8u7dygma", "reviewer": "Reviewer_aEEd", "summary": "This paper delves into the intersection of large language models (LLMs) and their applications in vision-and-language tasks, specifically focusing on text-to-image (T2I) generation. The authors identify a gap in the existing literature: no thorough analysis has been conducted on the synergy of LLMs and various visual modules for \"complex\" T2I generation tasks.\n\nTo address this they propose two novel frameworks: VPGEN: This framework offers a step-by-step approach to T2I generation, segmenting the process into three distinct stages: object/count generation, layout generation, and image generation. They use Vicuna to fine-tune to manage the first two stages, and the results demonstrate improved control over layout creation. The final stage, image generation, incorporates existing models like GLIGEN. Notably, VPGEN’s design capitalizes on the inherent knowledge of pre-trained LLMs, granting it the ability to recognize objects that have not been predefined—surpassing the capabilities of older layout-guided T2I techniques.\nVPEVAL: This authors propose this for evaluation of the generated images. Unlike conventional T2I evaluation methods which primarily gauge visual quality and image-text alignment through a singular visual module, VPEVAL emphasizes interpretability and a multiple modules to evaluate the generated image. In essence, it employs evaluation programs that activate a variety of visual modules, to distinct T2I skills.\nThe papers findings indicate that VPGEN+GLIGEN combination showcases relatively better performance, especially when precision in layouts and spatial relationships is paramount. The VPEVAL evaluation method seems to aligns with human assessment.\nIn summary the contributions are:\nThe introduction of VPGEN, an interpretable T2I generation framework that dissects the T2I process into three modules.\nThe proposal of VPEVAL, an evaluation framework for T2I tasks that enhances the explainability and thorough analysis by invoking diverse visual modules.\nA detailed analysis of various T2I models, highlighting the superior layout control of VPGEN and the human-centric alignment of VPEVAL.\n", "questions": "1. How would the authors compare their approach with Visual chatgpt ( https://arxiv.org/pdf/2303.04671.pdf) . This seem to have a image generation module? \n\n2. Does quantizing the bounding box coordinates into discrete bins loses precision? does this lead to misalignment between the predicted layout and actual image content.\n\n3. Unseen Objects: The ability to generate layouts of objects not seen during training (e.g., 'pikachu') can be an advantage, but it also raises questions about the model's ability to accurately represent unfamiliar objects in space. How well can it handle a completely novel entity in terms of spatial characteristics, especially when combined with other objects?\n\n4. How many in-context examples are used in the for the VPEval task? \n\n5. Can the authors provide results for count greater than 4 ( fig 5) . and can the model generate images like \" generate a circle with 3 cm radius and generate a circle with 6 cm radius , essentially saying one smaller than the other ? \n\n6. The model has a limited encoding capacity to represent all possible object combinations and numbers. So  to understand if an unusual or out-of-distribution combinations at test time could confuse the layout prediction stage, can the authors provide some complex compositional image generation results? \n\n7. what is the inference time?", "limitations": "The authors have spoken about the limitations. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall its a great work and the paper addresses some fundamental limitations in a unique way.\nThe paper addresses a very important problem in image generation realm. Most of the existing image generation models including the ones trained on massive datasets, are not good at generating images with spatial consistency, generating the correct number of objects , or understanding the size of the objects. This bias has been highlighted by various papers since 2018 and is still existing in modern generative models. The paper proposes a solution to the same by combining LLMs fine tuned on text to layout pairs and layout to image generation model. This fine tuning provides the LLMs with the ability to understand the spatial etc relationships between the generated objects. Thus removing a limitation from the prior results.\nThe proposed VPEval combined several modules to evaluate the generated quality of the images and also interpretable due to the generated programs. Using such visual programming is novel.\nAdaptability - as the modules in image generation and image evaluation improve the proposed approach would also improve in performance.\nOffering visual+textual explanations is a strength as it increases the interpretability of the evaluation.", "weaknesses": "The proposed 5 evaluation modules might still not capture the generated image quality and other complex semantics. Other image quality could be incorporated and evaluated.\n\nThe paper emphasizes heavily on the two step generation - object count and layout generation , but fails to conduct any ablation studies to prove the effectiveness of this pipeline. Since the process is decomposed into different steps, the interplay and coherence between these steps are vital. Errors or inconsistencies in earlier steps (like object/count generation) could cascade and affect the final image's quality.\n\nRelying on bounding boxes to represent layout may be simplistic. The boxes encode basic spatial relations for sure but might miss more detailed pose, occlusion, and depth information.\n\nAssociating object names in texts to layouts could be unreliable for ambiguous or synonymous words. The model may lack grounding to map words to visual concepts.\nFor example, consider the word \"bat\" - this could refer to: A baseball bat A flying bat animal\nOr the word \"apple\" could refer to: The fruit apple The technology company Apple\nWithout proper grounding between language and vision, the model may struggle to determine which visual concept is being referred to based on just the word alone.\nSome detailed examples to illustrate this:\nThe text \"a man holding a bat\" is ambiguous - is this referring to a baseball bat or a flying animal bat? The model may wrongly depict it without the proper grounding. The text \"a logo of an apple\" could wrongly depict the fruit when the technology company was meant. Synonyms like \"couch\" vs \"sofa\" would need to be mapped to the same visual concept. So in summary, relying purely on language without grounding it properly to visual concepts can lead to ambiguity in mapping words to the intended visual representations. Providing more context and grounding is important to resolve this.\n\nGranularity of Layout Representation: VPGEN decomposes bounding box coordinates into a [0,1] scale and quantizes them into 100 bins. Such discretization can potentially lead to a loss of finer details in bounding box representation, which might impact the accuracy of object placement in the generated images.\n\n2 human evaluators are slightly less to conclude that the results align with human and also the setting of human evaluation is a bit unclear. Like humans evaluate it subjectively (esp. with open ended tasks) . So could the authors shed some light on this setting? Also given that the correlation doesnt seem very high for objects ( 63.7?)\n\nMinor comments:\nThe word skill has been mentioned earlier without much context which might be ambiguous to the readers ( so just a minot presentation suggestion) .\nExplaining what the paper means by \"challenging\" T2I tasks might be helpful in terms of better clarity and presentation.\nScaling the approach to generate complex high-resolution images with many objects and intricate relations may be difficult. The text-to-layout-to-image pipeline has limitations. Any further results to prove or disprove this might be helpful.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the useful feedback and for pointing out our strengths in addressing important issues in the image generation community regarding spatial control and having an adaptable/interpretable evaluation framework.\n\n**W1. Image quality metric (FID).** Please see the general response.\n\n**W2. Error propagation analysis.** We analyze the performance of each step in the VPGen pipeline to determine how much error gets propagated.\n\n1. _Object Recall_: The ratio of whether correct objects are included in the generated layouts.\n2. _Object Count_: The ratio of the correct number of objects is included in the generated layouts.\n3. _Layout Accuracy_: VPEval accuracy of the generated layouts (using the generated layouts as the object detection results).\n4. _Image Accuracy_: VPEval accuracy of the final image by GLIGEN (in Table 1).\n\nAs shown in Table B, the layout accuracy is much higher than image accuracy, especially for Count/Spatial skills. This indicates that the major error source is the image rendering step (with GLIGEN), and using a more accurate layout-to-image generation model in the future would improve the accuracy of the VPGen pipeline. We will incorporate the results in the paper.\n\nTable B. Step-wise error analysis of VPGen. For the ‘Spatial’ skill, “front” and “behind” splits are skipped in this table since Vicuna13B does not generate depth information.\n\n| Skills  | Vicuna13B Object Recall (%)  | Vicuna13B Object Count (%) | Vicuna13B Layout Accuracy (%) | Layout-to-Image (GLIGEN) Accuracy (%) |\n|---|:---:|:---:|:---:|:---:|\n| Object | 99 | 99 | 99 |  97 |\n| Count | 99 | 99 | 99  |  72 |\n| Spatial | 98 | 98 | 88 | 34  |\n| Scale | 93 | 93 | 38 |  23 |\n\n**W3. Bounding box layout.**  While we can also guide image generation models with pose keypoints/segmentation/depth map, in VPGen, we choose the bounding box layout format because of its efficiency; bounding boxes require a much smaller number of tokens to represent compared to other formats (e.g., a xyxy-format bounding box can be represented with 4 tokens; while a 64x64 segmentation map requires 4096 tokens).\n\nPlease also note that our VPGen framework can be extended to pose/occlusion/depth guidance when we find an efficient way to generate them with LM and have access to an image generation model that can take such information as input.\n\n**W4. Words can be ambiguous.** We agree that the text prompts do not always have detailed information. While we focus on introducing the first VP framework for text-to-image generation, we suggest several ways to address the ambiguity below. As these ideas are involved with new method design/data collection/experiments, we leave this to future work.\n\n1. Multi-turn interaction\n```\nUser: “Draw a bat”\nSystem: “The word ‘bat’ is ambiguous, do you mean an animal or a baseball bat?”\nUser: “animal”\n```\n2. Show an exemplar image using a multimodal LM\n```\nUser: “Draw a bat like <image>”\n```\n\n**W5 & Q2. Granularity of layout.** We follow previous LM-based object detection to normalize/discretize the boxes [48, 49] (L137). In Table C, the 100 and 1000 bin settings show almost identical accuracies, suggesting that the current 100 bin discretization does not hurt the accuracy of object placements. We will add the results to the paper.\n\nTable C. 1000 vs. 100 bins.\n\n| # Bins  | Avg. VPEval skill-based Acc. | VPEval open-ended Acc. |\n|---|:---:|:---:|\n| 1000 | 47.8 | 69.1 | \n| 100 | 47.0 | 70.0  | \n\n**W6. Human eval setup.** For open-ended eval, we borrow the human evaluation scores (with two annotators) from TIFA [35]. Following TIFA, we use two human annotators for skill-based prompt evaluation as the prompts are short and straightforward (e.g., “two dogs”). Our annotators achieve high inter-annotator agreement with Cohen’s $\\kappa$ and Krippendorff’s $\\alpha = 0.85$ (> 0.8 indicates near-perfect [61; 62; 63], L304-306). Also, note that Spearman’s $\\rho=63.7$ of VPEval-Object is higher than other methods in Table 3.\n\n**Minor comments - presentation suggestions.** Thanks for the suggestions. We will explain/clarify ‘skill’ and ‘challenging’ in the paper.\n\n**Q1. Comparison to Visual ChatGPT.**  Visual ChatGPT directly calls a T2I model API (SD) with a given prompt to generate an image. In contrast, in our VPGen, LM has a more important role of semantic layout parsing with two steps: 1) generating object counts and 2) generating layouts given object counts.\n\n**Q2. Precision loss from discrete bins?** We find that using bins does not impact performance. Please see the answer to “W5 & Q2. Granularity of Layout Representation.”\n\n**Q3. Unseen object spatial control.** As suggested, we generate images with a series of prompts to see how well it handles placing unseen objects (Pikachus / Link) in different spatial relations (on a table / holding). As shown in Figure A (a,b,c) in the attached PDF, our layout generation LM can place the unseen objects in the correct locations.\n\n\n\n**Q4.  In-context examples used for VPEval.** For a fair comparison with TIFA [35], we follow the same 12 prompts used in TIFA’s question generation for in-context examples for VPEval program generation. We will include this detail in the paper.\n\n**Q5-1. Counts > 4.** We show the count skill results with 5, 6, and 7 in Table D. As expected, the trend continues from Fig 5, where higher counts are more challenging. We show the generation examples in Figure B (a,b,c,d) in the attached PDF. We will add the results to the paper.\n\nTable D. ‘Count’ accuracy of VPGen.\n\n| Counts |  1 |  2 |  3 |  4 |  5 |  6 |  7 |\n|---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n| VPEval Count Accuracy (%) $\\uparrow$ | 85 | 80 | 66 | 55 | 48 | 35 | 28 |\n\n\n\n**Q5-2. & Q6. Challenging/complex prompts.** Please see the general response.\n\n\n**Q7. Inference time.** On a single A6000 GPU, VPGen takes around 6s (2s for Vicuna 13B + 4s for GLIGEN) to generate an image and VPEval takes <1s (typically <0.5s) per evaluation module. We will add this information to the paper.\n"}, {"review_id": "nymSAF41XV", "reviewer": "Reviewer_1fZw", "summary": "This paper proposes a text-to-image (T2I) generation approach along with a new evaluation framework. This can be summarized in Figure 1. First, VPGen (Sec. 3) breaks T2I down into 3 steps, with an LM to generate a “program” of objects and layouts that is then fed into the final generation module. Second, VPEval (Sec. 4) is an interpretable evaluation for T2I that invokes diverse visual modules to evaluate the generated image and the prompt’s components.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: Both VPGen and VPEval are sound. It leverages existing tools to make the T2I process more controllable (via layout) and interpretable.\n\nS2: The paper is well-written.\n\nS3: Solid experiments. The paper compares various image generation models (Table 1) and provides results based on particular skills (skill-based prompts) and their combination (open-ended prompts). Further, the paper shows a good correlation of VPEval to human judgments. Though for this point, I do have a concern since the VPEval is “gamed” to showcase VPGen (see Weaknesses).\n\n", "weaknesses": "W1: It is unclear how generalizable/robust the proposed T2I approach and evaluation protocol are to “non-canonical” prompts shown in Figure 1. For VPGen, what if the prompts are really complex with multiple objects that interact with one another? What if the prompts do not include the numbers of objects (e.g., Pikachus on the table)? Similarly, since VPEval is geared toward showcasing VPGen, some aspects of evaluation are missing such as object attributes (L206) (e.g., a purple Pikachu and a pink Pikachu).\n\nW2: The novelty of both VPGen and VPEval should be highlighted further. Perhaps expand L99-105 in more detail and demonstrate L101-102 empirically. Furthermore, it would also be appreciated how the VPEval is compared to VQA-based approaches (L35) like TIFA and SeeTRUE (https://arxiv.org/abs/2305.10400), which in my opinion are also interpretable.  \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the useful feedback and for pointing out our strengths - a well-written paper and solid experiments. We hope to address your concerns below.\n\n**W1-1. Challenging/Complex prompts.** Please see the general response.\n\n**W1-2. Prompts without an exact number of counts.** Thanks for the suggestion, users can indeed provide prompts without specific numbers. We find that VPGen can successfully generate layouts/images even when prompts do not have explicit counts. As shown in Figure B (e) in the attached PDF, our VPGen generates two Pikachus from the prompt “Pikachus on the table”.\n\n**W1-3. Does VPEval check attributes?** Yes. While VPEval’s skill-based prompts do not have a specific split for ‘attribute’, our open-ended evaluation prompts do include prompts that have object attributes (e.g., “A red motorcycle parked by paint chipped doors.” has attributes ‘red’, ‘parked’, and ‘paint chipped’). In open-ended evaluation, the object attributes are usually evaluated with the VQA module. Figure 4 in the supplementary pdf also shows examples with attributes. For the mentioned example prompt “a purple Pikachu and pink Pikachu”, our program generator LM (ChatGPT) generates a program that checks for the pink and purple color attributes, as shown in Code A below.\n\nCode A. VPEval generated evaluation modules for the prompt “a purple Pikachu and pink Pikachu”.\n\n```python\nobjectEval(image, 'Pikachu')\nvqa(image, 'is there a purple Pikachu?', 'yes,no', 'yes')\nvqa(image, 'is there a pink Pikachu?', 'yes,no', 'yes')\ncountEval(objDet(image, 'Pikachu'), '==2')\n```\n\n**W2-1. Baseline (closed-vocab layout generator).** We will expand the discussion of text-to-layout-to-image generation (L99-105) with some more details. Regarding L101-102 _“However, the previous approaches train a new layout predictor module from scratch, so they are limited to predicting layouts with a predefined number of classes and cannot place new objects unseen during training\"_, we experiment with a layout generator with predefined 80 object classes of COCO (unlike our open-vocabulary LM Vicuna 13B). As shown in Figure A (d) in the attached PDF, the closed-vocabulary layout generator fails to generate layouts with the unseen object “Pikachu” and guides GLIGEN with the wrong layout (Note: The Pikachu in the background is generated by GLIGEN, and there is no layout bounding box for the Pikachu).\n\n\n\n**W2-2. VPEval compared to TIFA / SeeTRUE.** While both TIFA/SeeTRUE are also based on atomic questions and thus considered interpretable, the main difference between TIFA/SeeTRUE and our VPEval is the adoption of diverse evaluation models. While TIFA/SeeTRUE rely on a single evaluation module (VQA) that cannot cover different T2I skills, our VPEval adopts diverse evaluation models, including object detection, counting, OCR, etc. Sec. 5.4 shows that our diverse evaluation modules of VPEval achieve higher human correlation than TIFA on both skill-based and open-ended prompts. VPEval can also provide visual explanations (in the form of bounding boxes) and error messages, whereas TIFA/SeeTRUE cannot. Please also note that the SeeTRUE paper appeared on Arxiv on May 17, 2023, the same date as NeurIPS 2023 full paper submission deadline; we will cite SeeTRUE in the paper.\n"}, {"review_id": "wJfl8h1ump", "reviewer": "Reviewer_GZ2e", "summary": "This paper extends previous work in the vision and language space that use visual programs as an intermediate step, to the problem of text to image generation and subsequently, its evaluation. It proposes VPGen, a neuro-symbolic method that is composed of specific modules that count objects, generates layouts using a LLM and then generates an image using GLIGEN, a layout-to-image model. This allows for interpretable inspection of the intermediate steps as well as allows for more controllability in the image generation process. The paper also proposes VPEval, a method to evaluate T2I models using specific modules that also generate interpretable programs inspecting specific skills such as counting, existence of object, spatial, scale and text rendering. The proposed method (VPGen) outperforms previous approaches on skills such as count, spatial and scale, demonstrating that the approach allows for better fine-grained control over the image generation process. ", "questions": "* Does the domain of the image-text data used for fine-tuning the model in VPGen matter at all (for example if it were synthetic images vs real world images)?", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "## Quality, Originality and Significance\n* There have been several methods that were proposed over the last year for text to image generation, such as stable diffusion, Imagen, Parti, etc but the community still lacked an adequate method for evaluating the quality of the generated image in a way that is also interpretable. This paper provides both, an interpretable text to image generation pipeline that either outperforms or is competitive with existing (open source) state of the art text to image generation models, as well as a way to evaluate them.\n* Improvements in layout generation by using an LM, allowing generalization to unseen concepts instead of depending on pre-defined class set broadens the scope of the type of prompts that can be incorporated into this interpretable pipeline. \n* Human judgements and correlation analysis of existing evaluation methods compared with the proposed method demonstrates that VPEval is a suitable metric for T2I generation both in terms of interpretability while also being correlated with human judgement. \n\n## Clarity\n* The paper is very well written, and is very easy to follow. \n* The paper together with the appendix provides sufficient information on implementation details, qualitative examples and analysis of programs and prompts. \n", "weaknesses": "* Missing error analysis or categorization on the types of examples that VPGen fails to produce accurate images (both when the layout generator fails, as well as when GLIGEN fails would be useful). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the useful feedback and for pointing out our strengths of providing a strong interpretable T2I model, providing a much-needed interpretable T2I evaluation method, and having a well-written and detailed paper. We hope to address your questions and concerns below.\n\n**W1. Additional error category analysis.** Thanks for the suggestion, this is a good idea. We additionally investigated the failure cases of layout generation and show two error categories: 1) generating objects unspecified from the text prompt, and 2) not generating some objects mentioned in the text prompt. We show the error analysis results in Figure D in the attached PDF. We will include these examples/categorizations in the paper.\n\n\n**Q1. Training Domain Impact.** Our layout generation module (Vicuna 13B) in VPGen is trained on a mix of both simulated (PaintSkills dataset) and real-world images (MSCOCO/Flickr30K). We conduct an ablation study to determine if there is a difference between simulated/real-world images. We train the Vicuna on only PaintSkills and only Flickr30K and compare their results on both skill-based and open-ended prompts. For skill-based prompts, PaintSkills training shows a slightly higher average accuracy (46.8) than Flickr30K training (44.8). For open-ended prompts, PaintSkills training gets a VPEval score of (66.4), lower than Flickr30K training (71.1). This indicates that training on synthetic data can inject knowledge of specific skills such as counting and spatial relation understanding, but it is hard to cover diverse objects and attributes only with synthetic data. We will add these interesting findings to the paper.\n"}, {"review_id": "19WrxTG1aO", "reviewer": "Reviewer_PtVL", "summary": "This paper proposes a visual-program-based evaluation method VPEval to evaluate text-to-image models. Their method relies on LLM which can call different expert models in different tasks like object detection, OCR, spatial understanding, etc. to evaluate the consistency between the text and the generated image. The human evaluation shows that compared to traditional metrics, their evaluation method aligns with human better. Besides, they introduce a novel interpretable step-by-step text-to-image generation framework VPGen to improve the consistency between generated images and texts in count, spatial, and scale skills.", "questions": "1.\tAs the evaluation is based on ChatGPT, it would be good to include a API cost analysis.", "limitations": "The authors listed and discussed the limitations including the suboptimal performance with non-English prompts and unnatural images. Potential broader impacts like generating harmful images are also discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper writing is clear\n2.\tTheir framework VPGen is simple and novel. I like the idea to first train a LLM that generates image high-level layouts from the given prompt, then generates the image based on the layout.\n3.\tVPGen is effective with improved consistency in counting/spatial/scale relationships. \n4.\tThe evaluation framework VPEval based on visual program is also novel and has better interpretability compared to previous methods. Besides, it aligns with human judgement better.", "weaknesses": "1.\tThis paper is highly related and based on visual programing. Although the author cites the related papers like Visual Programming [11] and ViperGPT [12] in the related work section L110, a brief introduction of visual programming and a discussion of the difference between [11,12] and this paper are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the useful feedback and for pointing out our strengths of the effectiveness of VPGen and VPEval, as well as clear writing. We hope to address your questions and concerns below.\n\n**W1. Introduction of Visual Programming.** We’d like to bring your attention to Sec 1, where we explain how the recent LLM+visual modules (e.g., Visual Programming and ViperGPT) work (L38-31; “LLMs can also tackle certain vision-and-language tasks such as visual question answering and visual grounding, by generating visual programs (VP) that can control external visual modules and combine their outputs to get the final response”), and how our frameworks are different from them (L31-34; “However, no prior works have shown an analysis of combining LLMs and different visual modules for the challenging text-to-image (T2I) generation task. Our work proposes two novel interpretable/explainable VP frameworks combining LLMs and visual modules for T2I generation and evaluation”). In the next version, we will further expand the introduction of the Visual Programming framework and discuss the difference between the existing works and ours to make it more clear.\n\n**Q1. VPEval API Costs.** Thanks for the suggestion. VPEval costs about $0.007 (less than 1 cent) to evaluate one image with the ChatGPT 3.5 turbo API. We will add this information to the paper. Please also note that we will release all the generated programs from our evaluations for others to use (L225-227).\n"}, {"review_id": "4Tj6eT3GeA", "reviewer": "Reviewer_bJid", "summary": "This paper makes two contributions. VPGen is a T2I generation framework that first generates object/count, then layout, and finally image. VPEval is a T2I evaluation pipeline that provides a more comprehensive analysis correlated to human. Authors demonstrate that the step-by-step VPGen approach generates images more aligned with the input text compared to stable diffusion and other baselines. This is evaluated using the newly proposed VPEval scores.", "questions": "(1) It will be great if authors can also provide the standard evaluation benchmarks.\n\n(2) Can authors explain their design choice for using LM in VPGen? \n\n(3) More visualizations results will be appreciated.", "limitations": "Yes, authors adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed VPGen decomposition demonstrates better controllability for image generation, especially in terms of following the exact object count and spatial relation from text prompts. The new VPEval score offers a more comprehensive evaluation beyond just image quality. I believe this score will be useful for evaluating controllable generation.", "weaknesses": "Result section lacks the standard evaluation metric for image generation such as the FID and IS score. While the focus of this paper is on better controllability, and the proposed VPEval is clearly a better choice, I still like to see some benchmark on the image quality and diversity. \n\nFirst step of VPGen seems limited. In the examples provided, text has to include the exact number for each object. So is the first step just doing text parsing? Will it still work if I give it a sentence like “a small cozy office room”?\n\nAlso the motivation for training a LM is not clear to me. Many existing works can generate the layout bounding boxes directly, what is the extra benefit of training a LM? Can you do interactive editing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the useful feedback and for pointing out our strengths in providing useful image generation and evaluation frameworks. We hope to address your questions and concerns below.\n\n**W1. Standard image quality metric (FID).** Please see the general response.\n\n**W2-1. Prompts without an exact number of counts.** Thanks for the suggestion, users can indeed provide prompts without specific numbers. We find that VPGen can successfully generate layouts/images even when prompts do not have explicit counts. As shown in Figure B (e) in the attached PDF, our VPGen generates two Pikachus from the prompt “Pikachus on the table”.\n\n\n**W2-2. Prompts where details are unspecified.**\nRegarding your specific prompt “a small cozy office room”, we show two images from two different checkpoints (Flickr30k vs. Flickr30k+COCO+Paintskills) in Figure C (c) in the attached PDF. Note that the bounding boxes of the Flickr30k entities dataset are explicitly grounded to the Flickr30k captions, while the COCO captions and bounding boxes are annotated separately. This results in two different layout generation styles: 1) COCO-style - LM generates object layouts even if they are not explicitly mentioned in the prompt; 2) Flickr30k entities style - LM only generates objects that are explicitly mentioned in the prompt, and the downstream image generation model (GLIGEN) generates the image details. Users can construct their text-layout grounding style for their use cases. We add visualizations of such prompts to the paper.\n\n\n**W3 & Q2. Motivation for training an LM for VPGen.** In our initial experiment, we tested ChatGPT to generate spatial layouts by showing in-context examples and found the generated layouts were often inaccurate or not meaningful. For example, ChatGPT often generates a list of bounding boxes that is the same size as the entire image (e.g., `[object 1 (0,0,99,99), object 2 (0,0,99,99) object3 (0,0,99,99)]`). We conjecture that this is because their training corpus might not include a lot of bounding boxes. Then we decided to collect text-layout annotations and finetune a language model to generate spatial layouts from text prompts. \n\nFor a quantitative comparison, we implement VPGen with ChatGPT-3.5-turbo with 36 in-context examples covering different skills and compare it with (open-sourced) Vicuna13B trained on Flickr30k+COCO+PaintSkills. As shown in Table B, Vicuna13B-based VPGen shows higher skill-based and open-ended VPEval accuracies than ChatGPT-based VPGen. We will add this interesting result to the paper.\n\nTable B. ChatGPT vs. Vicuna13B.\n\n\n| Model  | VPEval Skill-based Acc. (%) | VPEval Open-ended Acc. (%) |\n|-----------|------------|-------------|\n| ChatGPT (36 examples) + GLIGEN |   42.2 |  66.3 |\n| Vicuna13B + GLIGEN | **47.0**  | **70.0**  |\n\n\n\n**Q3. More Visualizations.** Thanks for the suggestion. In the attached PDF, we provide different visualization results covering unseen objects (Figure A), counting (Figure B), challenging/complex prompts (Figure C), and layout error categorization (Figure D). We will include these and more visualizations in the paper.\n\n\n"}]

Author rebuttal:
We thank the reviewers for their valuable feedback and for recognizing our strengths:
- addressing an important/foundational problem in text-to-image generation (aEEd)
- developing strong/interpretable/useful text-to-image generation and evaluation frameworks (aEEd, 1fZw, GZ2e, PtVL, bJid)
- providing better spatial controllability for text-to-image generation (aEEd, 1fZw, PtVL, bJid)
- having generalizable/scalable frameworks (aEEd, GZ2e)
- providing solid experiments (1fZw, GZ2e, PtVL)
- and being well-written (1fZw, Gz2e).

**Attached figure-PDF**

In the attached PDF, we include visualizations that cover
- unseen objects (Figure A)
- counting beyond 4 (Figure B)
- challenging/complex prompts (Figure C)
- and layout error categorization (Figure D)

to address your comments.

**Common Answers**

**Reviewer aEEd W1. and bJid W1. - image quality metric (FID).** Thanks for the suggestion. In Table A, we compare our VPGen (Vicuna13B+GLIGEN) to its backbone Stable Diffusion (SD) v1.4, in FID (30K images of COCO val 2014 split) as well as VPEval Acc. Both VPGen checkpoints show better Skill-based Acc than SD v1.4 while achieving comparable Open-ended Acc. and FID.
In FID (lower the better), we find VPGen (Flickr30k) < SD v1.4 < VPGen (Flickr30k+COCO+Paintskills). We think a bit of increase (but still reasonably good) in the FID of the “Flickr30k+COCO+Paintskills'' checkpoint is because the layouts of PaintSkills are different from those of natural scenes (COCO and Flickr30k). We will add the results to the paper.

Table A. VPEval accuracy and FID.

| Model  | VPEval Skill-based Acc. (%) $\uparrow$ | VPEval Open-ended Acc. (%) $\uparrow$ | FID (COCO 30K) $\downarrow$  |
|---|:---:|:---:|:---:|
| SD v1.4 | 37.4   | 70.3  | 16.5 |
| VPGen (Flickr30k+COCO+Paintskills) | **47.0**   | 70.0  | 20.1 |
| VPGen (Flickr30k) | 44.6   | **71.2**  | **15.9** |


**Reviewer aEEd Q5-2. & Q6. and Reviewer 1fZw W1-1. Challenging/complex prompts.** As requested, we show images generated with the complex prompts (1) “A realistic photo of a Pomeranian dressed up like a 1980s professional wrestler with neon green and neon orange face paint and bright green wrestling tights with bright orange boots” (from DrawBench [24]) and (2) “a circle with 3 cm radius and a circle with 6 cm radius”, in Figure C (a,b) in the attached PDF.

VPGen understands the important parts of the prompts (e.g., generating ‘realistic’, ‘Pomeranian dog’, bright orange boots’; generating layouts of two circles in different sizes), but misses some aspects (e.g., the bigger circle is not twice the size of the smaller circle). This is probably because the training prompts (COCO, PaintSkills, Flickr30k) do not include many prompts written in these styles. We believe that scaling datasets with diverse sources can further improve VPGen.

Meta-review: The work received positive reviews. The reviewers found the approach simple and novel and the paper to be well written with good experiments. The rebuttal successfully addressed most of reviewer’s concerns. Among the negatives, the reviewers wished to see further error analysis and expressed some concern about the robustness of VPEval.

The AC agrees with the overall positive sentiment and believes the community would benefit from further exploring ideas such as VPEval which involve programatic verification of T2I models. VPGen results also indicate a greater degree of control over layout and semantics compared to existing literature. Hence the AC is happy to recommend acceptance. 

The AC requests the authors to incorporate the suggestions from reviewers and any new results presented during author-reviewer discussion. Congratulations!
Decision: Accept (poster)