Title: Visual Programming for Step-by-Step Text-to-Image Generation and Evaluation
Venue: NeurIPS 2023
Accepted: yes
Keywords: text-to-image generation; visual programming; text-to-image evaluation; step-by-step generation; interpretability; explainability
Abstract: As large language models have demonstrated impressive performance in many domains, recent works have adopted language models (LMs) as controllers of visual modules for vision-and-language tasks. While existing work focuses on equipping LMs with visual understanding, we propose two novel interpretable/explainable visual programming frameworks for text-to-image (T2I) generation and evaluation. First, we introduce VPGen, an interpretable step-by-step T2I generation framework that decomposes T2I generation into three steps: object/count generation, layout generation, and image generation. We employ an LM to handle the first two steps (object/count generation and layout generation), by finetuning it on text-layout pairs. Our step-by-step T2I generation framework provides stronger spatial control than end-to-end models, the dominant approach for this task. Furthermore, we leverage the world knowledge of pretrained LMs, overcoming the limitation of previous layout-guided T2I works that can only handle predefined object classes. We demonstrate that our VPGen has improved control in counts/spatial relations/scales of objects than state-of-the-art T2I generation models. Second, we introduce VPEval, an interpretable and explainable evaluation framework for T2I generation based on visual programming. Unlike previous T2I evaluations with a single scoring model that is accurate in some skills but unreliable in others, VPEval produces evaluation programs that invoke a set of visual modules that are experts in different skills, and also provides visual+textual explanations of the evaluation results. Our analysis shows that VPEval provides a more human-correlated evaluation for skill-specific and open-ended prompts than widely used single model-based evaluation. We hope that our work encourages future progress on interpretable/explainable generation and evaluation for T2I models.

Visual Programming for Text-to-Image Generation and Evaluation

Abstract

As large language models have demonstrated impressive performance in many domains, recent works have adopted language models (LMs) as controllers of visual modules for vision-and-language tasks. While existing work focuses on equipping LMs with visual understanding, we propose two novel interpretable/explainable visual programming frameworks for text-to-image (T2I) generation and evaluation. First, we introduce VPGEN, an interpretable step-by-step T2I generation framework that decomposes T2I generation into three steps: object/count generation, layout generation, and image generation. We employ an LM to handle the first two steps (object/count generation and layout generation), by finetuning it on textlayout pairs. Our step-by-step T2I generation framework provides stronger spatial control than end-to-end models, the dominant approach for this task. Furthermore, we leverage the world knowledge of pretrained LMs, overcoming the limitation of previous layout-guided T2I works that can only handle predefined object classes. We demonstrate that our VPGEN has improved control in counts/spatial relations/scales of objects than state-of-the-art T2I generation models. Second, we introduce VPEVAL, an interpretable and explainable evaluation framework for T2I generation based on visual programming. Unlike previous T2I evaluations with a single scoring model that is accurate in some skills but unreliable in others, VPEVAL produces evaluation programs that invoke a set of visual modules that are experts in different skills, and also provides visual+textual explanations of the evaluation results. Our analysis shows that VPEVAL provides a more humancorrelated evaluation for skill-specific and open-ended prompts than widely used single model-based evaluation. We hope that our work encourages future progress on interpretable/explainable generation and evaluation for T2I models.

1 Introduction

Large language models (LLMs) have shown remarkable performance on many natural language processing tasks, such as question answering, summarization, and story generation [1; 2; 3; 4; 5; 6; 7; 8]. Recent works have shown that LLMs can also tackle certain vision-and-language tasks such as visual question answering and visual grounding, by generating visual programs that can control external visual modules and combine their outputs to get the final response [9; 10; 11; 12; 13; 14; 15]. However, no prior works have combined LLMs and different visual modules for the challenging text-to-image (T2I) generation task. Our work proposes two novel interpretable/explainable visual programming (VP) frameworks combining LLMs and visual modules for T2I generation and evaluation.

First, we introduce VPGEN (Sec. 3), a new step-by-step T2I generation framework that decomposes the T2I task into three steps (object/count generation, layout generation, and image generation), where each step is implemented as a module and executed in an interpretable generation program, as shown

(a) VPGen: Step-by-Step T2I Generation

'two Pikachus on a table'

(b) VPEval: Explainable T2I Evaluation

'three dogs in the image'

imgGen(getLayout(getObjCounts(prompt), prompt), prompt)

countEval(img, 'dog', '==3')

>>> countEval(img, 'dog', '==3')

False

>>> prompt

= 'two Pikachus

on a table'

>>> obj2count

= getObjCounts(prompt)

# {'pikachu': 2, 'table": 1}

>>> layout

=

getLayout(obj2count, prompt)

# [('pikachu', (x1,y1,x2,y2),

#  ('pikachu', (x1,y1,x2,y2),

#  ('table',   (x1,y1,x2,y2)]

>>> img

= imgGen(

layout,

prompt)

Generation Program

Layout

Generation

Object / Count

Generation

Image

Generation

Generation Modules

OCR

Count

Object

Detect

Scale

Spatial

Evaluation Modules

VQA

Generated Image

There are

2

'dog'

objects,

not

3

.

Visual + Textual

Explanation of

Evaluation results

in Fig. 1 (a). We employ Vicuna [16], a powerful publicly available LLM, to handle the first two steps (object/count generation and layout generation), by finetuning it on text-layout pairs from multiple datasets [17; 18; 19], resulting in improved layout control for T2I generation. For the last image generation step, we use off-the-shelf layout-to-image generation models, e . g ., GLIGEN [20]. Our generation framework provides more interpretable spatial control than the widely used end-to-end T2I models [21; 22; 23; 24]. Furthermore, VPGEN leverages the world knowledge of pretrained LMs to understand unseen objects, overcoming the limitation of previous layout-guided T2I works that can only handle predefined object classes [25; 26; 27; 28]. In our analysis, we find that our VPGEN (Vicuna+GLIGEN) could generate images more accurately following the text description (especially about object counts, spatial relations, and object sizes) than strong T2I generation baselines, such as Stable Diffusion [21] (see Sec. 5.2).

Next, we introduce VPEVAL (Sec. 4), a new interpretable/explainable T2I evaluation framework based on evaluation programs that invoke diverse visual modules to evaluate different T2I skills [19] (see Sec. 4.1), and also provide visual+textual explanations of the evaluation results. Previous T2I evaluation works have focused on measuring visual quality [29; 30] and image-text alignment with a single visual module based on text retrieval [31], CLIP cosine similarity [32; 33], captioning [25], object detection [34; 19; 35], and visual question answering (VQA) [36; 37]. However, we cannot interpret the reasoning behind the scoring; i . e ., why CLIP assigns a higher score to an image-text pair than another. In addition, these modules are good at measuring some skills, but not reliable in other skills ( e . g ., VQA/CLIP models are not good at counting objects or accurately parsing text rendered in images). In VPEVAL, we break the evaluation process into a mixture of visual evaluation modules, resulting in an interpretable evaluation program. The evaluation modules are experts in different skills and provide visual+textual ( i . e ., multimodal) result/error explanations, as shown in Fig. 1 (b). We evaluate both multimodal LM and diffusion-based T2I models with two types of prompts: (1) skill-based prompts (Sec. 4.3) and (2) open-ended prompts (Sec. 4.4). Skill-based prompts evaluate a single skill per image, whereas open-ended prompts evaluate multiple skills per image. We adapt a large language model (GPT-3.5-Turbo [38]) to dynamically generate an evaluation program for each open-ended prompt, without requiring finetuning on expensive evaluation program annotations.

In our skill-based prompt experiments, while all T2I models faced challenges in the Count, Spatial, Scale, and Text Rendering skills, our VPGEN shows higher scores in the Count, Spatial, and Scale skills, demonstrating its strong layout control. We also show a fine-grained sub-split analysis on where our VPGEN gives an improvement over the baseline T2I models. In our open-ended prompt experiments, our VPGEN achieves competitive results to the T2I baselines. Many of these openended prompts tend to focus more on objects and attributes, which, as shown by the skill-based evaluation, T2I models perform quite well in; however, when precise layouts and spatial relationships are needed, our VPGEN framework performs better (Sec. 5.3). In our error analysis, we also find

that GLIGEN sometimes fails to properly generate images even when Vicuna generates the correct layouts, indicating that as better layout-to-image models become available, our VPGEN framework will achieve better results (Sec. 5.3). We also provide an analysis of the generated evaluation programs from VPEVAL, which proved to be highly accurate and comprehensive in covering elements from the prompts (Sec. 5.4). For both types of prompts, our VPEVAL method shows a higher correlation to human evaluation than existing single model-based evaluation methods (Sec. 5.4).

Our contributions can be summarized as follows: (1) VPGEN (Sec. 3), a new step-by-step T2I generation framework that decomposes the T2I task into three steps (object/count generation, layout generation, and image generation), where each step is implemented as a module and executed in an interpretable generation program; (2) VPEVAL (Sec. 4), a new interpretable/explainable T2I evaluation framework based on evaluation programs that execute diverse visual modules to evaluate different T2I skills and provide visual+textual explanations of the evaluation results; (3) comprehensive analysis of different T2I models, which demonstrates the strong layout control of VPGEN and high human correlation of VPEVAL (Sec. 5). We will release T2I model-generated images, VPEVAL programs, and a public LM (finetuned for evaluation program generation using ChatGPT outputs). We hope our research fosters future work on interpretable/explainable generation and evaluation for T2I tasks.

2 Related Works

Text-to-image generation models. In the T2I generation task, models generate images from text. Early deep learning models used the Generative Adversarial Networks (GAN) [39] framework for this task [40; 41; 42; 31]. More recently, multimodal language models [43; 22] and diffusion models [44; 45; 21; 46] have gained popularity. Recent advances in multimodal language models such as Parti [47] and MUSE [48], and diffusion models like Stable Diffusion [21], UnCLIP [23], and Imagen [24], have demonstrated a high level of photorealism in zero-shot image generation.

Bridging text-to-image generation with layouts. One line of research decomposes the T2I generation task into two stages: text-to-layout generation and layout-to-image generation [25; 26; 27; 28]. However, the previous approaches focus on a set of predefined object classes by training a new layout predictor module from scratch and therefore cannot place new objects unseen during training. In contrast, our VPGEN uses an LM to handle layout generation by generating objects/counts/positions in text, allowing flexible adaptation of pretrained LMs that can understand diverse region descriptions.

Language models with visual modules. Although large language models (LLMs) have shown a broad range of commonsense knowledge, most of them are trained only on text corpus and cannot understand image inputs to tackle vision-and-language (VL) tasks. Thus, recent works explore tackling VL tasks by solving sub-tasks with external visual modules and combining their outputs to obtain the final response [9; 10; 11; 12; 13; 14; 15]. The visual sub-tasks include describing images as text, finding image regions relevant to the text, editing images with text guidance, and obtaining answers from a VQA model. However, existing work focuses on converting visual inputs into text format so that LLMs can understand them. Our work is the first work using visual programming for interpretable and explainable T2I generation and evaluation.

Evaluation of text-to-image generation models. The text-to-image community has commonly used two types of automated evaluation metrics: visual quality and image-text alignment. For visual quality, Inception Score (IS) [29] and Fréchet Inception Distance (FID) [30] have been widely used. For image-text alignment, previous work used a single model to calculate an alignment score for image-text pair, based on text retrieval [31], cosine similarity [33], captioning [25], object detection [34; 19; 35], and visual question answering (VQA) [36; 37]. In this work, we propose the first T2I evaluation framework VPEVAL, based on interpretable and explainable evaluation programs which execute a diverse set of visual modules ( e . g ., object detection, OCR, depth estimation, object counting). Our VPEVAL evaluation programs provide visual+textual explanations of the evaluation result and demonstrate a high correlation with human judgments.

3 VPGEN: Visual Programming for Step-by-Step Text-to-Image Generation

We propose VPGEN, a novel visual programming framework for interpretable step-by-step text-toimage (T2I) generation. As illustrated in Fig. 2, we decompose the text-to-image generation task into

Object/Count

Generation

Layout

Generation

Image

Genera6on

pikachu (2) table (1)

Given an image caption,

determine objects and their

counts to draw an image.

Caption: two Pikachus on a table

LM

two Pikachus on a table

pikachu (x1,y1,x2,y2)

pikachu (x1,y1,x2,y2)

table (x1,y1,x2,y2)

L2I

pikachu (x1,y1,x2,y2)

pikachu (x1,y1,x2,y2)

table (x1,y1,x2,y2)

Given an image caption and objects,

determine coordinates of the objects.

Caption: two Pikachus on a table

Objects: pikachu (2) table (1)

Visualized Layout

two Pikachus

on a table

LM

three steps: (1) object/count generation, (2) layout generation, and (3) image generation. In contrast to previous T2I generation works that use an intermediate layout prediction module [25; 26; 27; 28], VPGEN represents all layouts (object description, object counts, and bounding boxes) in text, and employs an LM to handle the first two steps: (1) object/count generation and (2) layout generation. This makes it easy to adapt the knowledge of pretrained LMs and enables generating layouts of objects that are unseen during text-to-layout training ( e . g ., 'pikachu'). For layout representation, we choose the bounding box format because of its efficiency; bounding boxes generally require fewer tokens than other formats. 1

Two-step layout generation with LM. Fig. 2 illustrates how our LM generates layouts in two steps: (1) object/count generation and (2) layout generation. For the first step, we represent the scene by enumerating objects and their counts, such as 'obj1 (# of obj1) obj2 (# of obj2)' . For the second step, following previous LM-based object detection works [49; 50], we normalize 'xyxy' format bounding box coordinates into [0 , 1] and quantize them into 100 bins; a single object is represented as 'obj (xmin,ymin,xmax,ymax)' , where each coordinate is within { 0 , · · · , 99 } .

Training layout-aware LM. To obtain the layout-aware LM, we use Vicuna 13B [16], a public state-of-the-art language model finetuned from LLaMA [51]. We use parameter-efficient finetuning with LoRA [52] to preserve the original knowledge of the LM and save memory during training and inference. We collect text-layout pair annotations from training sets of three public datasets: Flickr30K entities [17], MS COCO instances 2014 [18], and PaintSkills [19], totaling 1.2M examples. See appendix for more training details.

Layout-to-Image Generation. We use a recent layout-to-image generation model GLIGEN [20] for the final step - image generation. The layout-to-image model takes a list of regions (bounding boxes and text descriptions) as well as the original text prompt to generate an image.

4 VPEVAL: Visual Programming for Explainable Evaluation of Text-to-Image Generation

VPEVAL is a novel interpretable/explainable evaluation framework for T2I generation models, based on visual programming. Unlike existing T2I evaluation methods that compute image-text alignment scores with an end-to-end model, our evaluation provides an interpretable program and visual+textual explanations for the evaluation results, as shown in Figs. 3 and 5. We propose two types of evaluation prompts: (1) skill-based evaluation and (2) open-ended evaluation. In skill-based evaluation, we define five image generation skills and use a set of skill-specific prompts and evaluation programs, as illustrated in Fig. 3. In open-ended evaluation, we use a diverse set of prompts that require multiple image generation skills. We adopt a language model to dynamically generate an evaluation program for each text prompt, as shown in Fig. 5. In the following, we describe evaluation skills (Sec. 4.1), visual evaluation modules (Sec. 4.2), skill-based evaluation with visual programs (Sec. 4.3), and open-ended evaluation with visual program generator LM (Sec. 4.4).

Skill-based Interpretable Evaluation Program Evaluation Results & Visual + Textual Explanation

Correct

Incorrect

Object

Prompt:  'a photo of a dog'

Program:

objectEval(img, 'dog')

Count

Prompt:  '3 dogs'

Program:

countEval(img, 'dog', '==3')

Scale

Prompt:  'a laptop that is bigger than a sports ball'

Program:

scaleEval(img, 'laptop, sports ball, bigger')

Spatial

Prompt:  'a spoon is in front of a potted plant'

Program:

spatialEval(img, 'spoon, potted plant, front')

Text Rendering

Prompt:  'a poster that reads 'shop''

Program:

textEval(img, 'shop')

❌

✅

No 'dog'

object

found.

There are 5

'dog'

objects, not 3.

No (obj1, obj2)

pair of ('spoon',

'potted plant')

with 'front'

(z1 <

z2) found.

No 'sports ball'

object found.

No 'shop'

text

found.

'dog'

object found.

There are 3

'dog'

objects.

('spoon', 'potted

plant') with

'front'

(z1 < z2)

found.

('laptop', 'sports

ball') with

'bigger'

(area1 >

area2) found.

'shop'

text found.

4.1 Evaluation Skills

Inspired by skill-based T2I analysis of PaintSkills [19], our VPEVAL measures five image generation skills: Object, Count, Spatial, Scale, and Text Rendering. Powered by evaluation programs with expert visual modules, our VPEVAL supports zero-shot evaluation of images (no finetuning of T2I models is required), detecting regions with free-form text prompts, new 3D spatial relations (front, behind), and new scale comparison and text rendering skills, which were not supported in PaintSkills [19]. In Fig. 3, we illustrate the evaluation process for each skill.

Object. Given a prompt with an object ( e . g ., 'a photo of a dog'), a T2I model should generate an image with that object present.

Count. Given a prompt with a certain number of an object ( e . g ., '3 dogs'), a T2I model should generate an image containing the specified number of objects.

Spatial. Given a prompt with two objects and a spatial relationship between them ( e . g ., 'a spoon is in front of a potted plant'), a T2I model should generate an image that contains both objects with the correct spatial relations.

Scale. Given a prompt with two objects and a relative scale between them ( e . g ., 'a laptop that is bigger than a sports ball'), a T2I model should generate an image that contains both objects, and each object should be of the correct relative scale.

Text Rendering. Given a prompt with a certain text to display ( e . g ., 'a poster that reads 'shop' '), a T2I model should generate an image that properly renders the text.

4.2 Visual Evaluation Modules

To measure the skills described above in Sec. 4.1, we use eight expert visual evaluation modules specialized for different tasks. The modules provide visual+textual explanations with their score. Visual explanations are generated by rendering bounding boxes on the images and textual explanations are generated through text templates, as shown in Fig. 3 and 5. We provide the Python pseudocode implementation for each module in Fig. 4.

Module definitions. objDet detects objects in an image based on a referring expression text and returns them (+ their 2D bounding boxes and depth), using Grounding DINO [53] and DPT [54]. ocr detects all text in an image and returns them (+ their 2D bounding boxes), using EasyOCR [55].

def objDet(img, obj_text):

det_objs_3d = depth(img, det_objs_2d)

return det_objs_3d

OCR

Count

Eval

Object

Detect

Scale

Eval

Spatial

Eval

def scaleEval(img, obj1_text, obj1_text, relation):

objects = objDet(img, 'obj1_text,obj2_text')

if target_relation == 'bigger':

return any(objects[1].area > objects[0].area)

...

def objectEval(img, object_text):

objects = objDet(img, object_text)

return len(objects) > 0

def spatialEval(img, obj1_text, obj1_text, relation):

objects = objDet(img, 'obj1_text,obj2_text')

if target_relation == 'right':

return any(objects[1].x > objects[0].x)

...

def ocr(img):

det_texts = find_text(img)

return det_texts

Object

Eval

Text

Eval

def countEval(img, object_text, count):

objects = objDet(img, object_text)

return len(objects) == target_count

def textEval(img, target_text):

texts = ocr(img)

return target_text in texts

VQA

Eval

def vqaEval(img, question, answer_choices, target_answer):

answer = vqa_model(img, question, answer_choices)

return answer == target_answer

vqa answers a multiple-choice question using BLIP-2 (Flan-T5 XL) [56]. It can handle phrases that cannot be covered only with objDet or ocr ( e . g ., pose estimation, action recognition, object attributes). objectEval evaluates if an object is in an image using objDet . countEval evaluates if an object occurs in an image a certain number of times using objDet and an equation ( e . g ., '==3', '<5'). spatialEval evaluates if two objects have a certain spatial relationship with each other. For six relations (above, below, left, right, front, and behind), it compares bounding box/depth values using objDet . For other relations, it uses vqa . scaleEval evaluates if two objects have a certain scale relationship with each other using objDet . For three scale relations (smaller, bigger, and same) it compares bounding box areas by using objDet . For other relations, it uses vqa . textEval evaluates if a given text is present in an image using ocr .

4.3 Skill-based Evaluation with Visual Programs

For skill-based evaluation, we create text prompts with various skill-specific templates that are used for image generation and evaluation with our programs. See appendix for the details of prompt creation. In Fig. 3, we illustrate our skill-based evaluation in VPEVAL. Given text prompts that require different image generation skills, our evaluation programs measure image-text alignment scores by calling the relevant visual modules. Unlike existing T2I evaluation methods, our evaluation programs provide visual+textual explanations of the evaluation results.

4.4 Open-ended Evaluation with Visual Program Generator LM

Although our evaluation with skill-specific prompts covers five important and diverse image generation skills, user-written prompts can sometimes be even more complex and need multiple evaluation criteria ( e . g ., a mix of our skills in Sec. 4.1 and other skills like attribute detection). For example, evaluating images generated with the prompt 'A woman dressed for a wedding is showing a watermelon slice to a woman on a scooter.' involves multiple aspects, such as two women (count skill), 'a woman on a scooter' (spatial skill), 'dressed for a wedding' (attribute detection skill), etc . To handle such open-ended prompts, we extend the VPEVAL setup with evaluation programs that can use many visual modules together (whereas single-skill prompts can be evaluated with a program of 1-2 modules). We generate open-ended prompt evaluation programs with an LLM, then the evaluation programs output the average score and the visual+textual explanations from their visual modules. The program generation involves choosing which prompt elements to evaluate and which modules to evaluate those elements (see Fig. 5).

Open-ended prompts. For open-ended evaluation, we use 160 prompts of TIFA v1.0 human judgment dataset [36] (we refer to these prompts as 'TIFA160'). The dataset consists of (1) text prompts from COCO [18], PaintSkills [19], DrawBench [24], and Partiprompts [47], (2) images generated by five baseline models (minDALL-E [57], VQ-Diffusion [58], Stable Diffusion v1.1/v1.5/v2.1 [21]), and (3) human judgment scores on the images (on 1-5 Likert scale).

Generating evaluation programs via in-context learning. As annotation of evaluation programs with open-ended prompts can be expensive, we use ChatGPT (GPT-3.5-Turbo) [38] to generate evaluation programs via in-context learning. To guide ChatGPT, we adapt the 12 in-context prompts from TIFA [36]. We show ChatGPT the list of visual modules and example text prompts and programs, then ask the model to generate a program given a new prompt, as illustrated in Fig. 5. For

objectEval(image, 'ram');

objectEval(image, 'evergreen');

countEval(image, 'ram', '>1');

countEval(image, 'evergreen', '==1');

vqa(image, 'what is in the foreground?', 'grassy

slope,beach,field,forest', 'grassy slope');

...

ChatGPT

Evaluation Results & Visual + Textual Explanation

Incorrect

❌

no 'ram' object found.

Q: 'what is in the

foreground?' A: grassy

slope.

'evergreen' object found.

there are 8 'evergreen'

objects, not 1.

Correct

✅

Correct

✅

# Task description + module description

Given an image description, generate programs that verifies if

the image description is correct.

...

# In-context examples

Description: A man posing for a selfie in a jacket and bow

tie.

...

objectEval(image, 'man');

vqa(image, 'who is posing for a selfie?',

'man,woman,boy,girl', 'man')

...

# New text prompt

Description: A white slope covers the background, while the

foreground features a grassy slope with several rams grazing

and one measly and underdeveloped evergreen in the foreground.

Example text prompt

Example evaluation

program

Open-ended Interpretable Evaluation Program

Incorrect

❌

reproducible and accessible evaluation, we release the evaluation programs so that VPEVAL users do not have to generate the programs themselves. We will also release a public LM (finetuned for evaluation program generation using ChatGPT outputs) that can run on local machines.

5 Experiments and Results

In this section, we introduce the baseline T2I models we evaluate (Sec. 5.1), compare different T2I models with skill-based (Sec. 5.2) and open-ended prompts (Sec. 5.3), and show the humancorrelation of VPEVAL (Sec. 5.4).

5.1 Evaluated Models

We evaluate our VPGEN (Vicuna+GLIGEN) and five popular and publicly available T2I models, covering both diffusion models (Stable Diffusion v1.4/v2.1 [21] and Karlo [59]), and multimodal autoregressive language models (minDALL-E [57] and DALL-E Mega [60]). Stable Diffusion v1.4 is the most comparable baseline to VPGEN, because the GLIGEN [20] in VPGEN uses frozen Stable Diffusion v1.4 with a few newly inserted adapter parameters for spatial control.

5.2 Evaluation on Skill-based Prompts

Diffusion models outperform multimodal LMs. In Table 1, we show the VPEVAL skill accuracies for each model. The diffusion models (Stable Diffusion, Karlo, and our VPGEN) show higher overall accuracy than the multimodal LMs (minDALL-E and DALL-E Mega).

Count/Spatial/Scale/Text Rendering skills are challenging. Overall, the five baseline T2I models achieve high scores (above 93% except for minDALL-E) in Object skill; i . e ., they are good at

generating a high-quality single object. However, the models score low accuracies in the other skills, indicating that these skills (Count, Spatial, Scale, and Text Rendering) are still challenging with recent T2I models.

Step-by-step generation improves challenging skills. The bottom row of Table 1 shows that our VPGEN achieves high accuracy in Object skill and strongly outperforms other baselines in Count (+12.7% than Karlo), Spatial (+24.8% than Stable Diffusion v2.1) and Scale (+9.9% than Karlo) skills. This result demonstrates that our step-by-step generation method is effective in aligning image layout with text prompts, while also still having an interpretable generation program. All models achieve low scores on the Text Rendering skill, including our VPGEN. A potential reason why our VPGEN model does not improve Text Rendering score might be because the text-to-layout training datasets of VPGEN (Flickr30K Entities [17], COCO [18], and PaintSkills [19]) contain very few images/captions about text rendering, opening room for future improvements ( e . g ., finetuning on a dataset with more images that focus on text rendering and including a text rendering module as part of our generation framework).

Fine-grained analysis in Count/Spatial/Scale skills. To better understand the high performance of our VPGEN in Count, Spatial, and Scale skills, we perform a detailed analysis of these skills with fine-grained splits. In Fig. 6, we compare our VPGEN (Vicuna+GLIGEN) and its closest baseline Stable Diffusion v1.4 on the three skills. Overall : VPGEN achieves better performance than Stable Diffusion on every split on all three skills. Count : While both models show difficulties with counting larger numbers, our VPGEN model achieves better performance with 50+ accuracy on all four numbers. Spatial : Our VPGEN model performs better on all six spatial relations. VPGEN shows higher accuracy on 2D relations (left/right/below/above) than on 3D depth relations (front/behind), while the Stable Diffusion is better at 3D relations than 2D relations. Scale : VPGEN generates more accurate layouts with two objects of different sizes (bigger/smaller), than layouts with two objects of similar sizes (same).

1

2

3

4

0.0

0.5

1.0

Acc.

Count

left

right

below

above

front

behind

Spatial

bigger

smaller

same

Scale

SD v1.4

5.3 Evaluation on Open-ended Prompts

Table 2 shows the VPEVAL score on the open-ended TIFA160 prompts. We calculate the score by averaging accuracy from the modules (see Fig. 5 for an example). The overall score trend of the baseline models (diffusion models > multimodal LMs) is consistent with the skillbased prompts. Unlike the trend in skill-based evaluation (Sec. 5.2), our VPGEN (Vicuna+GLIGEN) methods achieve similar performance to the Stable Diffusion baseline, while also providing interpretable generation steps. We perform an analysis of the open-ended prompts used for evaluation. In these prompts, object descriptions and attributes are the dominating prompt element (86.4% of elements), whereas descriptions of spatial layouts, only account for 13.6% of elements. See appendix for more

details. For prompts from PaintSkills [19] where spatial layouts are more important, we find that our VPGEN scores much higher than Stable Diffusion v1.4 (71.0 v.s. 63.5) and Stable Diffusion v2.1 (71.0 v.s. 68.4). Note that GLIGEN used in VPGEN is based on Stable Diffusion v1.4, making it the closest baseline, and hence future versions of GLIGEN based on Stable Diffusion v2.1 or other stronger layout-to-image models will improve our results further. Also, we find that GLIGEN sometimes fails to properly generate images even when VPGEN generates the correct layouts (see following paragraph).

Stable Diffusion v1.4

Object

'an image of

a tennis racket'

Count

'an image of

4 bo�les'

Spa�al

'a train is to the

right of a cow'

Scale

'a giraffe that is

bigger than a cup'

Text

'a sign that says

'night' '

'A cat lying a top on

the desk on a laptop.'

Stable Diffusion v1.4

'A zebra to the right

of a fire hydrant.'

'a photo of bear and dining table;

dining table is below bear'

'a drop-top sports car coming

around a bend in the road'

COCO

DrawBench

PaintSkills

Par�Prompt

NGIT

NT

VPGEN sources of error: layout/image generation. Since our VPGEN separates layout generation and image generation, we study the errors caused by each step. For this, we manually analyze the images generated from TIFA160 prompts, and label (1) whether the generated layouts align with the text prompts and (2) whether the final images align with the text prompts/layouts. GLIGEN sometimes fails to properly generate images even when Vicuna 13B generates the correct layouts, indicating that when better layout-to-image models become available, our VPGEN framework will achieve higher results. We include more details in the appendix.

Qualitative Examples. In Fig. 7, we provide example images generated by our VPGEN and Stable Diffusion v1.4 (the closest baseline) with various skill-based (top) and open-ended (bottom) prompts. See appendix for more qualitative examples with skill-based/open-ended prompts, prompts with unseen objects, counting ≥ 4 objects, and error analysis.

5.4 Human Evaluation of VPEVAL

To measure alignments with human judgments and our VPEVAL, we compare the human-correlations of VPEVAL and other metrics on both skill-based prompts and open-ended prompts.

Human correlation of VPEVAL on skill-based prompts. We ask two expert annotators to evaluate 20 images, from each of the five baseline models, for each of the five skills with binary scoring (total 20 × 5 × 5 = 500 images). The inter-annotator agreements were measured with Cohen's κ [61] = 0.85 and Krippendorff's α [62] = 0.85, indicating 'near-perfect' ( κ > 0.8 or α > 0.8) agreement [63; 64; 62]. We compare our visual programs with captioning (with metrics BLEU [65], ROUGE [66], METEOR [67], and SPICE [68]), VQA, and CLIP (ViT-B/32) based evaluation. We use BLIP-2 Flan-T5 XL, the state-of-the-art public model for image captioning and VQA.

In Table 3, our VPEVAL shows a higher overall correlation with human judgments (66.6) than single module-based evaluations (CLIP, Captioning, and VQA). Regarding per-skill correlations, VPEVAL shows especially strong human correlations in Count and Text Rendering. As the BLIP-2 VQA module also shows strong correlation on Object/Spatial/Scale, we also experiment with VPEVAL † : using BLIP-2 VQA for objectEval/spatialEval/scaleEval modules (instead of Grounding

DINO and DPT), which increases human correlation scores. Note that our object detection modules visually explain the evaluation results, as shown in Fig. 3, and we can have an even higher correlation when we have access to stronger future object detection models.

Human correlation of VPEVAL on open-ended prompts. We generate visual programs with our program generation LM on TIFA160 prompts [36]. The dataset consists of 800 images (160 prompts × 5 T2I models) and human judgments (1-5 Likert scale) along with other automatic metrics (BLIP-2 captioning, CLIP cosine similarity, and TIFA with BLIP-2) on the images. Table 4 shows that our VPEVAL achieves a better human correlation with TIFA (BLIP-2), and our VPEVAL † version achieves an even higher correlation. The results indicate that using various interpretable modules specialized in different skills complements each other and improves human correlation, while also providing visual+textual explanations.

Human analysis on the generated programs. Lastly, we also measure the faithfulness of the generated evaluation programs. For this, we sample TIFA160 prompts and analyze the evaluation programs by (1) how well the modules cover elements in the prompt; (2) how accurate the module outputs are when run. We

Table 4: Human correlation on open-ended evaluation with Spearman's ρ .

find that programs generated by our VPEVAL have very high coverage over the prompt elements and high per-module accuracy compared to human judgment. We include more details in the appendix.

6 Conclusion

We propose two novel visual programming frameworks for interpretable/explainable T2I generation and evaluation: VPGEN and VPEVAL. VPGEN is a step-by-step T2I generation framework that decomposes the T2I task into three steps (object/count generation, layout generation, and image generation), leveraging an LLM for the first two steps, and layout-to-image generation models for the last image generation step. VPGEN generates images more accurately following the text descriptions (especially about object counts, spatial relations, and object sizes) than strong T2I baselines, while still having an interpretable generation program. VPEVAL is a T2I evaluation framework that uses interpretable evaluation programs with diverse visual modules that are experts in different skills to measure various T2I skills and provide visual+textual explanations of evaluation results. In our analysis, VPEVAL presents a higher correlation with human judgments than single model-based evaluations, on both skill-specific and open-ended prompts. We hope our work encourages future progress on interpretable/explainable generation and evaluation for T2I models.

Limitations & Broader Impacts. See Appendix for limitations and broader impacts discussion.

References

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL , oct 2019.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer, and Pre-training Bart. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL , 2020.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In NeurIPS , 2020.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR , 21:1-67, 2019.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways, 2022.

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned Language Models Are Zero-Shot Learners. In ICLR , 2022.

OpenAI. GPT-4 Technical Report, 2023.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling Instruction-Finetuned Language Models, 2022.

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA. In AAAI , 2022.

Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A. Smith, and Jiebo Luo. PromptCap: PromptGuided Task-Aware Image Captioning, 2022.

Tanmay Gupta and Aniruddha Kembhavi. Visual Programming: Compositional visual reasoning without training. In CVPR , nov 2023.

Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual Inference via Python Execution for Reasoning, 2023.

Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models, 2023.

Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action, 2023.

Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, Yun Wang, Linjun Shou, Ming Gong, and Nan Duan. TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs, 2023.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.

Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. IJCV , 123(1):74-93, 2017.

Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV , 2014.

Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In ICCV , 2023.

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: Open-Set Grounded Text-to-Image Generation. In CVPR , 2023.

Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10674-10685, 2021.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Image Generation. In ICML , 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. In NeurIPS , 2022.

Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis. In CVPR , 2018.

Fuwen Tan, Song Feng, and Vicente Ordonez. Text2Scene: Generating Compositional Scenes From Textual Descriptions. In CVPR , 2019.

Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao. Object-Driven Text-To-Image Synthesis via Adversarial Training. In CVPR , pages 12166-12174, 2019.

Jiadong Liang, Wenjie Pei, and Feng Lu. Layout-Bridging Text-to-Image Synthesis, 2022.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved Techniques for Training GANs. In NIPS , 2016.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NIPS , 2017.

Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks. In CVPR , 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever, Jong Wook, Kim Chris, Hallacy Aditya, Ramesh Gabriel, Goh Sandhini, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICML , 2021.

Yupan Huang, Hongwei Xue, Bei Liu, and Yutong Lu. Unifying Multimodal Transformer for Bi-directional Image and Text Generation. In ACM MM . Association for Computing Machinery, 2021.

Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic Object Accuracy for Generative Text-to-Image Synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-1, 2020.

Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking Spatial Relationships in Text-to-Image Generation, 2022.

Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In ICCV , 2023.

Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. arXiv preprint arXiv:2305.10400 , 2023.

OpenAI. Chatgpt. https://openai.com/research/chatgpt , 2023.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. In NIPS , 2014.

Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating Images from Captions with Attention. In ICLR , 2016.

Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. Learning What and Where to Draw. In NIPS , 2016.

Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICML , 2016.

Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, and Aniruddha Kembhavi. X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers. In EMNLP , 2020.

Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML , 2015.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In NeurIPS , pages 1-25, 2020.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with TextGuided Diffusion Models. In ICML , 2022.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. Transactions on Machine Learning Research , 2022.

Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-To-Image Generation via Masked Generative Transformers, 2023.

Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, and Geoffrey Hinton. Pix2seq: A Language Modeling Framework for Object Detection. In ICLR , 2022.

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling. In ECCV , 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, 2023.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR , pages 1-26, 2022.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023.

René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 12159-12168, 2021.

Jaided AI. Easyocr: Ready-to-use ocr with 80+ supported languages and all popular writing scripts including latin, chinese, arabic, devanagari, cyrillic and etc., 2023.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pretraining with Frozen Image Encoders and Large Language Models, 2023.

Saehoon Kim, Sanghun Cho, Chiheon Kim, Doyup Lee, and Woonhyuk Baek. mindall-e on conceptual captions. https://github.com/kakaobrain/minDALL-E , 2021.

Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector Quantized Diffusion Model for Text-to-Image Synthesis. In CVPR , 2022.

Donghoon Lee, Jiseob Kim, Jisu Choi, Jongmin Kim, Minwoo Byeon, Woonhyuk Baek, and Saehoon Kim. Karlo-v1.0.alpha on coyo-100m and cc15m. https://github.com/kakaobrain/karlo , 2022.

Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phuc Lê Khac, Luke Melas, and Ritobrata Ghosh. Dall·e mini, 7 2021.

Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement , 20(1):37-46, 1960.

Klaus Krippendorff. Content Analysis: An Introduction to Its Methodology . Sage Publications, 4th edition, 2018.

Mary L McHugh. Interrater reliability: the kappa statistic. Biochem. Med. (Zagreb) , 22(3):276-282, 2012.

Julius Sim and Chris C Wright. The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy , 85(3):257-268, 03 2005.

Kishore Papineni, Salim Roukos, Todd Ward, and Wj Wei-jing Zhu. BLEU: a Method for Automatic Evaluation of Machine Translation. In ACL , 2002.

Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In ACL Workshop , 2004.

Satanjeev Banerjee and Alon Lavie. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In ACL Workshop , 2005.

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: Semantic Propositional Image Caption Evaluation. In ECCV , 2016.

Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/ huggingface/diffusers , 2022.

Appendix

In this appendix, we provide qualitative examples (Appendix A), additional VPGEN analysis (Appendix B), VPGEN error analysis (Appendix C), VPEVAL error analysis (Appendix D), VPGEN/VPEVAL implementation details (Appendix E), TIFA prompt element analysis details (Appendix F), limitations and broader impacts (Appendix G), and license information (Appendix H).

A Qualitative Examples

In Fig. 8 and Fig. 9, we provide example images generated by our VPGEN and Stable Diffusion v1.4 (the closest baseline of VPGEN) with various skill-based and open-ended prompts, respectively. In Fig. 10, we show VPGEN generation examples of placing objects that are unseen during the text-to-layout finetuning of Vicuna 13B.

Stable Diffusion v1.4

Object

'zebra'

Count

'3 boats'

Spa�al

'a truck is behind a

motorcycle'

Scale

'a remote that is

bigger than a cat'

Text

'a sign that

reads 'parking' '

Stable Diffusion v1.4

Object

'a picture of

a bus'

Count

'4 cows'

Spa�al

'a zebra is to

the le� of a bicycle'

Scale

'a truck that is

bigger than a bicycle'

Text

'a poster that

says 'trespassing' '

PAKKINNG

PANKEP

PAVKGKNG

PRING

LeEBq;

btelote

TRSOSSS

BCESSEE

ASSS

B Additional VPGEN Analysis

In the following, we provide additional analysis of VPGEN, including counting prompts, bounding box quantization, using GPT-3.5-Turbo as layout generation LM, and visual quality metric (FID).

Counting prompts: More than 4 objects & Unspecified number of objects. In addition to the Count skill results with counts of 1 to 4 in the main paper Fig. 6, we provide additional results of VPGEN with counts of up to 7 in Table 5 and show generation examples in Fig. 11 (a) - (d). As expected, the trend continues from Fig. 6, where higher counts are more challenging. We also experiment with prompts without specific numbers. We find that VPGEN can successfully generate layouts/images even when the prompts do not have explicit counts. As shown in Fig. 11 (e), our VPGEN generates two Pikachus from the prompt 'Pikachus on the table'.

Stable Diffusion v1.4

'Black and white photograph

of a bed with a laptop on it.'

'A car playing soccer,

digital art.'

'a photo of bear and bus;

bus is above bear'

'the skyline of

New York City'

COCO

DrawBench

PaintSkills

Par�Prompt

(a) Three pikachus are on a table

L°

Bounding box quantization. As mentioned in the main paper, we use normalized and quantized coordinates with 100 bins to represent bounding boxes, following LM-based object detection works [49; 50]. In Table 6, we compare 100 and 1000 bins for the bounding boxes. Increasing the granularity of the bounding box to 1000 does not increase the accuracy in both skill-based and open-ended prompts, suggesting that the current 100-bin quantization does not hurt the accuracy of object placements.

GPT-3.5-Turbo v.s. Vicuna 13B. In our initial experiment, we tested GPT-3.5-Turbo to generate spatial layouts by showing in-context examples and found that the generated layouts were often inaccurate or not meaningful. For example, GPT-3.5-Turbo often generates a list of bounding boxes that is the same size as the entire image ( e . g ., [object 1 (0,0,99,99), object 2 (0,0,99,99) object3 (0,0,99,99)] ). We conjecture that this is because their training corpus might not include many bounding boxes. Then, we collected text-layout annotations and trained a language model to generate spatial layouts from text prompts.

For a quantitative comparison, we implement VPGEN with GPT-3.5-turbo with 36 incontext examples that cover different skills and compare it with Vicuna 13B trained on Flickr30k+COCO+PaintSkills. As shown in Table 7, Vicuna 13B based VPGEN shows higher skill-based and open-ended VPEval accuracies than GPT-3.5-Turbo based VPGEN.

Visual quality metric (FID). In Table 8, we compare our VPGEN (Vicuna13B + GLIGEN) to its backbone Stable Diffusion v1.4, in FID [30] (30K images of COCO val 2014 split) and VPEval Accuracy. Both VPGEN checkpoints show better skill-based accuracy than Stable Diffusion v1.4 while achieving comparable open-ended accuracy. and FID. In FID (lower the better), we find VPGEN (Flickr30k) < Stable Diffusion v1.4 < VPGEN (Flickr30k+COCO+Paintskills). We think that a bit of increase (but still reasonably good) in the FID of the Flickr30k+COCO+Paintskills

Count >= 4

(a) 4 dogs

(b) 5 dogs

(c) 6 dogs

(d) 7 dogs

nn

Without exact count in prompt

(e) Pikachus on the table

checkpoint is because the layouts of PaintSkills are different from those of natural scenes (COCO and Flickr30k).

Training with real v.s. synthetic images. Our layout generation module (Vicuna 13B) in VPGEN is trained on a mix of both 3D simulator-based synthetic (PaintSkills) and real-world images (MSCOCO/Flickr30K). We conduct an ablation study of training only with real v.s. synthetic images. We train the Vicuna on only PaintSkills and only Flickr30K and compare their results on both skillbased and open-ended prompts in Table 9. For skill-based prompts, PaintSkills training shows slightly higher average accuracy (46.0) than Flickr30K training (43.9). For open-ended prompts, PaintSkills training gets a VPEval score of (65.6), lower than Flickr30K training (71.0). This indicates that training on synthetic data can inject knowledge of specific skills such as counting and spatial relation understanding, but it is hard to cover diverse objects and attributes only with synthetic data.

C VPGEN Error Analysis

Layout v.s. Image generation on skill-based prompts. We analyze the performance of each step in the VPGEN pipeline to determine how much error is propagated with the following four metrics.

Object Recall: The ratio of whether correct objects are included in the generated layouts.

Object Count: The ratio of the correct number of objects is included in the generated layouts.

Layout Accuracy: VPEval accuracy of the generated layouts (using the generated layouts as the object detection results).

Image Accuracy: VPEval accuracy of the final image by GLIGEN (in the main Table 1).

As shown in Table 10, the layout accuracy is much higher than image accuracy, especially for Count/Spatial skills. This indicates that the major error source is the image rendering step (with GLIGEN), and using a more accurate layout-to-image generation model in the future would improve the accuracy of our VPGEN pipeline.

Human evaluation on open-ended prompts. As mentioned in the main paper Sec. 5.3, we ask two expert human annotators to evaluate 50 randomly sampled prompts from TIFA160 [36], with the following two metrics: (1) layout accuracy: if the generated layouts align with the text prompts; (2) image accuracy: if the generated images align with the text prompts. The human annotators show high inter-annotator agreements for both layout accuracy (Cohen's κ = 0 . 73 , Krippendorff's α = 0 . 73 ) and image accuracy (Cohen's κ = 0 . 87 , Krippendorff's α = 0 . 87 ). The Vicuna 13B's layout accuracy is (92%) higher than GLIGEN's image correctness (65%). The result is consistent with the skill-based prompts error analysis result (Table 10) and suggests that a better layout-to-image generation model could further improve our VPGEN framework.

Common error categories. We show some common categories of errors from the text-to-layout generation step (with Vicuna 13B) and from the layout-to-image generation step (with GLIGEN), respectively. As shown in Fig. 12, the Vicuna 13B sometimes (a) generates objects that are not specified in prompts ( e . g ., 'chair' in the 1st example) or (b) misses objects that are mentioned in prompts ( e . g ., 'building' in the 5th example). As shown in Fig. 13, the GLIGEN sometimes (a) generates wrong objects ( e . g ., 'suitcase', instead of 'bench', in the 1st example), (b) misses some objects ( e . g ., 'umbrella' in the 3rd example), or (c) misses object details ( e . g ., 'closed oven' instead of 'oven with the door open' in the 5th example).

Challenging, non-canonical prompts. In Fig. 14, we show generation examples with challenging, non-canonical prompts (a) 'A realistic photo of a Pomeranian dressed up like a 1980s professional wrestler with neon green and neon orange face paint and bright green wrestling tights with bright orange boots' (from DrawBench [24]) and (b) 'a circle with 3 cm radius and a circle with 6 cm radius'. Our VPGEN understands the important parts of the prompts ( e . g ., generating 'realistic', 'Pomeranian dog', 'bright orange boots' and generating layouts of two circles in different sizes), but misses some aspects ( e . g ., the bigger circle is not twice the size of the smaller circle). This is probably because the training prompts (COCO, PaintSkills, Flickr30k Entities) do not include many prompts written in these styles. We believe that scaling datasets with various sources can further improve VPGEN.

D VPEVAL Error Analysis

Evaluation program analysis: coverage and accuracy. For the evaluation program analysis (main paper Sec. 5.4), we experiment with two expert annotators using 50 randomly sampled TIFA160. We ask two expert annotators to analyze the evaluation programs in two criteria: (1) how well the evaluation program covers the content from the text prompt; (2) how accurately each evaluation module evaluates the images. To make it easier for the annotators, for (2) module accuracy analysis, we only ask them to check the output of three randomly sampled evaluation modules. We find that, on average, our evaluation programs cover 94% of the prompt elements, and our modules are accurate 83% of the time. This shows that our VPEVAL evaluation programs are effective in selecting what to evaluate and are highly accurate while also providing interpretability and explainability. In Fig. 15, we visualize our evaluation programs that cover different parts of the prompts and evaluation modules that provide accurate outputs and visual+textual explanations.

E VPGEN/VPEVAL Implementation Details

E.1 VPGEN Details

Training layout-aware LM. To obtain layout-aware LM, we use Vicuna 13B [16], a public state-ofthe-art chatbot language model finetuned from LLaMA [51]. We use parameter-efficient finetuning with LoRA [52] to preserve the original knowledge of LM and save memory during training and inference. We collect the text-layout pair annotations from training sets of three public datasets: Flickr30K entities [17], MS COCO instances 2014 [18], and PaintSkills [19], totaling 1.2M examples. We use randomly selected 2,000 examples for the validation set and use the rest for training. For the object/count generation step, we set the maximum counts for a single object class as 7. We train

Vicuna 13B with per-gpu batch size 96 (= 24 batch x 4 gradient accumulation). When training Vicuna 13B with Flickr30K+COCO+PaintSkills dataset, we train the model for 2 epochs. When training only with Flickr30K dataset, we train the model for 6 epochs, to rough match the training time. Training takes 26 hours with 4 A6000 GPUs (each 48GB).

GLIGEN inference details. We use Huggingface Diffusers [69] implementation of GLIGEN [20] 2 . Following the default configuration, we use gligen_scheduled_sampling_beta = 0 . 3 , num_inference_steps = 50 , and fp16 precision during inference.

Inference time. On a single A6000 GPU, VPGEN takes around 6s (2s for Vicuna 13B and 4s for GLIGEN) to generate an image.

E.2 VPEVAL Details

VPEVAL prompts and evaluation programs. Our skill-based benchmark has an almost uniform distribution of objects and relations among the text prompts. This ensures that generation models cannot achieve high scores by performing highly on a few common objects, relations, and counts, etc . We create skill-specific prompts by composing templates with lists of objects (from 80 COCO objects), counts (1-4), relations (2D: above, below, left, right; 3D: front, behind), scale (smaller, bigger, same), and text (31 unique words). Our evaluation can be extended to any number of objects/counts/relations/ etc . In total, our skill-based benchmark has 400/1000/1000/1000/403 prompts for the object/count/spatial/size/text skills, respectively. In Table 11, we show templates and evaluation programs used for each skill.

Captioning/VQA/CLIP baseline details. For captioning, we generate a caption with BLIP-2 [56] and calculate text similarity metrics between the caption and the original text prompt. For VQA, we give the BLIP-2 (Flan-T5 XL) the image, a question, and a yes/no answer choice, with a prompt template 'Question: {question} Choices: yes, no Answer:' . The {question} is a formatted version of the text prompt ( e . g ., 'a photo of a dog' → 'is there a dog in the image?'). For the spatial and scale skill, we ask three questions: if object A is present ( e . g ., 'is there a dog in the image?') if object B is present ( e . g ., 'is there a cat in the image?'), and if the relationship between them is true ( e . g ., 'is the cat behind the dog?'). We mark each image as correct if the VQA model outputs 'yes' for all three questions. The first two questions prevent accidental false positives if an

(a) Generating unspecified objects

(b) Missing some objects

a cat jumping in the air to

get onto a table

A vehicle composed of two wheels

held in a frame one behind the other,

propelled by pedals and steered

with handlebars attached to the

front wheel.

A grey motorcycle on dirt

road next to a building.

A traffic light and a

signpost at a crossroads

intersection near a

waterway.

a library filled with kids

reading books

Error in Layout Generation (Vicuna 13B)

traffic lighe

(a) Wrong objects

(b) Missing objects

A photo of a suitcase and

bench; bench is left to

suitcase

A car and a truck

sitting a red light

A man standing by an

oven with the door open

An umbrella on top of

a spoon

An assortment of fruits

and vegetable all set

together

Error in Layout-to-Image Generation (GLIGEN)

(c) Wrong object details

coghtat

object is missing. For CLIP (ViT-B/32) [32], we encode the generated image and text prompts, and take the cosine similarity between the visual and text features as the final score.

Inference time. On a single A6000 GPU, each VPEVAL module typically takes less than 1s to run.

Program generation API cost. Using GPT-3.5 Turbo 3 as the program generator, VPEVAL takes about $0.007 (less than 1 cent) to generate programs from a prompt (4K input tokens, 0.5K output tokens).

F TIFA Prompt Element Analysis Details

To better understand the characteristics of the TIFA prompts [36] that we use in open-ended evaluation, we randomly sample 20 prompts and label whether each prompt has any elements that require an understanding of spatial layouts, as mentioned in the main paper Sec. 5.3. Each prompt has multiple elements about objects, attributes, spatial layouts, etc . For example, 'a zebra to the right of a fire hydrant.' has the elements: 'zebra', 'right', and 'fire hydrant'. From the 20 prompts, there are 81 total elements, and only 11/81 (13.6%) are relevant to object layouts ( e . g ., 'right'). The remaining 70 elements are all related to objects or attributes ( e . g ., 'a white cat with black ears and markings' has the elements 'cat', 'white cat', 'ears', 'black ears', 'markings', and 'black markings').

(b) a circle with 3 cm radius

and a circle with 6 cm radius

(a) A realistic photo of a Pomeranian dressed up like a 1980s professional wrestler with neon green and neon orange face paint and bright green wrestling tights with bright orange boots

objectEval(img, 'zebra');

objectEval(img, 'fire hydrant');

vqa(img, 'what animal is to the

right of the fire hydrant?', ....);

...

Evalua�on Program

a zebra

to the right

of a fire hydrant.

a zebra to the right

of a fire hydrant.

a zebra

to the right

of a fire hydrant.

a zebra

to the right

of a fire hydrant.

...

Correct

Incorrect

Incorrect

no

'zebra'

object found.

'fire hydrant' object found.

no

'zebra'

object found.

Q: what animal is to the right of the fire hydrant?

A: not zebra

Correct

Correct

Incorrect

Incorrect

'cat'

object found.

Q: what color is the cat?

A: white

Q: does the cat have black ears?

A: no

Q: does the cat have black markings?

A: no

Incorrect

Evalua�on Results & Visual + Textual Explana�on

...

...

a white

cat

with black

ears and markings

a

white cat with black

ears and markings

a white cat with black

ears

and markings

a white cat with black

ears and markings

objectEval(img, 'cat');

vqa(img, 'what color is the cat?', ...);

vqa(img, 'does the cat

have black ears?', ....);

vqa(img, 'does the cat

have black markings?', ....);

G Limitation and Broader Impacts

Limitations. Since our LMs were primarily trained with English-heavy datasets, they might not well understand prompts written in non-English languages. Likewise, as our generation/evaluation modules were primarily trained with natural images, they might not well handle images in different domains ( e . g ., fine-arts, medical images, etc ). Note that our framework supports easy extensions so that users can update existing modules or add new ones with better domain-specific knowledge.

Generating evaluation programs using LLMs can be costly both in terms of computational resources and price (API cost). Thus, we release the evaluation programs so that users do not have to re-generate the evaluation programs. In addition, we will release a public LM (finetuned for evaluation program generation using ChatGPT outputs) that can run on local machines.

Broader Impacts. While our interpretable VPGEN framework can be beneficial to many applications ( e . g ., user-controlled image generation/manipulation and data augmentation), it could also be used for potentially harmful cases ( e . g ., creating false information or misleading images), like other image generation frameworks.

H License

We will make our code and models publicly accessible. We use standard licenses from the community and provide the following links to the licenses for the datasets, codes, and models that we used in this paper. For further information, please refer to the links.

Apache License 2.0

GLIGEN:

MIT

Grounding DINO:

Apache License 2.0

DPT:

MIT

LAVIS (BLIP-2):

BSD 3-Clause

TIFA:

Apache License 2.0

minDALL-E:

Apache License 2.0, CC-BY-NC-SA 4.0

DALL-E mini (DALL-E Mega):

Apache License 2.0, MIT

EasyOCR:

Apache License 2.0

OpenAI API (ChatGPT):

MIT

Diffusers:

Apache License 2.0

Transformers:

Apache License 2.0

PyTorch:

BSD-style

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper makes two contributions. VPGen is a T2I generation framework that first generates object/count, then layout, and finally image. VPEval is a T2I evaluation pipeline that provides a more comprehensive analysis correlated to human. Authors demonstrate that the step-by-step VPGen approach generates images more aligned with the input text compared to stable diffusion and other baselines. This is evaluated using the newly proposed VPEval scores.\n\nSTRENGTHS:\nThe proposed VPGen decomposition demonstrates better controllability for image generation, especially in terms of following the exact object count and spatial relation from text prompts. The new VPEval score offers a more comprehensive evaluation beyond just image quality. I believe this score will be useful for evaluating controllable generation.\n\nWEAKNESSES:\nResult section lacks the standard evaluation metric for image generation such as the FID and IS score. While the focus of this paper is on better controllability, and the proposed VPEval is clearly a better choice, I still like to see some benchmark on the image quality and diversity. \n\nFirst step of VPGen seems limited. In the examples provided, text has to include the exact number for each object. So is the first step just doing text parsing? Will it still work if I give it a sentence like “a small cozy office room”?\n\nAlso the motivation for training a LM is not clear to me. Many existing works can generate the layout bounding boxes directly, what is the extra benefit of training a LM? Can you do interactive editing?\n\nQUESTIONS:\n(1) It will be great if authors can also provide the standard evaluation benchmarks.\n\n(2) Can authors explain their design choice for using LM in VPGen? \n\n(3) More visualizations results will be appreciated.\n\nLIMITATIONS:\nYes, authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposes a visual-program-based evaluation method VPEval to evaluate text-to-image models. Their method relies on LLM which can call different expert models in different tasks like object detection, OCR, spatial understanding, etc. to evaluate the consistency between the text and the generated image. The human evaluation shows that compared to traditional metrics, their evaluation method aligns with human better. Besides, they introduce a novel interpretable step-by-step text-to-image generation framework VPGen to improve the consistency between generated images and texts in count, spatial, and scale skills.

STRENGTHS:
1.	The paper writing is clear
2.	Their framework VPGen is simple and novel. I like the idea to first train a LLM that generates image high-level layouts from the given prompt, then generates the image based on the layout.
3.	VPGen is effective with improved consistency in counting/spatial/scale relationships. 
4.	The evaluation framework VPEval based on visual program is also novel and has better interpretability compared to previous methods. Besides, it aligns with human judgement better.

WEAKNESSES:
1.	This paper is highly related and based on visual programing. Although the author cites the related papers like Visual Programming [11] and ViperGPT [12] in the related work section L110, a brief introduction of visual programming and a discussion of the difference between [11,12] and this paper are missing.

QUESTIONS:
1.	As the evaluation is based on ChatGPT, it would be good to include a API cost analysis.

LIMITATIONS:
The authors listed and discussed the limitations including the suboptimal performance with non-English prompts and unnatural images. Potential broader impacts like generating harmful images are also discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends previous work in the vision and language space that use visual programs as an intermediate step, to the problem of text to image generation and subsequently, its evaluation. It proposes VPGen, a neuro-symbolic method that is composed of specific modules that count objects, generates layouts using a LLM and then generates an image using GLIGEN, a layout-to-image model. This allows for interpretable inspection of the intermediate steps as well as allows for more controllability in the image generation process. The paper also proposes VPEval, a method to evaluate T2I models using specific modules that also generate interpretable programs inspecting specific skills such as counting, existence of object, spatial, scale and text rendering. The proposed method (VPGen) outperforms previous approaches on skills such as count, spatial and scale, demonstrating that the approach allows for better fine-grained control over the image generation process.\n\nSTRENGTHS:\n## Quality, Originality and Significance\n* There have been several methods that were proposed over the last year for text to image generation, such as stable diffusion, Imagen, Parti, etc but the community still lacked an adequate method for evaluating the quality of the generated image in a way that is also interpretable. This paper provides both, an interpretable text to image generation pipeline that either outperforms or is competitive with existing (open source) state of the art text to image generation models, as well as a way to evaluate them.\n* Improvements in layout generation by using an LM, allowing generalization to unseen concepts instead of depending on pre-defined class set broadens the scope of the type of prompts that can be incorporated into this interpretable pipeline. \n* Human judgements and correlation analysis of existing evaluation methods compared with the proposed method demonstrates that VPEval is a suitable metric for T2I generation both in terms of interpretability while also being correlated with human judgement. \n\n## Clarity\n* The paper is very well written, and is very easy to follow. \n* The paper together with the appendix provides sufficient information on implementation details, qualitative examples and analysis of programs and prompts.\n\nWEAKNESSES:\n* Missing error analysis or categorization on the types of examples that VPGen fails to produce accurate images (both when the layout generator fails, as well as when GLIGEN fails would be useful).\n\nQUESTIONS:\n* Does the domain of the image-text data used for fine-tuning the model in VPGen matter at all (for example if it were synthetic images vs real world images)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a text-to-image (T2I) generation approach along with a new evaluation framework. This can be summarized in Figure 1. First, VPGen (Sec. 3) breaks T2I down into 3 steps, with an LM to generate a “program” of objects and layouts that is then fed into the final generation module. Second, VPEval (Sec. 4) is an interpretable evaluation for T2I that invokes diverse visual modules to evaluate the generated image and the prompt’s components.\n\nSTRENGTHS:\nS1: Both VPGen and VPEval are sound. It leverages existing tools to make the T2I process more controllable (via layout) and interpretable.\n\nS2: The paper is well-written.\n\nS3: Solid experiments. The paper compares various image generation models (Table 1) and provides results based on particular skills (skill-based prompts) and their combination (open-ended prompts). Further, the paper shows a good correlation of VPEval to human judgments. Though for this point, I do have a concern since the VPEval is “gamed” to showcase VPGen (see Weaknesses).\n\nWEAKNESSES:\nW1: It is unclear how generalizable/robust the proposed T2I approach and evaluation protocol are to “non-canonical” prompts shown in Figure 1. For VPGen, what if the prompts are really complex with multiple objects that interact with one another? What if the prompts do not include the numbers of objects (e.g., Pikachus on the table)? Similarly, since VPEval is geared toward showcasing VPGen, some aspects of evaluation are missing such as object attributes (L206) (e.g., a purple Pikachu and a pink Pikachu).\n\nW2: The novelty of both VPGen and VPEval should be highlighted further. Perhaps expand L99-105 in more detail and demonstrate L101-102 empirically. Furthermore, it would also be appreciated how the VPEval is compared to VQA-based approaches (L35) like TIFA and SeeTRUE (https://arxiv.org/abs/2305.10400), which in my opinion are also interpretable.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delves into the intersection of large language models (LLMs) and their applications in vision-and-language tasks, specifically focusing on text-to-image (T2I) generation. The authors identify a gap in the existing literature: no thorough analysis has been conducted on the synergy of LLMs and various visual modules for \"complex\" T2I generation tasks.\n\nTo address this they propose two novel frameworks: VPGEN: This framework offers a step-by-step approach to T2I generation, segmenting the process into three distinct stages: object/count generation, layout generation, and image generation. They use Vicuna to fine-tune to manage the first two stages, and the results demonstrate improved control over layout creation. The final stage, image generation, incorporates existing models like GLIGEN. Notably, VPGEN’s design capitalizes on the inherent knowledge of pre-trained LLMs, granting it the ability to recognize objects that have not been predefined—surpassing the capabilities of older layout-guided T2I techniques.\nVPEVAL: This authors propose this for evaluation of the generated images. Unlike conventional T2I evaluation methods which primarily gauge visual quality and image-text alignment through a singular visual module, VPEVAL emphasizes interpretability and a multiple modules to evaluate the generated image. In essence, it employs evaluation programs that activate a variety of visual modules, to distinct T2I skills.\nThe papers findings indicate that VPGEN+GLIGEN combination showcases relatively better performance, especially when precision in layouts and spatial relationships is paramount. The VPEVAL evaluation method seems to aligns with human assessment.\nIn summary the contributions are:\nThe introduction of VPGEN, an interpretable T2I generation framework that dissects the T2I process into three modules.\nThe proposal of VPEVAL, an evaluation framework for T2I tasks that enhances the explainability and thorough analysis by invoking diverse visual modules.\nA detailed analysis of various T2I models, highlighting the superior layout control of VPGEN and the human-centric alignment of VPEVAL.\n\nSTRENGTHS:\nOverall its a great work and the paper addresses some fundamental limitations in a unique way.\nThe paper addresses a very important problem in image generation realm. Most of the existing image generation models including the ones trained on massive datasets, are not good at generating images with spatial consistency, generating the correct number of objects , or understanding the size of the objects. This bias has been highlighted by various papers since 2018 and is still existing in modern generative models. The paper proposes a solution to the same by combining LLMs fine tuned on text to layout pairs and layout to image generation model. This fine tuning provides the LLMs with the ability to understand the spatial etc relationships between the generated objects. Thus removing a limitation from the prior results.\nThe proposed VPEval combined several modules to evaluate the generated quality of the images and also interpretable due to the generated programs. Using such visual programming is novel.\nAdaptability - as the modules in image generation and image evaluation improve the proposed approach would also improve in performance.\nOffering visual+textual explanations is a strength as it increases the interpretability of the evaluation.\n\nWEAKNESSES:\nThe proposed 5 evaluation modules might still not capture the generated image quality and other complex semantics. Other image quality could be incorporated and evaluated.\n\nThe paper emphasizes heavily on the two step generation - object count and layout generation , but fails to conduct any ablation studies to prove the effectiveness of this pipeline. Since the process is decomposed into different steps, the interplay and coherence between these steps are vital. Errors or inconsistencies in earlier steps (like object/count generation) could cascade and affect the final image's quality.\n\nRelying on bounding boxes to represent layout may be simplistic. The boxes encode basic spatial relations for sure but might miss more detailed pose, occlusion, and depth information.\n\nAssociating object names in texts to layouts could be unreliable for ambiguous or synonymous words. The model may lack grounding to map words to visual concepts.\nFor example, consider the word \"bat\" - this could refer to: A baseball bat A flying bat animal\nOr the word \"apple\" could refer to: The fruit apple The technology company Apple\nWithout proper grounding between language and vision, the model may struggle to determine which visual concept is being referred to based on just the word alone.\nSome detailed examples to illustrate this:\nThe text \"a man holding a bat\" is ambiguous - is this referring to a baseball bat or a flying animal bat? The model may wrongly depict it without the proper grounding. The text \"a logo of an apple\" could wrongly depict the fruit when the technology company was meant. Synonyms like \"couch\" vs \"sofa\" would need to be mapped to the same visual concept. So in summary, relying purely on language without grounding it properly to visual concepts can lead to ambiguity in mapping words to the intended visual representations. Providing more context and grounding is important to resolve this.\n\nGranularity of Layout Representation: VPGEN decomposes bounding box coordinates into a [0,1] scale and quantizes them into 100 bins. Such discretization can potentially lead to a loss of finer details in bounding box representation, which might impact the accuracy of object placement in the generated images.\n\n2 human evaluators are slightly less to conclude that the results align with human and also the setting of human evaluation is a bit unclear. Like humans evaluate it subjectively (esp. with open ended tasks) . So could the authors shed some light on this setting? Also given that the correlation doesnt seem very high for objects ( 63.7?)\n\nMinor comments:\nThe word skill has been mentioned earlier without much context which might be ambiguous to the readers ( so just a minot presentation suggestion) .\nExplaining what the paper means by \"challenging\" T2I tasks might be helpful in terms of better clarity and presentation.\nScaling the approach to generate complex high-resolution images with many objects and intricate relations may be difficult. The text-to-layout-to-image pipeline has limitations. Any further results to prove or disprove this might be helpful.\n\nQUESTIONS:\n1. How would the authors compare their approach with Visual chatgpt ( https://arxiv.org/pdf/2303.04671.pdf) . This seem to have a image generation module? \n\n2. Does quantizing the bounding box coordinates into discrete bins loses precision? does this lead to misalignment between the predicted layout and actual image content.\n\n3. Unseen Objects: The ability to generate layouts of objects not seen during training (e.g., 'pikachu') can be an advantage, but it also raises questions about the model's ability to accurately represent unfamiliar objects in space. How well can it handle a completely novel entity in terms of spatial characteristics, especially when combined with other objects?\n\n4. How many in-context examples are used in the for the VPEval task? \n\n5. Can the authors provide results for count greater than 4 ( fig 5) . and can the model generate images like \" generate a circle with 3 cm radius and generate a circle with 6 cm radius , essentially saying one smaller than the other ? \n\n6. The model has a limited encoding capacity to represent all possible object combinations and numbers. So  to understand if an unusual or out-of-distribution combinations at test time could confuse the layout prediction stage, can the authors provide some complex compositional image generation results? \n\n7. what is the inference time?\n\nLIMITATIONS:\nThe authors have spoken about the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}