SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jaemin Cho Abhay Zala Mohit Bansal
{jmincho, aszala, mbansal}@cs.unc.edu
https://vp-t2i.github.io
As large language models have demonstrated impressive performance in many domains, recent works have adopted language models (LMs) as controllers of visual modules for vision-and-language tasks. While existing work focuses on equipping LMs with visual understanding, we propose two novel interpretable/explainable visual programming frameworks for text-to-image (T2I) generation and evaluation. First, we introduce VPGEN, an interpretable step-by-step T2I generation framework that decomposes T2I generation into three steps: object/count generation, layout generation, and image generation. We employ an LM to handle the first two steps (object/count generation and layout generation), by finetuning it on text-Layout pairs. Our step-by-step T2I generation framework provides stronger spatial control than end-to-end models, the dominant approach for this task. Furthermore, we leverage the world knowledge of pretrained LMs, overcoming the limitation of previous layout-guided T2I works that can only handle predefined object classes. We demonstrate that our VPGEN has improved control in counts/spatial relations/scales of objects than state-of-the-art T2I generation models. Second, we introduce VPEVAL, an interpretable and explainable evaluation framework for T2I generation based on visual programming. Unlike previous T2I evaluations with a single scoring model that is accurate in some skills but unreliable in others, VPEVAL produces evaluation programs that invoke a set of visual modules that are experts in different skills, and also provides visual+textual explanations of the evaluation results. Our analysis shows that VPEVAL provides a more human-correlated evaluation for skill-specific and open-ended prompts than widely used single model-based evaluation. We hope that our work encourages future progress on interpretable/explainable generation and evaluation for T2I models.
Large language models (LLMs) have shown remarkable performance on many natural language processing tasks, such as question answering, summarization, and story generation [1; 2; 3; 4; 5; 6; 7; 8]. Recent works have shown that LLMs can also tackle certain vision-and-language tasks such as visual question answering and visual grounding, by generating visual programs that can control external visual modules and combine their outputs to get the final response [9; 10; 11; 12; 13; 14; 15]. However, no prior works have combined LLMs and different visual modules for the challenging text-to-image (T2I) generation task. Our work proposes two novel interpretable/explainable visual programming (VP) frameworks combining LLMs and visual modules for T2I generation and evaluation.
First, we introduce VPGEN (Sec. 3), a new step-by-step T2I generation framework that decomposes the T2I task into three steps (object/count generation, layout generation, and image generation), where each step is implemented as a module and executed in an interpretable generation program, as shown
Figure 1: Illustration of the proposed visual programming frameworks for text-to-image (T2I) generation and evaluation. In (a) VPGEN, we first generate a list of objects, then object positions, and finally an image, by executing three modules step-by-step. In (b) VPEVAL, we use evaluation programs with a mixture of evaluation modules that handle different skills and provide visual+textual explanation of evaluation results.
in Fig. 1 (a). We employ Vicuna [16], a powerful publicly available LLM, to handle the first two steps (object/count generation and layout generation), by finetuning it on text-Layout pairs from multiple datasets [17; 18; 19], resulting in improved layout control for T2I generation. For the last image generation step, we use off-the-shelf layout-to-image generation models, e.g., GLIGEN [20]. Our generation framework provides more interpretable spatial control than the widely used end-to-end T2I models [21; 22; 23; 24]. Furthermore, VPGEN leverages the world knowledge of pretrained LMs to understand unseen objects, overcoming the limitation of previous layout-guided T2I works that can only handle predefined object classes [25; 26; 27; 28]. In our analysis, we find that our VPGEN (Vicuna+GLIGEN) could generate images more accurately following the text description (especially about object counts, spatial relations, and object sizes) than strong T2I generation baselines, such as Stable Diffusion [21] (see Sec. 5.2).
Next, we introduce VPEVAL (Sec. 4), a new interpretable/explainable T2I evaluation framework based on evaluation programs that invoke diverse visual modules to evaluate different T2I skills [19] (see Sec. 4.1), and also provide visual+textual explanations of the evaluation results. Previous T2I evaluation works have focused on measuring visual quality [29; 30] and image-text alignment with a single visual module based on text retrieval [31], CLIP cosine similarity [32; 33], captioning [25], object detection [34; 19; 35], and visual question answering (VQA) [36; 37]. However, we cannot interpret the reasoning behind the scoring; i.e., why CLIP assigns a higher score to an image-text pair than another. In addition, these modules are good at measuring some skills, but not reliable in other skills (e.g., VQA/CLIP models are not good at counting objects or accurately parsing text rendered in images). In VPEVAL, we break the evaluation process into a mixture of visual evaluation modules, resulting in an interpretable evaluation program. The evaluation modules are experts in different skills and provide visual+textual (i.e., multimodal) result/incorrect explanations, as shown in Fig. 1 (b). We evaluate both multimodal LM and diffusion-based T2I models with two types of prompts: (1) skill-based prompts (Sec. 4.3) and (2) open-ended prompts (Sec. 4.4). Skill-based prompts evaluate a single skill per image, whereas open-ended prompts evaluate multiple skills per image. We adapt a large language model (GPT-3.5-Turbo [38]) to dynamically generate an evaluation program for each open-ended prompt, without requiring finetuning on expensive evaluation program annotations.
In our skill-based prompt experiments, while all T2I models faced challenges in the Count, Spatial, Scale, and Text Rendering skills, our VPGEN shows higher scores in the Count, Spatial, and Scale skills, demonstrating its strong layout control. We also show a fine-grained sub-split analysis on where our VPGEN gives an improvement over the baseline T2I models. In our open-ended prompt experiments, our VPGEN achieves competitive results to the T2I baselines. Many of these open-ended prompts tend to focus more on objects and attributes, which, as shown by the skill-based evaluation, T2I models perform quite well in; however, when precise layouts and spatial relationships are needed, our VPGEN framework performs better (Sec. 5.3). In our error analysis, we also find
that GLIGEN sometimes fails to properly generate images even when Vicuna generates the correct layouts, indicating that as better layout-to-image models become available, our VPGEN framework will achieve better results (Sec. 5.3). We also provide an analysis of the generated evaluation programs from VPEVAL, which proved to be highly accurate and comprehensive in covering elements from the prompts (Sec. 5.4). For both types of prompts, our VPEVAL method shows a higher correlation to human evaluation than existing single model-based evaluation methods (Sec. 5.4).
Our contributions can be summarized as follows: (1) VPGEN (Sec. 3), a new step-by-step T2I generation framework that decomposes the T2I task into three steps (object/count generation, layout generation, and image generation), where each step is implemented as a module and executed in an interpretable generation program; (2) VPEVAL (Sec. 4), a new interpretable/explainable T2I evaluation framework based on evaluation programs that execute diverse visual modules to evaluate different T2I skills and provide visual+textual explanations of the evaluation results; (3) comprehensive analysis of different T2I models, which demonstrates the strong layout control of VPGEN and high human correlation of VPEVAL (Sec. 5). We will release T2I model-generated images, VPEVAL programs, and a public LM (finetuned for evaluation program generation using ChatGPT outputs). We hope our research fosters future work on interpretable/explainable generation and evaluation for T2I tasks.
Text-to-image generation models. In the T2I generation task, models generate images from text. Early deep learning models used the Generative Adversarial Networks (GAN) [39] framework for this task [40; 41; 42; 31]. More recently, multimodal language models [43; 22] and diffusion models [44; 45; 21; 46] have gained popularity. Recent advances in multimodal language models such as Parti [47] and MUSE [48], and diffusion models like Stable Diffusion [21], UnCLIP [23], and Imagen [24], have demonstrated a high level of photorealism in zero-shot image generation.
Bridging text-to-image generation with layouts. One line of research decomposes the T2I generation task into two stages: text-to layout generation and layout-to-image generation [25; 26; 27; 28]. However, the previous approaches focus on a set of predefined object classes by training a new layout predictor module from scratch and therefore cannot place new objects unseen during training. In contrast, our VPGEN uses an LM to handle layout generation by generating objects/counts/positions in text, allowing flexible adaptation of pretrained LMs that can understand diverse region descriptions.
Language models with visual modules. Although large language models (LLMs) have shown a broad range of commonsense knowledge, most of them are trained only on text corpus and cannot understand image inputs to tackle vision-and-language (VL) tasks. Thus, recent works explore tackling VL tasks by solving sub-tasks with external visual modules and combining their outputs to obtain the final response [9; 10; 11; 12; 13; 14; 15]. The visual sub-tasks include describing images as text, finding image regions relevant to the text, editing images with text guidance, and obtaining answers from a VQA model. However, existing work focuses on converting visual inputs into text format so that LLMs can understand them. Our work is the first work using visual programming for interpretable and explainable T2I generation and evaluation.
Evaluation of text-to-image generation models. The text-to-image community has commonly used two types of automated evaluation metrics: visual quality and image-text alignment. For visual quality, Inception Score (IS) [29] and Fréchet Inception Distance (FID) [30] have been widely used. For image-text alignment, previous work used a single model to calculate an alignment score for image-text pair, based on text retrieval [31], cosine similarity [33], captioning [25], object detection [34; 19; 35], and visual question answering (VQA) [36; 37]. In this work, we propose the first T2I evaluation framework VPEVAL, based on interpretable and explainable evaluation programs which execute a diverse set of visual modules (e.g., object detection, OCR, depth estimation, object counting). Our VPEVAL evaluation programs provide visual+textual explanations of the evaluation result and demonstrate a high correlation with human judgments.
3 VPGEN: Visual Programming for Step-by-Step Text-to-Image Generation
We propose VPGEN, a novel visual programming framework for interpretable step-by-step text-to-image (T2I) generation. As illustrated in Fig. 2, we decompose the text-to-image generation task into
Figure 2: Interpretable step-by-step text-to-image generation with VPGEN. VPGEN decomposes T2I generation into three steps: (1) object/count generation, (2) layout generation, and (3) image generation, and executes the three modules step-by-step.
three steps: (1) object/count generation, (2) layout generation, and (3) image generation. In contrast to previous T2I generation works that use an intermediate layout prediction module [25; 26; 27; 28], VPGEN represents all layouts (object description, object counts, and bounding boxes) in text, and employs an LM to handle the first two steps: (1) object/count generation and (2) layout generation. This makes it easy to adapt the knowledge of pretrained LMs and enables generating layouts of objects that are unseen during text-to-query training (e.g., 'pikachu'). For layout representation, we choose the bounding box format because of its efficiency; bounding boxes generally require fewer tokens than other formats.<sup>1</sup>
Two-step layout generation with LM. Fig. 2 illustrates how our LM generates layouts in two steps: (1) object/count generation and (2) layout generation. For the first step, we represent the scene by enumerating objects and their counts, such as 'obj1 (# of obj1) obj2 (# of obj2)'. For the second step, following previous LM-based object detection works [49; 50], we normalize 'xyxy' format bounding box coordinates into  $[0,1]$  and quantize them into 100 bins; a single object is represented as 'obj (xmin, ymin, xmax, ymax)'; where each coordinate is within  $\{0, \dots, 99\}$ .
Training layout-aware LM. To obtain the layout-aware LM, we use Vicuna 13B [16], a public state-of-the-art language model finetuned from LLaMA [51]. We use parameter-efficient finetuning with LoRA [52] to preserve the original knowledge of the LM and save memory during training and inference. We collect text-Layout pair annotations from training sets of three public datasets: Flickr30K entities [17], MS COCO instances 2014 [18], and PaintSkills [19], totaling 1.2M examples. See appendix for more training details.
Layout-to-Image Generation. We use a recent layout-to-image generation model GLIGEN [20] for the final step - image generation. The layout-to-image model takes a list of regions (bounding boxes and text descriptions) as well as the original text prompt to generate an image.
4 VPEVAL: Visual Programming for Explainable Evaluation of Text-to-Image Generation
VPEVAL is a novel interpretable/explainable evaluation framework for T2I generation models, based on visual programming. Unlike existing T2I evaluation methods that compute image-text alignment scores with an end-to-end model, our evaluation provides an interpretable program and visual+textual explanations for the evaluation results, as shown in Figs. 3 and 5. We propose two types of evaluation prompts: (1) skill-based evaluation and (2) open-ended evaluation. In skill-based evaluation, we define five image generation skills and use a set of skill-specific prompts and evaluation programs, as illustrated in Fig. 3. In open-ended evaluation, we use a diverse set of prompts that require multiple image generation skills. We adopt a language model to dynamically generate an evaluation program for each text prompt, as shown in Fig. 5. In the following, we describe evaluation skills (Sec. 4.1), visual evaluation modules (Sec. 4.2), skill-based evaluation with visual programs (Sec. 4.3), and open-ended evaluation with visual program generator LM (Sec. 4.4).
<sup>1</sup>For example, a xyxy-format bounding box can be represented with 4 tokens, while a 64x64 segmentation map requires 4096 tokens.
Skill-based Interpretable Evaluation Program Evaluation Results & Visual + Textual Explanation
Prompt: "a photo of a dog"  
Program: objectEval(Img, "dog")
Prompt:"3 dogs"  
Program: countEval (img, "dog",=="=3")
Prompt: "a spoon is in front of a potted plant"  
Program: spatialEval(Img, "spoon, potted plant, front")
Prompt: "a laptop that is bigger than a sports ball"  
Program: scaleeval (1mg, "Laptop, sports ball, bigger")
Prompt: "a poster that reads 'shop' Program: textEval(Img, "shop")
"dog" object found.
(spoon,"potted plant") with "front" (z1 < z2) found.
("laptop", "sport ball") with "bigger" (area1 > area2) found.
"shop" text found
No "dog" object found.
No (obj1, obj2)  
pair of ("spoon", "potted plant")  
with "front" (z1 < z2) found.
No "Sports ball object found.
Figure 3: Illustration of skill-based evaluations in VPEVAL (Sec. 4.3). Left: Given text prompts that require different image generation skills, our interpretable evaluation programs evaluate images by executing relevant visual modules. Right: Our evaluation programs output binary scores and provide visual (bounding boxes of detected objects/text) + textual explanations of the evaluation results.
No "shop" text found.
4.1 Evaluation Skills
Inspired by skill-based T2I analysis of PaintSkills [19], our VPEVAL measures five image generation skills: Object, Count, Spatial, Scale, and Text Rendering. Powered by evaluation programs with expert visual modules, our VPEVAL supports zero-shot evaluation of images (no finetuning of T2I models is required), detecting regions with free-form text prompts, new 3D spatial relations (front, behind), and new scale comparison and text rendering skills, which were not supported in PaintSkills [19]. In Fig. 3, we illustrate the evaluation process for each skill.
Object. Given a prompt with an object (e.g., "a photo of a dog"), a T2I model should generate an image with that object present.
Count. Given a prompt with a certain number of an object (e.g., "3 dogs"), a T2I model should generate an image containing the specified number of objects.
Spatial. Given a prompt with two objects and a spatial relationship between them (e.g., "a spoon is in front of a potted plant"), a T2I model should generate an image that contains both objects with the correct spatial relations.
Scale. Given a prompt with two objects and a relative scale between them (e.g., "a laptop that is bigger than a sports ball"), a T2I model should generate an image that contains both objects, and each object should be of the correct relative scale.
Text Rendering. Given a prompt with a certain text to display (e.g., "a poster that reads 'shop'"), a T2I model should generate an image that properly renders the text.
4.2 Visual Evaluation Modules
To measure the skills described above in Sec. 4.1, we use eight expert visual evaluation modules specialized for different tasks. The modules provide visual+textual explanations with their score. Visual explanations are generated by rendering bounding boxes on the images and textual explanations are generated through text templates, as shown in Fig. 3 and 5. We provide the Python pseudocode implementation for each module in Fig. 4.
Module definitions. objDet detects objects in an image based on a referring expression text and returns them (+ their 2D bounding boxes and depth), using Grounding DINO [53] and DPT [54].OCR detects all text in an image and returns them (+ their 2D bounding boxes), using EasyOCR [55].
Figure 4: Python pseudocode implementation of visual modules used in VPEVAL.
vqa answers a multiple-choice question using BLIP-2 (Flan-T5 XL) [56]. It can handle phrases that cannot be covered only with objDet orOCR (e.g., pose estimation, action recognition, object attributes). objectEval evaluates if an object is in an image using objDet. countEval evaluates if an object occurs in an image a certain number of times using objDet and an equation (e.g., “==3”, “<5”). spatialEval evaluates if two objects have a certain spatial relationship with each other. For six relations (above, below, left, right, front, and behind), it compares bounding box/depth values using objDet. For other relations, it uses vqa scaleEval evaluates if two objects have a certain scale relationship with each other using objDet. For three scale relations (smaller, bigger, and same) it compares bounding box areas by using objDet. For other relations, it uses vqa.textEval evaluates if a given text is present in an image usingOCR.
4.3 Skill-based Evaluation with Visual Programs
For skill-based evaluation, we create text prompts with various skill-specific templates that are used for image generation and evaluation with our programs. See appendix for the details of prompt creation. In Fig. 3, we illustrate our skill-based evaluation in VPEVAL. Given text prompts that require different image generation skills, our evaluation programs measure image-text alignment scores by calling the relevant visual modules. Unlike existing T2I evaluation methods, our evaluation programs provide visual+textual explanations of the evaluation results.
4.4 Open-ended Evaluation with Visual Program Generator LM
Although our evaluation with skill-specific prompts covers five important and diverse image generation skills, user-written prompts can sometimes be even more complex and need multiple evaluation criteria (e.g., a mix of our skills in Sec. 4.1 and other skills like attribute detection). For example, evaluating images generated with the prompt 'A woman dressed for a wedding is showing a watermelon slice to a woman on a scooter.' involves multiple aspects, such as two women (count skill), 'a woman on a scooter' (spatial skill), 'dressed for a wedding' (attribute detection skill), etc. To handle such open-ended prompts, we extend the VPEVAL setup with evaluation programs that can use many visual modules together (whereas single-skill prompts can be evaluated with a program of 1-2 modules). We generate open-ended prompt evaluation programs with an LLM, then the evaluation programs output the average score and the visual+textual explanations from their visual modules. The program generation involves choosing which prompt elements to evaluate and which modules to evaluate those elements (see Fig. 5).
Open-ended prompts. For open-ended evaluation, we use 160 prompts of TIFA v1.0 human judgment dataset [36] (we refer to these prompts as 'TIFA160'). The dataset consists of (1) text prompts from COCO [18], PaintSkills [19], DrawBench [24], and Participments [47], (2) images generated by five baseline models (minDALL-E [57], VQ-Diffusion [58], Stable Diffusion v1.1/v1.5/v2.1 [21]), and (3) human judgment scores on the images (on 1-5 Likert scale).
Generating evaluation programs via in-context learning. As annotation of evaluation programs with open-ended prompts can be expensive, we use ChatGPT (GPT-3.5-Turbo) [38] to generate evaluation programs via in-context learning. To guide ChatGPT, we adapt the 12 in-context prompts from TIFA [36]. We show ChatGPT the list of visual modules and example text prompts and programs, then ask the model to generate a program given a new prompt, as illustrated in Fig. 5. For
Figure 5: Evaluation of open-ended prompts in VPEVAL (Sec. 4.4). Left: We generate evaluation programs with ChatGPT via in-context learning. Right: Our evaluation programs consist of multiple modules evaluating different elements from a text prompt. We output the final evaluation score by averaging the outputs of each evaluation module.
Table 1: VPEVAL scores of T2I generation models on skill-based prompts (see Sec. 5.2 for analysis). F30: Flickr30K Entities, C: COCO, P: PaintSkills.
Model	VPEVAL Skill Score (%) ↑
Object	Count	Spatial	Scale	Text Rendering	Average
Stable Diffusion v1.4	97.3	47.4	22.9	11.9	8.9	37.7
Stable Diffusion v2.1	96.5	53.9	31.3	14.3	6.9	40.6
Karlo	95.0	59.5	24.0	16.4	8.9	40.8
minDALL-E	79.8	29.3	7.0	6.2	0.0	24.4
DALL-E Mega	94.0	45.6	17.0	8.5	0.0	33.0
VPGEN (F30)	96.8	55.0	39.0	23.3	5.2	43.9
VPGEN (F30+C+P)	96.8	72.2	56.1	26.3	3.7	51.0
reproducible and accessible evaluation, we release the evaluation programs so that VPEVAL users do not have to generate the programs themselves. We will also release a public LM (finetuned for evaluation program generation using ChatGPT outputs) that can run on local machines.
In this section, we introduce the baseline T2I models we evaluate (Sec. 5.1), compare different T2I models with skill-based (Sec. 5.2) and open-ended prompts (Sec. 5.3), and show the human-correlation of VPEVAL (Sec. 5.4).
5.1 Evaluated Models
We evaluate our VPGEN (Vicuna+GLIGEN) and five popular and publicly available T2I models, covering both diffusion models (Stable Diffusion v1.4/v2.1 [21] and Karlo [59]), and multimodal autoregressive language models (minDALL-E [57] and DALL-E Mega [60]). Stable Diffusion v1.4 is the most comparable baseline to VPGEN, because the GLIGEN [20] in VPGEN uses frozen Stable Diffusion v1.4 with a few newly inserted adapter parameters for spatial control.
5.2 Evaluation on Skill-based Prompts