SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Diffusion models outperform multimodal LMs. In Table 1, we show the VPEVAL skill accuracies for each model. The diffusion models (Stable Diffusion, Karlo, and our VPGEN) show higher overall accuracy than the multimodal LMs (minDALL-E and DALL-E Mega).
Count/Spatial/Scale/Text Rendering skills are challenging. Overall, the five baseline T2I models achieve high scores (above  $93\%$  except for minDALL-E) in Object skill; i.e., they are good at
generating a high-quality single object. However, the models score low accuracies in the other skills, indicating that these skills (Count, Spatial, Scale, and Text Rendering) are still challenging with recent T2I models.
Step-by-step generation improves challenging skills. The bottom row of Table 1 shows that our VPGEN achieves high accuracy in Object skill and strongly outperforms other baselines in Count (+12.7% than Karlo), Spatial (+24.8% than Stable Diffusion v2.1) and Scale (+9.9% than Karlo) skills. This result demonstrates that our step-by-step generation method is effective in aligning image layout with text prompts, while also still having an interpretable generation program. All models achieve low scores on the Text Rendering skill, including our VPGEN. A potential reason why our VPGEN model does not improve Text Rendering score might be because the text-to-query training datasets of VPGEN (Flickr30K Entities [17], COCO [18], and PaintSkills [19]) contain very few images/captions about text rendering, opening room for future improvements (e.g., finetuning on a dataset with more images that focus on text rendering and including a text rendering module as part of our generation framework).
Fine-grained analysis in Count/Spatial/Scale skills. To better understand the high performance of our VPGEN in Count, Spatial, and Scale skills, we perform a detailed analysis of these skills with fine-grained splits. In Fig. 6, we compare our VPGEN (Vicuna+GLIGEN) and its closest baseline Stable Diffusion v1.4 on the three skills. Overall: VPGEN achieves better performance than Stable Diffusion on every split on all three skills. Count: While both models show difficulties with counting larger numbers, our VPGEN model achieves better performance with  $50+$  accuracy on all four numbers. Spatial: Our VPGEN model performs better on all six spatial relations. VPGEN shows higher accuracy on 2D relations (left/right/below/above) than on 3D depth relations (front/behind), while the Stable Diffusion is better at 3D relations than 2D relations. Scale: VPGEN generates more accurate layouts with two objects of different sizes (bigger/smaller), than layouts with two objects of similar sizes (same).
Figure 6: VPEVAL score comparison of our VPGEN (Vicuna+GLIGEN) and Stable Diffusion v1.4 on fine-grained splits in Count/Spatial/Scale skills.
5.3 Evaluation on Open-ended Prompts
Table 2 shows the VPEVAL score on the open-ended TIFA160 prompts. We calculate the score by averaging accuracy from the modules (see Fig. 5 for an example). The overall score trend of the baseline models (diffusion models  $>$  multimodal LMs) is consistent with the skill-based prompts. Unlike the trend in skill-based evaluation (Sec. 5.2), our VPGEN (Vicuna+GLIGEN) methods achieve similar performance to the Stable Diffusion baseline, while also providing interpretable generation steps. We perform an analysis of the open-ended prompts used for evaluation. In these prompts, object descriptions and attributes are the dominating prompt element (86.4% of elements), whereas descriptions of spatial layouts, only account for 13.6% of elements. See appendix for more
Table 2: VPEVAL scores on open-ended prompts (see Sec. 5.3). GLIGEN in VPGEN has Stable Diffusion v1.4 backbone. F30: Flickr30K Entities, C: COCO, P: PaintSkills.
Model	Score (%) ↑
Stable Diffusion v1.4	70.6
Stable Diffusion v2.1	72.0
Karlo	70.0
minDALL-E	47.5
DALL-E Mega	67.2
VPGEN (F30)	71.0
VPGEN (F30+C+P)	68.3
details. For prompts from PaintSkills [19] where spatial layouts are more important, we find that our VPGEN scores much higher than Stable Diffusion v1.4 (71.0 v.s. 63.5) and Stable Diffusion v2.1 (71.0 v.s. 68.4). Note that GLIGEN used in VPGEN is based on Stable Diffusion v1.4, making it the closest baseline, and hence future versions of GLIGEN based on Stable Diffusion v2.1 or other stronger layout-to-image models will improve our results further. Also, we find that GLIGEN sometimes fails to properly generate images even when VPGEN generates the correct layouts (see following paragraph).
Figure 7: Images generated by Stable Diffusion v1.4 and our VPGEN (Vicuna 13B+GLIGEN) for skill-based (top) and open-ended (bottom) prompts.
VPGEN sources of error: layout/image generation. Since our VPGEN separates layout generation and image generation, we study the errors caused by each step. For this, we manually analyze the images generated from TIFA160 prompts, and label (1) whether the generated layouts align with the text prompts and (2) whether the final images align with the text prompts/sections. GLIGEN sometimes fails to properly generate images even when Vicuna 13B generates the correct layouts, indicating that when better layout-to-image models become available, our VPGEN framework will achieve higher results. We include more details in the appendix.
Qualitative Examples. In Fig. 7, we provide example images generated by our VPGEN and Stable Diffusion v1.4 (the closest baseline) with various skill-based (top) and open-ended (bottom) prompts. See appendix for more qualitative examples with skill-based/open-ended prompts, prompts with unseen objects, counting  $\geq 4$  objects, and error analysis.
5.4 Human Evaluation of VPEVAL
To measure alignments with human judgments and our VPEVAL, we compare the human-correlations of VPEVAL and other metrics on both skill-based prompts and open-ended prompts.
Human correlation of VPEVAL on skill-based prompts. We ask two expert annotators to evaluate 20 images, from each of the five baseline models, for each of the five skills with binary scoring (total  $20 \times 5 \times 5 = 500$  images). The inter-annotator agreements were measured with Cohen's  $\kappa$  [61] = 0.85 and Krippendorff's  $\alpha$  [62] = 0.85, indicating 'near-perfect' ( $\kappa > 0.8$  or  $\alpha > 0.8$ ) agreement [63; 64; 62]. We compare our visual programs with captioning (with metrics BLEU [65], ROUGE [66], METEOR [67], and SPICE [68]), VQA, and CLIP (ViT-B/32) based evaluation. We use BLIP-2 Flan-T5 XL, the state-of-the-art public model for image captioning and VQA.
In Table 3, our VPEVAL shows a higher overall correlation with human judgments (66.6) than single module-based evaluations (CLIP, Captioning, and VQA). Regarding per-skill correlations, VPEVAL shows especially strong human correlations in Count and Text Rendering. As the BLIP-2 VQA module also shows strong correlation on Object/Spatial/Scale, we also experiment with VPEVAL $^{\dagger}$ : using BLIP-2 VQA for objectEval/spatialEval/scaleEval modules (instead of Grounding
Table 3: Human correlation study of skill-based evaluation. We measure Spearman's  $\rho$  correlation between human judgment and different automated metrics on the skill-based prompts (Sec. 4.3). VPEVAL†: using BLIP-2 VQA for objectEval/spatialEval/scaleEval modules.
Eval Metric	Human-metric correlation (Spearman's ρ) ↑
Object	Count	Spatial	Scale	Text	Overall
CLIP Cosine similarity (ViT-B/32)	35.2	38.6	35.4	13.7	40.0	20.4
BLIP-2 Captioning - BLEU	11.9	31.4	26.3	24.0	23.6	-3.4
BLIP-2 Captioning -ROUGE	15.7	26.5	28.0	12.2	28.3	11.9
BLIP-2 Captioning -METEOR	33.7	20.7	40.5	25.1	26.6	29.3
BLIP-2 Captioning -SPICE	56.1	20.9	40.6	27.3	18.6	28.1
BLIP-2 VQA	63.7	63.1	38.9	26.1	31.3	65.0
VPEVAL	34.5	63.8	48.9	29.4	85.7	73.5
VPEVAL†	63.7	63.8	51.2	29.5	85.7	79.0
DINO and DPT), which increases human correlation scores. Note that our object detection modules visually explain the evaluation results, as shown in Fig. 3, and we can have an even higher correlation when we have access to stronger future object detection models.
Human correlation of VPEVAL on open-ended prompts. We generate visual programs with our program generation LM on TIFA160 prompts [36]. The dataset consists of 800 images (160 prompts  $\times$  5 T2I models) and human judgments (1-5 Likert scale) along with other automatic metrics (BLIP-2 captioning, CLIP cosine similarity, and TIFA with BLIP-2) on the images. Table 4 shows that our VPEVAL achieves a better human correlation with TIFA (BLIP-2), and our VPEVAL $^{\dagger}$  version achieves an even higher correlation. The results indicate that using various interpretable modules specialized in different skills complements each other and improves human correlation, while also providing visual+textual explanations.
Human analysis on the generated programs. Lastly, we also measure the faithfulness of the generated evaluation programs. For this, we sample TIFA160 prompts and analyze the evaluation programs by (1) how well the modules cover elements in the prompt; (2) how accurate the module outputs are when run. We
Table 4: Human correlation on open-ended evaluation with Spearman's  $\rho$  .
Metrics	ρ (↑)
BLIP-2 Captioning
BLEU-4	18.3
ROUGE-L	32.9
METEOR	34.0
SPICE	32.8
Cosine-similarity
CLIP (ViT-B/32)	33.2
LM + VQA module
TIFA (BLIP-2)	55.9
LM + multiple modules
VPEVAL (Ours)	56.9
VPEVAL† (Ours)	60.3
find that programs generated by our VPEVAL have very high coverage over the prompt elements and high per-module accuracy compared to human judgment. We include more details in the appendix.
We propose two novel visual programming frameworks for interpretable/explainable T2I generation and evaluation: VPGEN and VPEVAL. VPGEN is a step-by-step T2I generation framework that decomposes the T2I task into three steps (object/count generation, layout generation, and image generation), leveraging an LLM for the first two steps, and layout-to-image generation models for the last image generation step. VPGEN generates images more accurately following the text descriptions (especially about object counts, spatial relations, and object sizes) than strong T2I baselines, while still having an interpretable generation program. VPEVAL is a T2I evaluation framework that uses interpretable evaluation programs with diverse visual modules that are experts in different skills to measure various T2I skills and provide visual+textual explanations of evaluation results. In our analysis, VPEVAL presents a higher correlation with human judgments than single model-based evaluations, on both skill-specific and open-ended prompts. We hope our work encourages future progress on interpretable/explainable generation and evaluation for T2I models.
We thank the reviewers for their valuable comments and feedback. This work was supported by ARO W911NF2110220, DARPA MCS N66001-19-2-4031, NSF-AI Engage Institute DRL211263, ONR N00014-23-1-2356, and DARPA ECOLE Program No. HR00112390060. The views, opinions, and/or findings contained in this article are those of the authors and not of the funding agency.
[55] Jaded AI. Easyocr: Ready-to-useOCR with  $80+$  supported languages and all popular writing scripts including latin, chinese, arabic, devanagari, cyrillic and etc., 2023.
[56] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pretraining with Frozen Image Encoders and Large Language Models, 2023.
[57] Saehoon Kim, Sanghun Cho, Chiheon Kim, Doyup Lee, and Woonhyuk Baek. mindall-e on conceptual captions. https://github.com/kakaobrain/minDALL-E, 2021.
[58] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector Quantized Diffusion Model for Text-to-Image Synthesis. In CVPR, 2022.
[59] Donghoon Lee, Jiseob Kim, Jisu Choi, Jongmin Kim, Minwoo Byeon, Woonhyuk Baek, and Saehoon Kim. Karlo-v1.0.alpha on coyo-100m and cc15m. https://github.com/kakaobrain/karlo, 2022.
[60] Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phuc Lê Khac, Luke Melas, and Ritobrata Ghosh. Dall-e mini, 7 2021.
[61] Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46, 1960.
[62] Klaus Krippendorff. Content Analysis: An Introduction to Its Methodology. Sage Publications, 4th edition, 2018.
[63] Mary L McHugh. Interrater reliability: the kappa statistic. Biochem. Med. (Zagreb), 22(3):276-282, 2012.
[64] Julius Sim and Chris C Wright. The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy, 85(3):257-268, 03 2005.
[65] Kishore Papineni, Salim Roukos, Todd Ward, and Wj Wei-jing Zhu. BLEU: a Method for Automatic Evaluation of Machine Translation. In ACL, 2002.
[66] Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In ACL Workshop, 2004.
[67] Satanjeev Banerjee and Alon Lavie. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In ACL Workshop, 2005.
[68] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: Semantic Propositional Image Caption Evaluation. In ECCV, 2016.
[69] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers, 2022.
In this appendix, we provide qualitative examples (Appendix A), additional VPGEN analysis (Appendix B), VPGEN error analysis (Appendix C), VPEVAL error analysis (Appendix D), VPGEN/VPEVAL implementation details (Appendix E), TIFA prompt element analysis details (Appendix F), limitations and broader impacts (Appendix G), and license information (Appendix H).
A Qualitative Examples
In Fig. 8 and Fig. 9, we provide example images generated by our VPGEN and Stable Diffusion v1.4 (the closest baseline of VPGEN) with various skill-based and open-ended prompts, respectively. In Fig. 10, we show VPGEN generation examples of placing objects that are unseen during the text-to layout finetuning of Vicuna 13B.
Figure 8: Images generated by Stable Diffusion v1.4 and our VPGEN (Vicuna 13B+GLIGEN) for our skill-based prompts.
B Additional VPGEN Analysis
In the following, we provide additional analysis of VPGEN, including counting prompts, bounding box quantization, using GPT-3.5-Turbo as layout generation LM, and visual quality metric (FID).
Counting prompts: More than 4 objects & Unspecified number of objects. In addition to the Count skill results with counts of 1 to 4 in the main paper Fig. 6, we provide additional results of VPGEN with counts of up to 7 in Table 5 and show generation examples in Fig. 11 (a) - (d). As expected, the trend continues from Fig. 6, where higher counts are more challenging. We also experiment with prompts without specific numbers. We find that VPGEN can successfully generate layouts/images even when the prompts do not have explicit counts. As shown in Fig. 11 (e), our VPGEN generates two Pikachus from the prompt "Pikachus on the table".
Figure 9: Images generated by Stable Diffusion v1.4 and our VPGEN (Vicuna 13B+GLIGEN) for open-ended prompts.
(a) Three pikachus are on a table
(b) A pikachu is to the left of a cat
Figure 10: Images generated by VPGEN (Vicuna 13B+GLIGEN) from prompts including objects (e.g., 'Pikachu', 'Link', 'master sword') that are not included in text-to layout annotations (Flickr30K Entities, COCO, PaintSkills) for Vicuna fine-tuning.
(c) A purple pikachu and a pink pikachu
(d) Link is holding a master sword
Bounding box quantization. As mentioned in the main paper, we use normalized and quantized coordinates with 100 bins to represent bounding boxes, following LM-based object detection works [49; 50]. In Table 6, we compare 100 and 1000 bins for the bounding boxes. Increasing the granularity of the bounding box to 1000 does not increase the accuracy in both skill-based and open-ended prompts, suggesting that the current 100-bin quantization does not hurt the accuracy of object placements.
GPT-3.5-Turbo v.s. Vicuna 13B. In our initial experiment, we tested GPT-3.5-Turbo to generate spatial layouts by showing in-context examples and found that the generated layouts were often inaccurate or not meaningful. For example, GPT-3.5-Turbo often generates a list of bounding boxes that is the same size as the entire image (e.g., [object 1 (0,0,99,99), object 2 (0,0,99,99) object3 (0,0,99,99)]). We conjecture that this is because their training corpus might not include many bounding boxes. Then, we collected text layout annotations and trained a language model to generate spatial layouts from text prompts.
For a quantitative comparison, we implement VPGEN with GPT-3.5-turbo with 36 in-context examples that cover different skills and compare it with Vicuna 13B trained on Flickr30k+COCO+PaintSkills. As shown in Table 7, Vicuna 13B based VPGEN shows higher skill-based and open-ended VPEval accuracies than GPT-3.5-Turbo based VPGEN.
Visual quality metric (FID). In Table 8, we compare our VPGEN (Vicuna13B + GLIGEN) to its backbone Stable Diffusion v1.4, in FID [30] (30K images of COCO val 2014 split) and VPEval Accuracy. Both VPGEN checkpoints show better skill-based accuracy than Stable Diffusion v1.4 while achieving comparable open-ended accuracy. and FID. In FID (lower the better), we find VPGEN (Flickr30k) < Stable Diffusion v1.4 < VPGEN (Flickr30k+COCO+Paintskills). We think that a bit of increase (but still reasonably good) in the FID of the Flickr30k+COCO+Paintskills
(a) 4 dogs
Figure 11: Images generated by our VPGEN with prompts requiring counting skills. (a) - (d) show the generation examples with count  $\geq 4$ . (e) shows generation from a prompt that does include exact count ('pikachus').
Count  $> = 4$
(b) 5 dogs
(c) 6 dogs
(d) 7 dogs
Without exact count in prompt
(e) Pikachus on the table
Table 5: VPEVAL Count skill accuracy of VPGEN with different number of objects.
VPEval Count Acc. (%) ↑
1	2	3	4	5	6	7
VPGEN	85.3	80.1	65.6	55.2	48.0	35.3	28.1
checkpoint is because the layouts of PaintSkills are different from those of natural scenes (COCO and Flickr30k).
Training with real v.s. synthetic images. Our layout generation module (Vicuna 13B) in VPGEN is trained on a mix of both 3D simulator-based synthetic (PaintSkills) and real-world images (MSCOCO/Flickr30K). We conduct an ablation study of training only with real v.s. synthetic images. We train the Vicuna on only PaintSkills and only Flickr30K and compare their results on both skill-based and open-ended prompts in Table 9. For skill-based prompts, PaintSkills training shows slightly higher average accuracy (46.0) than Flickr30K training (43.9). For open-ended prompts, PaintSkills training gets a VPEval score of (65.6), lower than Flickr30K training (71.0). This indicates that training on synthetic data can inject knowledge of specific skills such as counting and spatial relation understanding, but it is hard to cover diverse objects and attributes only with synthetic data.
C VPGEN Error Analysis
Layout v.s. Image generation on skill-based prompts. We analyze the performance of each step in the VPGEN pipeline to determine how much error is propagated with the following four metrics.
1. Object Recall: The ratio of whether correct objects are included in the generated layouts.
2. Object Count: The ratio of the correct number of objects is included in the generated layouts.
3. Layout Accuracy: VPEval accuracy of the generated layouts (using the generated layouts as the object detection results).
4. Image Accuracy: VPEval accuracy of the final image by GLIGEN (in the main Table 1).
As shown in Table 10, the layout accuracy is much higher than image accuracy, especially for Count/Spatial skills. This indicates that the major error source is the image rendering step (with GLIGEN), and using a more accurate layout-to-image generation model in the future would improve the accuracy of our VPGEN pipeline.
Human evaluation on open-ended prompts. As mentioned in the main paper Sec. 5.3, we ask two expert human annotators to evaluate 50 randomly sampled prompts from TIFA160 [36], with the following two metrics: (1) layout accuracy: if the generated layouts align with the text prompts; (2) image accuracy: if the generated images align with the text prompts. The human annotators show high inter-annotator agreements for both layout accuracy (Cohen's  $\kappa = 0.73$ , Krippendorff's  $\alpha = 0.73$ ) and image accuracy (Cohen's  $\kappa = 0.87$ , Krippendorff's  $\alpha = 0.87$ ). The Vicuna 13B's layout accuracy is  $(92\%)$  higher than GLIGEN's image correctness  $(65\%)$ . The result is consistent with the skill-based prompts error analysis result (Table 10) and suggests that a better layout-to-image generation model could further improve our VPGEN framework.
Table 6: Bounding box quantization: 1000 v.s. 100 bins.
# Bins	VPEval skill-based Acc. (%)↑	VPEval open-ended Acc. (%)↑
1000	49.8	68.2
100 (default)	51.0	68.3
Table 7: GPT-3.5-Turbo v.s. Vicuna 13B in VPGEN.
Model	VPEval skill-based Acc. (%) ↑	VPEval open-ended Acc. (%) ↑
GPT-3.5-Turbo (36 examples) + GLIGEN	40.7	65.5
Vicuna 13B + GLIGEN	51.0	68.3
Common error categories. We show some common categories of errors from the text-to layout generation step (with Vicuna 13B) and from the layout-to-image generation step (with GLIGEN), respectively. As shown in Fig. 12, the Vicuna 13B sometimes (a) generates objects that are not specified in prompts (e.g., 'chair' in the 1st example) or (b) misses objects that are mentioned in prompts (e.g., 'building' in the 5th example). As shown in Fig. 13, the GLIGEN sometimes (a) generates wrong objects (e.g., 'suitcase', instead of 'bench', in the 1st example), (b) misses some objects (e.g., 'umbrella' in the 3rd example), or (c) misses object details (e.g., 'closed oven' instead of 'oven with the door open' in the 5th example).
Challenging, non-canonical prompts. In Fig. 14, we show generation examples with challenging, non-canonical prompts (a) "A realistic photo of a Pomeranian dressed up like a 1980s professional wrestler with neon green and neon orange face paint and bright green wrestling tights with bright orange boots" (from DrawBench [24]) and (b) "a circle with  $3\mathrm{cm}$  radius and a circle with  $6\mathrm{cm}$  radius". Our VPGEN understands the important parts of the prompts (e.g., generating 'realistic', 'Pomeranian dog', 'bright orange boots' and generating layouts of two circles in different sizes), but misses some aspects (e.g., the bigger circle is not twice the size of the smaller circle). This is probably because the training prompts (COCO, PaintSkills, Flickr30k Entities) do not include many prompts written in these styles. We believe that scaling datasets with various sources can further improve VPGEN.
D VPEVAL Error Analysis
Evaluation program analysis: coverage and accuracy. For the evaluation program analysis (main paper Sec. 5.4), we experiment with two expert annotators using 50 randomly sampled TIFA160. We ask two expert annotators to analyze the evaluation programs in two criteria: (1) how well the evaluation program covers the content from the text prompt; (2) how accurately each evaluation module evaluates the images. To make it easier for the annotators, for (2) module accuracy analysis, we only ask them to check the output of three randomly sampled evaluation modules. We find that, on average, our evaluation programs cover  $94\%$  of the prompt elements, and our modules are accurate  $83\%$  of the time. This shows that our VPEVAL evaluation programs are effective in selecting what to evaluate and are highly accurate while also providing interpretability and explainability. In Fig. 15, we visualize our evaluation programs that cover different parts of the prompts and evaluation modules that provide accurate outputs and visual+textual explanations.
E VPGEN/VPEVAL Implementation Details
E.1 VPGEN Details