# Uncertainty in Visual Generative AI
## 3 Methodology  
To measure uncertainty in visual GAI, we design an agnostic pipeline to compare textual image descriptions to the textual inputs (called \"prompts\") as shown in Figure 1. First, a database of prompts (green data block of Figure 1) is needed, which will be used as inputs into the text-to-image visual generative AI model. Next, this text-to-image (blue block of Figure 1) model generates an image based on the prompt. Then, the image that is produced is sent to an image-to-text (grey block of Figure 1) model to create an AI-generated caption of the image. Finally, the resulting caption is to be evaluated against the textual prompt used to originally generate the image (orange block of Figure 1). During the evaluation step, uncertainty is quantified as the similarity gap between the original textual prompt (from the database) and the resulting caption (provided by the image-to-text model). This is an agnostic, modular pipeline that can utilize different datasets, models, and evaluation methods to measure uncertainty in similar problems. The remainder of this section discusses the specific dataset, model, and evaluation used in this study.  
The pipeline in Figure 1 was customized for this study with the selected dataset, models, and evaluation methods shown in Figure 2. This process is explained more in-depth in Sections 3.1-3.4. The textual prompt dataset was provided by the modified version of the Sternberg and Nigro dataset used in [111], which produced 495 initial prompts.  
The selected text-to-image model was Craiyon V3 [35; 36; 37]. Craiyon performs 2 unique steps. First, it creates its own version of the initial prompt, which we will call the \"Craiyon prompt\" (e.g., the initial prompt is \"soap\" and the Craiyon prompt adds details such that the new prompt is \"a bar of soap on a white background\"). This Craiyon prompt is used to create nine images by default. Therefore, every initial prompt yields 1 Craiyon prompt and 9 resulting images. Of the 495 initial problems, 49 were removed for quality reasons, leaving 446 initial prompts with corresponding Craiyon prompts. Craiyon creates 9 images per prompt, so the 446 remaining prompts were turned into 4014 images by Craiyon.  
Figure 1: Uncertainty in visual GAI evaluation agnostic pipeline.  
Figure 2: Customized pipeline for study.  
All 4014 images were passed through four image-to-text models--GIT [75], BLIP [76], BLIP-2 [78], and InstructBLIP [79]--for later comparison to one another. Due to various quality control reasons discussed in Section 3.3, not every image had a sufficient caption generated. Therefore, the insufficient captions were removed from the analysis. Thus, there were 16,004 total captions (3942 for GIT and 3994 for each BLIP-family model).  
These captions were then evaluated on a variety of metrics, including machine translation methods and the cosine similarity of word embeddings. Seven machine translation methods were selected: Bilingual Evaluation Understudy (BLEU) (BLEU-1, BLEU-2, BLEU-3, and BLEU-4 were used, where the number represents the number of matching n-grams BLEU looks for) [89], Recall-Oriented Understudy for Gisting Evaluation--Longest common subsequence (ROUGE-L) [90], Metric for Evaluation of Translation with Explicit ORdering (METEOR) [91], and Semantic Propositional Image Caption Evaluation (SPICE) [95]. For the cosine similarity method, six models were selected: Word2 Vec [97; 98], Global Vectors (GloVe) [101], FastText [102], Distilled Robustly optimized Bidirectional Encoder Representations from Transformers approach (DistilRoBERTa) [110], Mini Language Model 12 Layer (MiniLM-L12) [112], and MiniLM 6 Layer (MiniLM-L6) [112].