# Uncertainty in Visual Generative AI
## 4 Results and Discussion
### Cosine Similarity Results  
When using the initial prompt to compare the cosine similarity to the captions in Table 6, we had a wide variety of scores based on the word embeddings from various vector space models (Word2 Vec, GloVe, and FastText) and pre-trained language models (DistilRoBERTa, MiniLM-L12, and MiniLM-L6). There is a clear gap of at least 0.05 between the vector space models and the pre-trained language models. Word2 Vec scored the captions the lowest of the vector space models, but higher than any of the pre-trained language models, with values around 32%. GloVe scored captions higher, around the 40% similarity mark, but FastText gave the highest similarity scores, near 50% similarity. Though these scores are higher than the machine translation values, 50% would correspond with a neutral prompt/caption, meaning they are neither dissimilar nor similar. All the pre-trained language models gave relatively low similarity scores within the 18-26% range. This performance is expected to a degree since a one-word initial prompt is typically being compared to a multi-word sentence. Using the example from Figure 6, the initial prompt is \"soap\" and the GIT-created caption is \"a bar of soap on a white background\". Even though \"soap\" is in the caption, there are several other words that influence the sentence vector, which would have to cancel one another out perfectly to be left with a vector equivalent to the word embedding for \"soap\". Table 7 includes the highest scores from this study across the board. Opposed to comparing the initial prompt, the Craiyon prompt is used for comparison to the generated caption. Since the Craiyon prompts were structured more similarly to the generated captions, it is not surprising that the values in Table 7 are higher than those in Table 6. However, similar behaviors exist between the models presented in both tables; vector space models provide higher similarities to the pre-trained language models. Word2 Vec still assigned the lowest similarity scores amongst the vector space models, but there is about a 10% increase from the average similarity using the initial prompt. We see more significant jumps into the 70s for GloVe and FastText, which is approximately a 30% increase from the initial prompt scores in Table 6. All the pre-trained language models remained in the 20-30% range with single-digit increases from when the initial prompt was used. Machine translation methods yielded consistently low scores in comparison to the cosine similarity scores;
2. For the cosine similarity metrics, the Craiyon prompts yielded higher scores than the initial prompts when comparing them with the generated captions;
3. Vector space models (Word2 Vec, GLoVe, and FastText) were most generous with their similarity scores compared to pre-trained language models;
4. Image-to-text models minimally affected the similarity scores. These major results show how various text evaluation methods can be used to evaluate \"uncertainty\" within generative AI models. As the various image-to-text models did not seem to impact the uncertainty quantification, the evaluation metrics are what researchers should study. The \"best\" evaluation metric is dependent on validation by human judgment and how much \"uncertainty\" a human believes to be associated with a prompt-caption pair. These results suggest there is a significant amount of uncertainty within the AIGC based on our metrics. One potential explanation for this is the existence of many data elements classified as Case B (the prompts do not align with the images) or E (neither the prompts nor the images align with one another) from Figure 3. The detrimental results of this can be seen in Figure 6, where the Craiyon prompt was \"a bar of soap on a white background\" and the InstructBLIP caption was \"a block of cheese on a white background\".