SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SNLI-VE-only variant is behind with an average score of 79.7, while achieving the highest scores for 2 out of 7 datasets. This shows that integrating synthetic training data leads to notable improvements on synthetic images on DrawBench (+4%), EditBench (+11.7%), , COCO t2i (+5.5%), PickaPic-Con (+2.2%). Nevertheless, the inclusion of synthetic training data did not enhance performance on the COCO-Con dataset, comprised solely of natural images. This indicates that the variation in image types could be a contributing factor that calls for additional exploration. Notably, the last row shows a simple average between  $\mathrm{VQ}^2$  and our leading fine-tuned PaLI model, that produces higher performance, suggesting that they complement each other effectively.
(a) "the orange lollipop is sad and the red lollipop is surprised"  
Q: What is the orange lollipop feeling? A: sad
(1) Winoground
(b) "Someone in a blue hat standing on a snowy hill"  
Q: What is the person wearing?  
A: blue hat
(2) CocoCon
(c) "A black apple and a green backpack"  
Q: What color is the apple? A: black
(3) DrawBench
Figure 4: Contradicting captions and the question/answer pairs with lower  $VQ^2$  alignment score, indicating the contradiction reason.
Figure 5: Highly correlated  $\mathrm{VQ}^2$  and PaLI scores vs. human rankings of text-to-image models Winoground Results. Table 3 shows the performance of the different methods on the challenging Winoground dataset, which requires strong visual reasoning and compositional understanding skills. Our zero-shot approach,  $\mathrm{VQ}^2$ , achieves state-of-the-art results on this dataset, surpassing other strong baselines, with a group score of  $30.5\%$ . This again indicates that  $\mathrm{VQ}^2$ 's approach that decomposes the alignment task into multiple question-answer pairs is a promising path for image-text alignment.
**Contradiction Generation.** We assessed the  $\mathrm{VQ}^2$  method's capacity to detect image-text contradictions, as shown in fig. 4. Contradiction generation relies on identifying the question-answer pair with the lowest VQA score, signaling the least likely alignment between the image and the text. Three paper authors evaluated whether a particular contradiction (consisting of a question and an answer) accurately represents the primary discrepancy between the image and the text. The majority vote among the authors determined the final outcome, yielding the following accuracy rates:  $88\%$  for Coco-Con,  $74\%$  for DrawBench, and  $80\%$  for Winoground. This indicates that our method is capable of identifying these contradictions by investigating the structure and content of the given caption and image. As a result, our method can achieve strong results, particularly on datasets that require compositional understanding.
Comparing Generative Models.  $\mathrm{VQ^2}$  s ability to compare between generative models is demonstrated on the results of DrawBench and COCO-t2i, which include generated images from different models, together with human quality ratings. fig. 5 shows that the  $\mathrm{VQ^2}$  and our fine-tuned PaLI ranking correlates very well with human ranking  $(R^2 >0.92)$ . In addition, since unlike human annotations, the  $\mathrm{VQ^2}$  score is consistent across datasets, it offers a way to evaluate dataset difficulty on an absolute
Table 4: Comparison of human-labeled quality scores for top-ranked images with model breakdown
Dataset	Model	Random	CLIP	PaLI	VQ2
COCO t2i	SD 1.4	68.6	74.6	88.2	86.4
SD 2.1	71.3	81.2	84.5	87.3
DrawBench	SD 1.4	66.7	77.4	77.4	87.1
SD 2.1	59.0	78.0	87.0	82.0
Reranking Using Alignment Assessment. Alignment scores can also be used for reranking candidate generations, on top of evaluation. To demonstrate this, we re-rank the image candidate per prompt in the DrawBench and COCO-t2i datasets. We do so using  $\mathrm{VQ}^2$  and CLIP and measure the human-labeled quality of the top-ranked image for each method. The results, presented in table 4, show that ranking with  $\mathrm{VQ}^2$  consistently achieves higher quality scores when compared to ranking with CLIP. One such example is shown in fig. 6, where both  $\mathrm{VQ}^2$  and our top-performing fine-tuned PaLI model demonstrate superior ranking by placing the brown-and-white cats above the white-only cats. This consistency between  $\mathrm{VQ}^2$  and PaLI highlights their alignment evaluation models' potential for enhancing text-to-image systems, which contrasts with the divergent ranking exhibited by CLIP.
$VQ^2$  score: 0.878 (1st)
CLIP similarity: 0.236 (4<sup>th</sup>)
Figure 6: Four COCO-t2i text-to-image model outputs ranked by VQSQR scores, correlating with top PaLI model. Image order and CLIP (RN50x64) similarity scores given, but not aligned with  $\mathrm{VQ}^2/\mathrm{PaLI}$  ranks.
PaLI ft. score: 0.992 (1st)
$VQ^2$  score: 0.846  $(2^{nd})$
PaLI ft. score: 0.992 (2nd)
CLIP similarity: 0.238 (3rd)
A brown and white cat is in a suitcase
$VQ^2$  score: 0.731  $(3^{rd})$
PaLI ft. score: 0.803  $(3^{rd})$
CLIP similarity: 0.253 (1st)
$VQ^2$  score: 0.717  $(4^{th})$
PaLI ft. score: 0.437 (4th)
CLIP similarity: 0.25 (2nd)
Our work advances research in visual entailment (VE) [27], visual question answering (VQA) [46], text-to-image alignment evaluation, and cross-task consistency for multi-modal models, with a focus on enhancing the semantic understanding of image-caption relationships.
Textual Entailment (TE) [26, 47] evaluates the truthfulness of a textual hypothesis given a textual premise, providing a key benchmark for the semantic capabilities of neural network models [48-50, 35]. Recently, TE has been adapted to the multimodal domain as Visual Entailment (VE) [27] to assess the semantic alignment between images and text. Vision-and-language models like CLIP [15], CoCa [22], BLIP [6], BLIP2 [7] and OFA [23] often act as bag-of-words models, lacking a deep comprehension of language compositionality [16]. Our approach addresses this by generating multiple questions probing diverse semantic aspects, thereby improving performance on challenging compositional tasks like Winoground [24] and unnatural images as in Drawbench [5].
Unlike DrawBench [5] and DALL-Eval [51] which depend on human feedback and operate within a discrete set of alignments, our approach produces automated scores for a broader range of text-image alignments, facilitating efficient evaluation of vision-and-language models. Our approach also surpasses the recently proposed TIFA [44], which may be due to employing more question-answer pairs and tailored models for question generation and answering.
Several works have explored cross-task consistency in multi-modal models across various modalities. VQA studies have tackled inconsistencies and enhanced consistency using data augmentation and contrastive loss. NLP researchers have improved consistency across tasks or within a single task by employing counterfactual instances or contrast sets [26, 47]. Our research aligns with studies that evaluate natural text and images [52]; however, extending the focus to synthetic images and texts, and aligning with synthetic image understanding research [53-58]. We introduce two unique approaches to address the complexities of image-text alignment.
Another related effort is PickScore [29], which predicts human preferences for image quality and aesthetics by ranking or choosing between two images. In contrast, our methods independently score a single image and focus specifically on image-text alignment.
We recognize that in some cases, making a binary decision for whether a text and an image are aligned may be difficult, also for humans. To tackle this limitation, we provided human annotators with comprehensive guidelines, which resulted in a high inter-annotator agreement (Fleiss-Kappa score of 0.722 with  $80\%$  of the cases where all annotators agreed on the entailment label).
Although many images in our datasets were obtained by others and not created by us, we made an effort to ensure that they do not contain harmful or potentially harmful content, such as NSFW or biased imagery. During the annotation process, three individuals examined each image and indicated if it could be considered offensive. Additionally, two of the authors manually reviewed the images
for any harmful content. However, we understand that the perception of harmful or offensive content may vary among individuals and may be subject to personal interpretation.
We addressed the task of image-text alignment evaluation, which we find very close to the Visual Entailment (VNLI) task. We first introduced the SeeTRUE benchmark, which covers the mix of real and synthetic text and image pairs in text-to-image and image-to-text generation tasks, and includes challenging cases based on generated contradictions. We then proposed two methods,  $\mathrm{VQ}^2$  and end-to-end VNLI, which outperform strong baselines on SeeTRUE and can serve as a starting point for future research on the task.
In future work, we would like to employ our automatic evaluation models for guiding the training of text-to-image and image-to-text models towards more aligned outputs, following recent trends in text-to-text generation [59, 60]. For example, such models may be useful either for filtering training examples or as a reward when training models using reinforcement learning.
[1] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[2] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[3] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674-10685, 2022.
[4] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2022.
[5] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
[6] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[7] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
[8] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716-23736, 2022.
[9] Gary Marcus, Ernest Davis, and Scott Aaronson. A very preliminary analysis of dall-e 2. arXiv preprint arXiv:2204.13807, 2022.
[10] Nan Liu, Shuang Li, Yilun Du, Josh Tenenbaum, and Antonio Torralba. Learning to compose visual relations. Advances in Neural Information Processing Systems, 34:23166-23178, 2021.
[11] Royi Rassin, Shauli Ravfogel, and Yoav Goldberg. Dalle-2 is seeing double: flaws in word-to-concept mapping in text2image models. arXiv preprint arXiv:2210.10606, 2022.
Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
[51] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. arXiv preprint arXiv:2202.04053, 2022.
[52] Adyasha Maharana, Anita Kamath, Christopher Clark, Mohit Bansal, and Aniruddha Kembhavi. Exposing and addressing cross-task inconsistency in unified vision-language models. arXiv preprint arXiv:2303.16133, 2023.
[53] Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. arXiv preprint arXiv:2212.10015, 2022.
[54] Hang Li, Jindong Gu, Rajat Koner, Sahand Sharifzadeh, and Volker Tresp. Do dall-e and flamingo understand each other? arXiv preprint arXiv:2212.12249, 2022.
[55] Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images. arXiv preprint arXiv:2303.07274, 2023.
[56] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-image models with human preference. arXiv preprint arXiv:2303.14420, 2023.
[57] Ali Borji. Generated faces in the wild: Quantitative comparison of stable diffusion, midjourney and dall-e 2. arXiv preprint arXiv:2210.00586, 2022.
[58] Andreas Stöckl. Evaluating a synthetic image dataset generated with stable diffusion. arXiv preprint arXiv:2211.01777, 2022.
[59] Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. Increasing faithfulness in knowledge-grounded dialogue with controllable features. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 704-718, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.58. URL https://aclanthology.org/2021.acl-long.58.
[60] Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. mFACE: Multilingual summarization with factual consistency evaluation, 2022.
[61] Norman P Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. A domain-specific supercomputer for training deep neural networks. Communications of the ACM, 63(7):67-78, 2020.
[62] Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. Scaling up models and data with t5x and seqio, 2022. URL https://arxiv.org/abs/2203.17189.
[63] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. Jax: composable transformations of python+ numpy programs. 2018.
A.1 Dataset Supplementary Materials
1. Dataset documentation, metadata, and download instructions: anonymous.
2. Intended uses: we hope SeeTRUE will be used by researchers to evaluate image-text matching models.
3. Author statement: We bear all responsibility in case of violation of right in using our benchmark.
4. Each dataset's license is described below. Our additional human annotations and ge-generated images are licensed under CC-BY 4.0 license https://creativecommons.org/licenses/by/4.0/legalcode.
5. Hosting & preservation: the dataset will be hosted in Huggingface Datasets, accessible and available for open research.
6. SeeTRUE fields are presented in table 5.
Additional licensing details: we do publish the datasets we annotated in this work, and do not re-publish the existing SNLI-VE and Winoground datasets. Full licenses:
1. MS COCO [28]: https://cocodataset.org/#termsofuse
2. EditBench [25]: https://research.google/resources/datasets/editbench/, https://www.apache.org/licenses/License-2.0
3. DrawBench [5]: https://Imagen.research.google/, https://docs.google.com/spreadsheets/d/1y7nAbmR4FREi6npB1u-Bo3GFwd0PYJc617rBOxIRHY/edit#gid=0
4. Pick-a-Pick [29]: https://huggingface.co/datasets/yuvalkirstain/pickapic_v1
5. SNLI-VE [27]: https://github.com/necla-msl/SNLI-VE
6. Winoground [24]: https://huggingface.co/datasets/facebook/winoground
Table 5: SeeTRUE Rows Examples
image	text	label	original_dataset_id	dataset_source
img1	A zebra to the right of a fire hydrant.	0	text_133_image_1228	drawbench
img2	A group of people standing next to bags of luggage.	1	text_105_image_1377	coco_t2i
img3	a tiny figurine is surrounded by cell phones on a table.	1	3786	editbench
A.2 Additional Examples
We added additional examples from (a) instances from SeeTRUE fig. 7; (b) the  $VQ^2$  method fig. 8 to visualize the question-answer pairs; and (c) the ConGen technique fig. 9. This will underscore the versatility of ConGen in handling both real and synthetic images, as well as the variety of contradictions it can generate.
A.3 Human Annotation Process
In order to provide reliable human labels for our datasets, we conducted an annotation process using the SeeTRUE platform. The process comprised several steps, including setting qualification requirements, providing instructions, and evaluating annotator agreement.
We set the basic requirements for our annotation task as follows: a percentage of approved assignments above  $98\%$ , more than 5,000 approved HITs, and annotator locations limited to the US, UK, Australia, or New Zealand. We selected 5 examples from our dataset for a qualification test and screened the annotators' results. fig. 10 displays a sample of the Mechanical Turk user interface. The payment for the crowd-workers was 15-18 USD hourly.
image
(image)	text (string)	label
(int64)	original_dataset_id
(string)	dataset_source
(string)
"A zebra to the right of a fire hydrant."	0	"text_133_image_1228"	"drawbench"
"A group of people standing next to bags of luggage."	1	"text_105_image_1377"	"coco_t2i"
"The old tow truck tries to lift an overturned truck...	0	"text_178_image_2319"	"coco_t2i"
"Abraham Lincoln touches his toes while George Washington...	0	"text_103_image_993"	"drawbench"
"A cat on the lid of a toilet looking perturbed."	1	"text_3_image_47"	"coco_t2i"
A brown bird and a blue bear
Figure 7: An example from SeeTRUE Huggingface Dataset.
Question	Answer (based on text)	VQA "yes" probability	VQ2 score
Aside from a blue bear, what other animal is depicted in the picture?	A brown bird	0.26	0.4
Aside from the brown bird, what other animal is in the picture?	a blue bear	0.41
What is the blue animal?	bear	0.44
Aside from a bear, what other animal is depicted?	bird	0.37
What color is the bear?	blue	0.5
What color is the bird in the picture?	brown	0.42
A wine glass on top of a dog
Question	Answer (based on text)	VQA "yes" probability	VQ2 score
What is on top of a dog?	A wine glass	0.82	0.78
A wine glass is on top of what animal?	a dog	0.78
What is the wine glass on top of?	dog	0.77
What is on top of a dog?	glass	0.74
On what part of the dog is the wine glass located?	top	0.64
Where is the wine glass located?	top of a dog	0.88
What kind of glass is on the dog?	wine	0.82
What is on top of a dog?	wine glass on top	0.79
Figure 8: Examples of the  $VQ^2$  method. The answers extracted from the text are named entities, nouns, and multi-word spans, such as adjectives and locations.
The instructions provided were as follows:
Evaluate the given image and text to determine if they match, selecting either "Yes" or "No". Some images may be synthetically generated by a text-to-image model. To assess the match, mentally generate a textual description for the image (no need to write it down) and compare this generated description to the given text. If the descriptions closely resemble each other, mark "Yes". If not, mark "No" and provide feedback on the specific issue causing the misalignment, focusing on the primary issue if multiple misalignments are present. If you encounter an image or text that may be offensive due to bias, race, NSFW content, etc., mark the checkbox to indicate this issue.
Full agreement metrics are presented in table 6. As shown in the table, the percentage of cases where all annotators agreed and the Fleiss-Kappa scores vary across the datasets, with COCO-Con
large white and pink rose planter surrounded by lush
greenery
large white and pink rose planter surrounded by dead greenery
a red-haired bowie is painted on the wall
a blue-haired bowie is painted on the wall
Figure 9: Examples from the ConGen method to generate contradictions. The top caption is the original, positive one, and the bottom one is the contradiction generated by ConGen.
milk pouring into a coffee cup on a wooden table
milk pouring into a coffee cup on a plastic table
Figure 10: Annotation interface for determining whether a given image-text pair are aligned.
a man with a mustache is sitting in front of some pine trees
Does the image present all the details described in the text correctly?
If 'No', what is the model feedback? (*main* misalignment)
\*Might be offensive
exhibiting the highest level of agreement and Drawbench the lowest. These differences highlight the varying levels of complexity within the datasets.
Table 6: Agreement metrics for different datasets.
Dataset	Full	Drawbench	COCO t2i	COCO-Con	PickaPic-Con
# Items	8,527	1,968	2,586	1,992	1,981
% all agreed	80	76	78	86	77
Fless-Kappa	0.72	0.66	0.68	0.81	0.69
A.4 Comparing  $VQ^2$  variants
$VQ^2$  consists of two main parts: generating question-answer pairs and assessing question-answer pair alignment against the image. We have experimented with different variants of the  $VQ^2$  zero-shot method.
Assessing question-answer pair alignment methods Given a question-answer pair, we would like to assess the question based on the image and compare it to the information in the text. We experimented with several configurations for answer alignment:
1. Type A: Given a question-answer pair  $(q_j, a_j)$  generated from the text, we answer the question using a VQA model and obtain an answer based on the image  $a_j^I = VQA(q_j, I)$ . We compare a pair of  $(q_j, a_j)$  with a pair of  $(q_j, a_j^I)$  using an Natural Language Inference (NLI) model, where the pair based on the text serves as the premise and the other as the hypothesis. We define the alignment score  $s_j$  as the probability of the NLI model for answering "entailed".
2. Type B: As done in type A, we answer the question using a VQA model and obtain the answer  $a_{j}^{I} = VQA(q_{j},I)$ . We use the VQA model again to compare the two answers and determine whether they are the same. The question is formulated as "Is  $\{a_j\} == \{a_j^I\}$  in this image?" We define the alignment score  $s_j$  as the probability of the VQA model for answering "yes".
3. Type C: We reformulate each question and answer candidate pair  $(q_j, a_j)$  into a new yes-no predicate question  $q_j'$  using the format "is  $\{a_j\}$  true for  $\{q_j\}$  in this image?". The VQA model is then invoked to answer the predicate question  $(q_j')$  over image  $I$ . We define the alignment score  $s_j$  as the probability of the model for answering "yes".
Generating question-answer pairs To produce question-answer pairs from the text, we first extract informative spans in the text  $T$ . We extract as answer candidates all named entities and noun phrases in  $T$  using spaCy. We noticed that for short text  $T$ , this method doesn't produce enough question-answer pairs to assess the alignment between the text and the image. Thus, we extend the answer candidates by adding multi-word spans, such as adjectives ("black and white") and location "in the air". We use the extended answer candidate extraction in all of our experiments. The number of question-answer pairs is dependent on the length of the text  $T$ . In EditBench, the text contains 12.5 words on average, and it results in average of 34.9 question-answer pairs with extended answer candidates.
Table 7 summarize the results of the  $VQ^2$  variants on the EditBench dataset. Our zero-shot  $VQ^2$  method is  $VQ^2$  type C, it outperforms other configurations and it is more efficient, since it requires a single run of the VQA model for the question-answer pair assessment.