SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Michal Yarom\*
Yonatan Bitton $^{\mathrm{H,G*}}$
Soravit ChangpinyoG
Roee AharoniG
Jonathan Herzig $^{\mathrm{G}}$
Oran Lang
Eran Ofek $^{G}$
Idan Szpektor
$^{\mathrm{G}}$ Google Research The Hebrew University of Jerusalem
Michalyarom@google.com, yonatan.bitton@mail.huji.ac.il
Automatically determining whether a text and a corresponding image are semantically aligned is a significant challenge for vision-language models, with applications in generative text-to-image and image-to-text tasks. In this work, we study methods for automatic text-image alignment evaluation. We first introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets from both text-to-image and image-to-text generation tasks, with human judgements for whether a given text-image pair is semantically aligned. We then describe two automatic methods to determine alignment: the first involving a pipeline based on question generation and visual question answering models, and the second employing an end-to-end classification approach by finetuning multimodal pretrained models. Both methods surpass prior approaches in various text-image alignment tasks, with significant improvements in challenging cases that involve complex composition or unnatural images. Finally, we demonstrate how our approaches can localize specific misalignments between an image and a given text, and how they can be used to automatically re-rank candidates in text-to-image generation.<sup>1</sup>
Figure 1: Overview of our approach to text-image alignment evaluation using SeeTRUE. We curate diverse pairs of real and synthetic text and images and use automatic contradiction generation and human evaluation to create a benchmark dataset. We propose two methods for text-image alignment evaluation:  $\mathrm{VQ}^2$  and VNLI, demonstrated with example pairs.
*Equal contribution. Yonatan participated in this work as part of an internship at Google Research.
Data and code are attached to this submission.
The recent success and proliferation of multi-modal large language models (LLMs) for text-to-image and image-to-text generation [1-8] make such technology increasingly useful for a wide range of creative applications. However, such models still struggle in generating semantically-aligned image-text pairs; in text-to-image generation, models do not cope with complex specifications [9, 10] or fail to map words in the prompt to visual entities [11, 12]. In image captioning, object hallucination is a long-standing challenge [13] with generated captions still being inferior to human-written ones [14].
Given the above, the task of automatically determining whether a given text-image pair is semantically aligned is highly important, as it is useful both for evaluating and for improving text-to-image and image-to-text models. However, existing evaluation approaches are still far from ideal; common methods like CLIP [15] or BLIP [6, 7] are based on encoding the image and text as fixed-size embeddings, making it hard to model complex semantics [16]. In addition, while the task is relevant both to text-to-image and image-to-text generation, it is usually studied in silo while considering only one of the applications, thus impeding progress.
In this work, we promote a comprehensive approach to evaluating image-text alignment. We introduce SeeTRUE, a diverse evaluation suite which includes a wide range of image-text pairs with human judgments that determine if the image and text are semantically aligned. SeeTRUE encompasses both real and synthetic images and text, allowing the assessment of text-image alignment models' generalization capabilities across various tasks and 31,855 labeled examples from diverse sources. As part of constructing SeeTRUE, we also introduce a novel method for generating contradicting captions from existing ones by prompting a large language model with tailored instructions.
We present two approaches for automatic image-text alignment evaluation. The first,  $\mathrm{VQ}^2$ , utilizes question generation and visual question answering by generating questions related to the text [17] and ensuring that the correct answer is obtained when asking these questions with the provided image. The second method, Visual Entailment  ${}^{2}$  (VNLI),involves directly fine-tuning a large pretrained multimodal model to predict if a given image-text pair is semantically aligned. Both strategies are inspired by recent studies on evaluating factual consistency between two texts [18-21].
We conduct comprehensive experiments on SeeTRUE, demonstrating that both our  $\mathrm{VQ^2}$  and VNLI methods outperform a wide range of strong baselines, including various versions of CLIP [15], COCA [22], BLIP [6, 7], and OFA [23]. While previous work showed that vision-and-language models tend to exhibit sub-optimal "bag-of-words" behavior [16], the  $\mathrm{VQ^2}$  method particularly excels on datasets with compositional challenges, achieving state-of-the-art results on the Winoground dataset [24] e.g. by improving the group score from  $16\%$  to  $30.5\%$ . Our methods also demonstrate improved performance when evaluating synthetic images (e.g. on DrawBench [5] and EditBench [25]). Finally, we showcase how  $\mathrm{VQ^2}$  can identify specific sources of misalignment for a given text-image pair and how our methods can re-rank generated image candidates for a given prompt.
To summarize, our contributions are as follows: (1) We introduce the SeeTRUE benchmark for meta-evaluation of image-text alignment. (2) We introduce a novel method to generate contradicting image captions from given captions with LLMs. (3) We suggest two reference-free metrics for image-text alignment evaluation:  $\mathrm{VQ}^2$ , based on question generation and visual question answering, and VNLI, based on fine-tuning large multimodal language models. (4) We conduct extensive evaluation of the above approaches against strong baselines, demonstrating superior performance over multiple datasets. (5) We release our evaluation suite, models and code to foster future work.
2 SeeTRUE: A Comprehensive Text-Image Alignment Benchmark
We begin by introducing SeeTRUE, a diverse benchmark for meta-evaluation of image-text alignment methods, covering the 4-way combinations of real and synthetic text-and-image pairs. It addresses limitations in current benchmarks, which mainly focus on natural images and often lack challenging negative captions. SeeTRUE allows to better assess the generalization abilities of text-image alignment models across various tasks.
Defining how image-text alignment is assessed has a direct impact on the construction of evaluation datasets. As images can display more details than described in their caption or text prompt, we define
2We use the terms Entailment and Natural Language Inference (NLI) interchangeably.
Table 1: SeeTRUE: a benchmark for image-text alignment encompassing 31,855 real and synthetic image-text pairs from diverse datasets and tasks. An example from each dataset is presented below.
Real Text + Real Images	Real Text + Synthetic Images	Synthetic + Real	Synthetic + Synthetic
Dataset Name	SNLI-VE	Winoground	DrawBench	EditBench	COCO t2i	COCO-Con	PickaPic-Con
# Test Examples	17,901	1,600	1,968	3,827	2,586	1,992	1,981
% Positive / Total	33.3%	50%	55.7%	36.9%	63.6%	52.7%	44.1%
Labeled in this work?	X	X	✓	✓	✓	✓	✓
Image
Text	the player swings his bat	the heavy oncoming traffic is contrasted with the light outgoing traffic	A blue cup and a green cell phone	a few pink candles and some cream on top of a cake.	A person on a snow board high up in the air.	A giraffe leaned over in a plush field next to some cows	a doctor wearing a white coat in the middle of a street
Human Label	True	True	False	True	False	False	True
image-text alignment as the case where all the details described in the text are accurately represented within the image. Inspired by the Textual Entailment task [26] which judges for two pieces of text whether one (the "hypothesis") can be inferred given the other (the "premise"), our definition maps the image to the premise and the text to the hypothesis, resulting in the task of predicting whether the information in the text can be inferred from the given image.
We describe the datasets included in our benchmark, with a high-level overview in Table 1.
Real text and real images. For pairs of human-written text and real (non-generated) images, we include the SNLI-VE [27] and Winoground [24] datasets. SNLI-VE is a widely adopted VNLI dataset containing an image, a text, and a label of the alignment between the two - entailment, contradiction, or neutral. Winoground is a challenging dataset for compositional understanding, where each example includes two images and two text captions, where the task is to match each text to its corresponding image. The captions only differ in a few words, which should result in distinct visual interpretations. For example, "some plants surrounding a lightbulb" vs. "a lightbulb surrounding some plants".
Real text and synthetic images. For datasets that represent text-to-image generation tasks we use EditBench [25] which offers prompts and images generated by various text-to-image models given those prompts, accompanied by alignment ratings. To encourage more diversity in the data, we also create new datasets by generating images using Stable Diffusion models [3] (V1.4 and V2.1) and Imagen [5] by prompting them with COCO [28] captions and text prompts from DrawBench [5], creating the COCO text-to-image ("COCO t2i") and the DrawBench text-to-image datasets.
Synthetic text and real images. This category includes a new dataset which we name COCO-Con. COCO-Con is generated using a novel automatic method which we describe in detail in Section 2.3. Specifically, we generate synthetic contradicting captions for COCO images based on their original captions by prompting a large language model, and verify the resulting captions with human raters.
Synthetic text and synthetic images. We utilize PickaPic [29], a source of user-generated and ranked synthetic images. We create synthetic captions using BLIP2 [7] and employ our automatic contradiction generation method (Section 2.3) to produce unaligned captions. This category evaluates synthetic text that is generated by image captioning models, e.g. for improving textual image search.
We note that some of the datasets are only used for testing (e.g., Winoground, DrawBench, EditBench) while others include both training and test sets (e.g., SNLI-VE, COCO t2i, COCO-Con, PickaPic-Con). This allows us to investigate different training configurations and their effect on performance.
(a) SeeTRUE
Figure 2: (a) The SeeTRUE generation process. (i) An image-text pair from a dataset is used to generate a contradicting caption using ConGen. (ii) An image (real or synthetic) is passed through a captioning model to generate a caption, which is then passed to ConGen to generate a contradicting caption. (iii) A text-to-image model is applied on captions from the dataset to create multiple imagedataset pairs. All the resulting examples are evaluated by human raters to create SeeTRUE. (b) The contradiction generation process (ConGen) takes a caption as input and instructs an LLM to generate variants that contradict it. An NLI model is used to select the variant with the lowest entailment score.
2.2 Human Annotation and Evaluation
To standardize the labeling scheme across datasets, we follow TRUE [19] and use binary annotations for alignment/misalignment. In datasets with three-way annotations (e.g. Entailment, Contradiction, Neutral) we convert the labels to binary labels by collapsing all non-entailment/non-alignment labels to a single negative label.
Some datasets, such as COCO-Con and PickaPic-Con, start with automatically generated labels, while others lack annotations entirely (e.g. DrawBench). To make sure we have high quality labels we conduct human annotation for all test examples in such datasets. We ask three crowd-workers from Amazon Mechanical Turk (AMT) to evaluate whether a given image-text pair is aligned, by answering the question: "Does the image present all the details described in the text correctly?" with "Yes" or "No". If the answer is "No", the workers are also requested to describe the main misalignment to enhance the annotation quality. While the random chance of agreement is  $25\%$ , the annotators reached consensus in  $80\%$  of cases. Furthermore, we measured a Fleiss-Kappa [30] score of 0.722, showing a good level of agreement between the annotators. Full annotation details, AMT user interface example, and agreement numbers per dataset can be found in appendix A.3.
The datasets we annotated include DrawBench, COCO t2i, COCO-Con and PickaPic-Con, with statistics presented in Table 1. These datasets vary in their positive/negative distribution, with COCO t2i having the highest percentage of positives (63.6%) and DrawBench having the lowest (36.9%). The agreement with the auto-label is  $94\%$  for COCO-Con and  $77\%$  for PickaPic-Con. To prevent the inclusion of offensive images, particularly those that are synthetically generated, annotators are asked to mark any images that may be considered offensive and these were discarded.
2.3 ConGen: Generating Contradicting Captions by Prompting LLMs
We propose an automatic method for generating unaligned captions from existing, aligned image-and-text pairs, with the goal of creating challenging examples for evaluation and training. Our method is inspired by the concept of contrast sets: given an original example with a corresponding label, we create a minimally perturbed example where the perturbation changes the corresponding
Figure 3: The  $VQ^2$  pipeline: (a) given a text and an image, (b) generate question and answer pairs from the text, (c) re-write each pair as a yes-no question and obtain the 'yes' answer probability from the image as an alignment score. (d) Finally, average all alignment pair scores as the final  $VQ^2$  score.
label [31-34]. Contrast sets address the issue of supervised models exploiting data artifacts in i.i.d. train/test splits to achieve high test scores, while their performance degrades significantly on samples outside their training distribution.
To create contrast sets for image-text alignment, we go over the text captions from the image-text pairs in the COCO and PickaPic datasets, covering both natural and synthetic images. For each caption we instruct PaLM [35], a large language model, to generate several contradicting captions via few-shot inference with 7 positive and 8 negative examples. For instance, for the caption "a knife sitting next to carrots on top of a cutting board", the model replaces the word knife with spoon (see Fig. 2, left). We then use a Natural Language Inference (NLI) model [18] to score whether the generated caption is indeed contradicting the original, and select the generated caption with the highest contradiction score. Figure 2 illustrates this process. Human annotators verified that the resulting contradicting captions are of high quality, with  $94\%$  agreement with human labels in COCO and  $77\%$  agreement with human labels in PickaPic (more details in section 2.2).
Using our SeeTRUE benchmark, we would like to reassess the performance of multimodal alignment approaches. In this section we introduce two image-text alignment methods. In Section 4 we will compare their performance against established, previously published methods.
3.1  $VQ^2$ : Zero-Shot Alignment via Question Generation and Visual Question Answering
Inspired by recent work on factual consistency evaluation in text-to-text tasks [36, 18, 37], we propose a zero-shot approach for automatically evaluating image-text alignment based on question generation and question answering. Figure 3 provides an overview of the method. The motivation is to extract question-answer pairs, which capture the important details of the text, and then to validate that they are presented correctly in the image. For a given image-text pair  $\{I,T\}$ , we first extract a set of candidate answer spans  $\{a_j\}_{j=1}^N$  from the given text  $T$ . Then, we use a question generation (QG) model to generate a question for each answer candidate  $q_j = QG(a_j,T)$ . Each generated question-answer pair  $(q_j,a_j)$  is scored with a question answering (QA) model, and if  $QA(q_j,a_j,T)$  returns a low score, we filter out the corresponding pair. This results in a subset of  $M$  question-answer pairs  $\{(q_j,a_j)\}_{j=1}^M$ .
Each generated question-answer pair  $(q_j, a_j)$  is then independently validated based on the image  $I$  using a visual question answering (VQA) model, obtaining an answer alignment score  $s_j = VQA(q_j, a_j, I)$  (more details on how this score is computed are given in 3.1). The overall alignment score for a image-text pair, denoted as the  $VQ^2$  score, is the average over all  $s_j$  scores for all the generated  $(q_j, a_j)$  pairs. We next describe each step in more detail.
Generating question-answer pairs. We follow the  $VQ^2A$  method [17] to generate question and answer pairs given an image caption in three steps. The purpose is to generate high quality question-answer pairs, which capture the most important details of the text. First, answer spans are extracted from text  $T$  using SpaCy [38], based on Part-of-Speech (POS) and dependency parse tree annotations. Then, for each answer span, a question  $q_j$  is generated given the answer span and the full caption as input using a T5-XXL model fine-tuned on SQuAD1.1 [39]. Finally, each candidate question-answer pair  $(q_j, a_j)$  is validated by answering  $q_j$  on  $T$  using a QA model, which is trained by fine tuning a
T5-XXL model on SQuAD2.0 [40] and Natural Questions [41]. Finally, we match the output answer  $a_{j}^{\prime}$  to the expected answer  $a_{j}$  using token-level F1 comparison. As suggested in [17], if the answer comparison F1 score is lower than 0.54 the question-answer pair is filtered out.
Assessing question-answer pair alignment against the image. To determine if the information conveyed by the text  $T$  is presented correctly in the image  $I$ , we use a VQA model based on PaLI-17B [42] as follows. We reformulate each question and answer candidate pair  $(q_j, a_j)$  into a new yes-no predicate question  $q_j'$  using the format "is  $\{a_j\}$  true for  $\{q_j\}$  in this image?". For example, for the text "two girls are sitting on some grass", and the automatically induced question-answer pair {"what are the girls sitting on?", "some grass"}, the reformulated question is "is on some grass true for what are the girls sitting on? in this image?". The VQA model is then invoked to answer the predicate question  $(q_j')$  over image  $I$ . We define the alignment score  $s_j$  as the probability of the model for answering "yes". We note that we also experimented with other answer alignment methods, e.g. ones that directly ask the generated question without formulating it as a yes/no question. However, the yes-no approach worked best. More details can be found in appendix A.4.
3.2 End-to-end VNLI Models
Another approach is to train end-to-end Visual NLI models (VNLI) that receive an image and text as input, and directly predict an alignment score. We do so by fine-tuning multimodal pretrained models while formatting the examples as yes/no questions using the prompt: "Does this image entail the description: {text}?", followed by a binary "yes" or "no" answer. In inference time we measure the probabilities of predicting "yes" or "no", and use the relative ratio between the two as the alignment score. Specifically, we finetune BLIP2 [7] and PaLI-17B [42] using a dataset comprising 110K text-image pairs labeled with alignment annotations. This includes 44K examples from COCO-Con, 3.5K from PickaPic-Con, 20K from COCO t2i and 40K from the training split of the SNLI-VE dataset. We generate COCO-Con and COCO t2i based on the COCO train split and PickaPic-Con with a distinct set of images, to ensure that there is no overlap with samples in the SeeTRUE benchmark. More technical details and training hyperparameters are described in appendix A.7.
4.1 Models and Metrics
We evaluate  $VQ^2$  and fine-tuned VNLI models based on PaLI and BLIP2 (Section 3) against several state-of-the-art multimodal models: (a) CLIP [15] and two larger versions - CLIP RN50x64 and CLIP ViT-L 14 [43], (b) CoCa [22], (c) BLIP Large [6], (d) BLIP2 FlanT5-XXL [7], and (e) OFA Large [23], and (f) TIFA [44]. First five models were typically trained with either a contrastive objective or an image-text matching objective that samples positive or negative caption-label pairs. TIFA, like  $VQ^2$ , employs a VQA model with generated question-answer pairs. However, TIFA contrasts textual and visual answer candidates provided by the model, while our method checks if the textual answer is accurate given the image.
We assess each method's ability to detect misalignments in each dataset in SeeTRUE. We use a binary labeling scheme and report the Area Under the ROC Curve (ROC AUC) for each method. For Winoground, we use existing metrics: (1) text score: accuracy in selecting the right caption for an image; (2) image score: accuracy in choosing the correct image given a caption; (3) group score: accuracy requiring all four image-caption pairs to be correct for a successful example.
We present our main results in Table 2. Notably, our  $\mathrm{VQ}^2$  approach excels as the top-performing zero-shot model across all datasets, surpassing other zero-shot baselines and even outperforming most of the fine-tuned models while achieving the highest score on the challenging Winoground dataset. This shows the robustness of the  $\mathrm{VQ}^2$  approach, which decomposes the alignment decision by generating multiple yes/no verification questions.
When looking at finetuned models, the PaLI variant finetuned on all the available datasets outperforms all the rest with an average score of 82.9, achieving the best results on 3 out of 7 datasets. The
Table 2: Main Results on SeeTRUE, split into zero-shot and end-to-end fine-tuned methods across the real and synthetic image-text eval-sets. The numbers in the table are ROC AUC. Note that TIFA and  $VQ^2$  require higher computational cost. They generate question-answer pairs and use a VLM model, while the other models use the image-text directly.
Text & Images	Real + Real	Real + Synthetic	Synthetic + Real	Synthetic + Synthetic	Avg.
Model	SNLI-VE	Winoground	DrawBench	EditBench	COCO t2i	COCO-Con	PickaPic-Con
zero-shot	CLIP RN50x64	66.6	53.6	59.2	67.1	58.8	71.1	66.8	63.3
CLIP ViT-L14	65.8	53.3	60.5	62.1	58.8	70.7	66.8	62.6
COCA ViT-L14	68.5	53.1	67.4	66.3	62.1	74.2	68.1	65.7
COCA ViT-L14 (f.t on COCO)	70	53.1	66.2	68.3	66.2	76.5	67.2	66.8
BLIP	75.2	58.2	60.5	68	70.7	84.2	76.6	70.5
BLIP2	76.4	56.9	58.5	67.5	66.9	84.3	76.9	69.6
BLIP 2 (f.t. COCO)	75.9	60	65.7	70	73.3	85.8	78	72.7
PaLI	65.4	53.6	60.2	56.7	53.3	65.5	60.5	59.3
TIFA	-	58.0	73.4	67.8	72.0	-	-	-
VQ2 (Ours)	88.0	63.5	82.6	73.6	83.4	87.1	81.7	80.0
f.t. snl-ve	OFA Large (470M)	80.5	53.3	77.6	70.9	67.5	75.4	69.5	70.7
BLIP2 (12B)	82.3	58.5	64.3	58.7	60.5	82.6	66.9	67.7
PaLI (17B)	95.1	61.7	82.8	65.5	77.7	91.2	83.7	79.7
PaLI + Synthetic Data	94.2	61.8	86.8	77.2	83.2	91	85.9	82.9
Avg(VQ2, PaLI+Syn)	93.9	63.5	87.8	78.4	85.1	93	87.3	84.1
Table 3: Results on the Winoground dataset, reporting text score, image score, and group score.
Model	Text Score	Image Score	Group Score
VQ2 (Ours)	47.00	42.20	30.50
PaLI (ft SNLI-VE + Synthetic Data)	46.5	38	28.75
PaLI (ft SNLI-VE)	45.00	41.50	28.70
BLIP2 (f.t. COCO)	44.00	26.00	23.50
IAISlarge [45]	42.50	19.75	16.00
VinVL [24]	37.75	17.75	14.50
TIFA	19.00	12.50	11.30
CLIP RN50x64	26.50	13.75	10.25
OFA Large (f.t. SNLI-VE)	27.70	14.30	9.00
COCA ViT-L14 (f.t on COCO)	28.25	11.50	8.25
Random Chance [24]	25.00	25.00	16.67
Humans [24]	89.50	88.50	85.50