What You See is What You Read? Improving Text-Image Alignment Evaluation
Authors: Michal Yarom
Yonatan Bitton
Soravit Changpinyo
Roee Aharoni
Jonathan Herzig
Oran Lang
Eran Ofek
Idan Szpektor
Venue: NeurIPS 2023 poster (2023)
Keywords: Vision-and-language
Image-text alignment
Text-to-image generation
Image-to-text generation
Multi-modal models
Synthetic images
Meta-evaluation
Visual-question-answering
TL;DR: Introducing SeeTRUE, an image-text alignment benchmark; we showcase two methods using question generation, visual QA, and classification, excelling in alignment tasks and enabling text-to-image reordering.

Abstract: Automatically determining whether a text and a corresponding image are semantically aligned is a significant challenge for vision-language models, with applications in generative text-to-image and image-to-text tasks. In this work, we study methods for automatic text-image alignment evaluation. We first introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets from both text-to-image and image-to-text generation tasks, with human judgements for whether a given text-image pair is semantically aligned. We then describe two automatic methods to determine alignment: the first involving a pipeline based on question generation and visual question answering models, and the second employing an end-to-end classification approach by finetuning multimodal pretrained models. Both methods surpass prior approaches in various text-image alignment tasks, with significant improvements in challenging cases that involve complex composition or unnatural images. Finally, we demonstrate how our approaches can localize specific misalignments between an image and a given text, and how they can be used to automatically re-rank candidates in text-to-image generation.

What You See is What You Read? Improving Text-Image Alignment Evaluation
Michal Yarom G
Yonatan Bitton H.G-
H,G+
Soravit Changpinyo
Roee Aharoni
Jonathan Herzig
Oran Lang
Eran Ofek
Idan Szpektor
GGoogle Research
Thee Hebrew University of Jerusalem
mi michalyyarom@goolle com, yonatan. bitton@mail. www.nnmmml..ui.......m ac. ac. i1
Abstract
Automatically determining whether text and corresponding image tically aligned is significant challenge for vision-language models, with appli- cations in generative text-to-image and image-to-text tasks. In this work, we study methods for automatic text-image alignment evaluation. We first introduce SecTRUE: comprehensive evaluation set, spanning multiple datasets from both text-to-image and image-to-text generation tasks, with human judgements for whether given text-image pair is semantically aligned. We then describe automatic methods to determine alignment: the first involving a pipeline based on question generation and visual question answering models, and the second em- ploying an end-to-end classification approach by finetuning multimodal pretrained models. Both methods surpass prior approaches various rext-image alignment tasks, with significant improvements in challenging cases that involve complex composition or unnatural images. Finally, we demonstrate how our approaches can localize specific misalignments between image and given text, and how they can be used to automatically re-rank candidates in text-to-image generation
SeeTRUE
VQ Zero-Shot evaluation
Real Text
Synthetic Text
A dog C. Who ranting on the chair on the
An SUV and man
The car resting the the
Real Images
directions.
house:
chasing
The dog is resting on the chair on the porch the
Prt
is going direction him.
VQA
A yellow and bux crulting through the deser:.
End to End
Synthetic Images
big boot is going down
The zait is resting on the
yellow and ock bus crussigg through the rainforest.
Sentuer
VNL
The dog resting on the chair the porch the
Figure 1: Overview of our approach to text-image alignment evaluation using SeeTRUE. We curate diverse pairs of real and synthetic text and images and use automatic contradiction generation and human evaluation to create a benchmark dataset. We propose two methods for text-image alignment evaluation: VQ² and VNLI, demonstrated with example pairs.
"Equal contribution. Yonatan participated this work as part of an internship Google Rescarch. Data and code are attached to this submission.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
The recent success and proliferation of multi-modal large language models (LLMs) for text-to-image and image-to-tex, generation [1H8] make such technology increasingly useful for wide range of creative applications. However, such models still struggle in generating semantically-aligeed image- text pairs; in text-to-image generation, models do not cope with complex specifications [9. or fail to map words in the prompt to visual entities [2]. In image captioning, object hallucination is a long-standing challenge
13 with generated captions still being inferior to human- wrriten ones [14)
Given the above, the task of automatically determining whether given text-image pair is semantically aligned is highly important, as it is useful both for evaluating and for improving text-to-image and image-to-fext models. However, existing evaluation approaches are still far from ideal; common methods like CLIP [15] or BLIP 6 7 are based on encoding the image and text as fixed-size embeddings, making it hard to model complex semantics [16]. In addition, while the task is relevant both to text-to-image and image-to-text generation, it is usually studied in silo while considering only one of the applications, thus impeding progress.
In this work, we promote a comprehensive approach to evaluating image- text alignment. We introduce TRE,, a diverse evaluation suite which includes wide range of image-tex pairs with human judgments that determine if the image and text are semantically aligned. SeeTTRUF encompasses both real and synthetic images and text, allowing the assessment of text-image alignment models' generalization capabilities across various tasks and 31,855 labeled examples from diverse sources. As part of constructing SecTRUE, we also introduce novel method for generating contradicting captions from existing ones by prompting a large language model with tailored instructions.
We present two approaches for automatic image-text alignment evaluation. The first, vQ utilizes question generation and visual question answering by generating questions related to the text [17] and ensuring that the correct answer is obtained when asking these questions with the provided image. The second method, Visual Entailment" (VNLI), involves directly finetuning a large pretrained multimodal model predict if given image-text pair is semantically aligned. Both strategies inspired by recent studies on evaluating factual consistency hetween two texts [8 21
We conduct comprehensive experiments on SecTRUE, demonstrating that both our VQ² and VNLI methods outperform a wide range of strong baselines, including various versions of CLIP [IS) 66 71. and OFA (23]. While previous work showed that risionaand-language
COCA [22], BLIP
models tend to exhibit sub-optimal "bag-of-words" behavior [16]. the VQ² method particularly excels on datasets with compositional challenges, achieving state-of-the of-te- results on the Winoground dataset [24] e.g. by improving the group score from 16% to 30.5%. Our methods also demonstrate improved performance when evaluating synthetic images (c.g. on DrawBende [5] and EditBench (55). Finally, we showease how VQ can identify specific sources of misalignment for given text-image pair and how our methods re-rank generated image candidates for given prompt.
To summarize, our contributions are as follows: (1) We introduce the SeeTRUE benchmark for meta- evaluation of image-text alignment. (2) We introduce novel method to generate contradicting image captions from given captions with LLMs. (3) We suggest two reference-free metrics for image-text alignment evaluation: VQ based question generation and visual question answering, and VNLI, based on fine-tuning large multimodal language models. (4) We conduct extensive evaluation of the above approaches against strong baselines, demonstrating superior performance over multiple datasets. (5) We release our evaluation suite, models and code to foster future work.
2 SeeTRUE: A Comprehensive Text-Image Alignment Benchmark
We begin by introducing TeEE, a diverse benchmark for meta- meta-evaluatton of image-text alignment methods, covering the 4-way combinations of real and synthetic text-and-ine -imddiiigeeeeee pairs. It addresses limitations in current benchmarks, which mainly focus natural images and often lack challeng- ing negative captions. SeeTRUE allows to better assess the generalization abilities of text-image alignment models across various tasks.
Defining how image-text alignment is assessed has a direct impact on the construction of evaluation datasets. As images can display more details than described in their caption or text prompt, we define
We use the terms Entailment and Natural Language Inference (NLJ) interchangeably.

---

Table SeeT a benchmark for image-tex text alignment encompassing 31,855 real and synthetic image-text pairs from diverse datasets and tasks. An example from each dataset is presented below.
Real Real Images
Real Synthetic Images
Synthetic Real
Synthetic
Syuthetic
Dutasel Name
SNLLVE
Winogroumd 1,600
Draw Bench
EdittBncc
COCO 2,586 63.6%
COCO Con
PickaPic Con
# Test Examples
17,901 33.3%
1,968 55.7%
3,827 36.9%
1,992 52.7%
1,981
% Positive/ / Total Labeled work?
50%
44.1%
Image
the heavy oncoming traffic is contrasted with the light outgoing
the player swings his
A blue cup and cell phone
pink candles and on top of
A person an show hoard high up the air.
girafte leaned over in plush field next some cows
wearing white
a
Text
the middle of street
cake.
Human Label
True
True
Fabse
True
Fabse
False
True
image-text alignment as the case where all the details described in the text are accurately represented within the image. Inspired by the Textual Entailment task [26] which judges for two pieces of text whether one (the "hypothesis") can be inferred given the other (the "premise"), our definition maps the image to the premise and the text to the hypothesis, resulting in the task of predicting whether the information in the text can be inferred from the given image.
2.1 Datasets
We describe the datasets included in our benchmark, with a high-level overview in Table I
Real text and real images. For pairs of human-written text and real (non-generated) images, we include the SNLI-VE (27] and Winoground [24] datasets. SNLI-VE is a widely adopted VNI.I dataset containing an image, text, and label of the alignment between the two entailment, contradiction, neutral. Winoground is challenging dataset for compositional understanding, where each example includes two images and two text captions, where the task is to match each text to its corresponding image. The captions only differ in few words, which should result in distinct visual interpretations. For example, "some plants surrounding a lightbulb" vs. "a lightbulb surrounding some plants".
Real text and synthetic images. For datasets that represent text-to-image generation tasks EditBench [25] which offers prompts and images generated by various text-to-image models given those prompts, accompanied by alignment ratings. To encourage more diversity in the data, also create new datasets by generating images using Stable Diffusion models B (V1.4 and V2.1) and Imagen [3] by prompting them with COCO (28 captions and text prompts from DrawBench [3], creating the COCO text-to-imt to--aage ("COCO t2i and the DrawBende text-to-image datasets.
Synthetic text and real images. This category includes new dataset which we name COCO-Con COCO-Con is generated using novel automatic method which we describe detail in Section 2.3 Specifically, we generate synthetic contradicting captions for COCO images based their original captions by prompting a large language model, and verify the resulting captions with human raters.
Synthetic text and synthetic images. We utilize PickaPic 29 source of user-generated and ranked synthetic images. We create synthetic captions using BLIP2 7 and employ our automatic contradiction generation method (Section 2.3 produce unaligned captions. This category evaluates synthetic text that is generated by image captioning models, e.g. for improving textual image search.
We note that some of the datasets are only used for testing (e.g.. Winoground, DrawBench, EditBench) while others include both training and test sets (e.g.. SNLI-VE, COCO 12i, COCO-Con, PickaPic- Con). This allows us to investigate different training configurations and their effect performance.

---

(a) See TRUE
knife sitting to carrots on top cutting board.
Goptoo
yellow and black bus cruising through the rainforest
sandwich next to yellow stuffed animal.
GonGen
ConGen
T21
Human Eval
Knife sitting next to carrots on top cutting board.
yellow and black bus cruising through the rainforest (ii)
(i)
(113)
(b) ConGen
Captions
the new viper sports car
@ dodge viper sports car driving down a road in the desert
sports driving on the trees and mountains the background
driving on a desert road
LLM: "Pravide a coption that contradicts all context captions by
minimal change
input
the new wew.vvieerppppp sports sLLvs>
dodge viger sports car driving
silver sports car driving on the street with trees and mountains in the background
driving desert
torst Musstoie the desert
hovsses
NL model: Select the generated caption with the lowest entailment score
Entailment
0.45
0.03
Score
Figure 2: (a) The SecTRUE generation process. (i) An image-text pair from dataset is used to generate a contradicting caption using ConGen. (ii) An image (real or synthetic) is passed through a captioning model to generate caption, which is then passed ConGen generate contradicting caption. (iii) A text-to-image model is applied on captions from the dataset to create multiple image- text pairs. All the resulting examples evaluated by human raters create TeET. (b) The contradiction generation process (ConGen) takes a caption as input and instructs an LLM to generate variants that contradict it. An NLI model used select the variant with the lowest entailment score.
2.2 Human Annotation and Evaluation
To standardize the labeling scheme across datasets, we follow TRUE 19 and use binary annotations for alignnent/misalggmeent In datasets with three-way annotations (e.g. Entailment, Contradiction, Neutral) we convert the labels to binary labels by collapsing all wwweentalmmee//oonaiiinneen labels to single negative label.
Some datasets, such COCO-Con and PickaPic-Con. start with automatically generated labels. while others lack annotations entirely (e.g. DrawBench). To make sure we have high quality labels we conduct human annotation for all test examples in such datasets. We ask three wookews from Amazon Mechanical Turk (AMT) to evaluate whether given image-text pair is aligned, by answering the question: "Does the image present all the details described in the text correctly?" with "Yes" or "No". If the answer is "No", the workers are also requested to describe the main misalignment to enhance the annotation quality. While the random chance of agreement is 25%, the annotators reached consensus in 80% of cases. Furthermore, we measured Fleiss- of 0.722, showing a good level agreement between the annotators. Full annotation details, AMT user interface example, and agreement numbers per dataset be found in appendix A.3
(30] score
The datasets we annotated include DrawBench, COCO 12i, COCO-Con and PickaPic-Con, with statistics presented in Table These datasets vary in their positive/negative distribution, with COCO 12i having the highest percentage of positives (63.6%) and DrawBench having the lowest (36.9%). The agreement with the auto-label is 94% for COCO-Con and 77% for PickaPic-Con. To prevent the inclusion of offensive images, particularly those that are synthetically generated, annotators are asked to mark any images that may be considered offensive and these were discarded.
2.3 ConGen: Generating Contradicting Captions by Prompting LLMs
We propose an automatic method for generating unaligned captions from existing, aligned image- and-text pairs, with the goal of creating challenging examples for evaluation and training. Our method is inspired by the concept of contrast sets: given an original example with a corresponding label, create minimally perturbed example where the perturbation changes the corresponding

---

black apple and green backpack.
High quality question-answer pairs (based on the text)
Validating the QA pairs (based on the image)
Final
Score
Besides green backpack, the picture?
black
0:155
VQ'A
apple
VQA
0.845
picture?
apple
VQ² score:
What green strap It?
badkpack
0.860
0.68
What color backpack?
0.938
(a)
(b)
[c]
(d)
Figure 3: The VQ² pipeline: (a) given a text and an image, (b) generate question and answer pairs from the text, (c) re-write each pair as yes-no question and obtain the 'yes' answer probability from the image as an alignment score. (d) Finally, average all alignment pair scores as the final VQ² score.
Contrast sets address the issue of supervised models exploiting data artifacts in i.i.d. train/test splits to achieve high test scores, while their performance degrades significantly on samples outside their training distribution.
label B1 34.
To create contrast sets for image-text alignment, go over the text captions from the image- text pairs in the COCO and PickaPic datasets, covering both natural and synthetic images. For each caption instruct PaLM [35] large language model, to generate several contradicting captions via few-shot inference with positive and 8 negative examples. For instance, for the caption "a knife sitting next to carrots top of a cutting board", the model replaces the word knife with spoon (see Fig. 2 left). We then use a Natural Language Inference (NLI) model [88] to score whether the generated caption is indeed contradicting the original, and select the generated caption with the highest contradiction score. Figure 2 resulting contradicting captions are of high quality, with 94% agreement with human labels in COCO and 77% agreement with human labels in PickaPic (more details in section 2.2
illustrates this process. Human annotators verified that the
3 Methods
Using our SecTRUE benchmark, we would like to reassess the performance of multimodal alignment approaches. In this section we introduce two image-tex alignment methods. In Section 4 we will compare their performance against established, previously published methods.
3.1
VQ²: Zero-Shot Alignment via Question Generation and Visual Question Answering
Inspired by recent work on factual consistency evaluation in text-to-text tasks [6
18 37
we propose
zero-shot approach for automatically evaluating image-text alignment based on question generation and question answering. question-answer pairs, which capture the important details of the text, and then validate that they are presented correctly in the image. For a given image-text pair {7, T), we first extract a of candidate answer spans {az} from the given text T. Then, we use a question generation (QG) model to generate a question for each answer candidate qj QG(a,, Each generated question-ansse answer pair (qj. aj) is scored with a question answering (QA) model, and if Q QA(gj, aj. T) returns low score, we filter out the corresponding pair. This results subset of M question-ande answer pairs/ (g) 0,) 13)
provides an overview of the method. The motivation is to extract
Each generated question-answer pair (gj. aj) is then independently validated based on the image I using visual question answering (VQA) model, obtaining an VQA(;, aj, I) (more details on how this score is computed are given in 1.1.. The overall alignment score for a image-text pair, denoted as the VQ² score, is the average over all sy scores for all the generated (qj, a,) pairs. We next describe each step in more detail.
alignment score sj
Generating question-answer pairs. We follow the VQ2A method 17 to generate question and answer pairs given an image caption in three steps. The purpose is to generate high quality question- answer pairs, which capture the most important details of the text. First, answer spans are extracted from text T' using SpaCy (88, based on Part-of-Speecl (POS) and dependency parse tree annotations. Then, for each answer span, question új is generated given the answer span and the full caption as input using a T5-XXI model fine-tuned on SQuAD1.1 (39. Finally, each candidate question-answer pair (g,, az) is validated by answering 9ú on T using a QA model, which is trained by fine tuning
5

---

T5-XXL model on SQuAD2.0 40 and Natural Questions 41. Finally, we match the output answer aj the expected answer oj using token-level Fl comparison. As suggested [7]. if the answer comparison Fl score is lower than 0.54 the question-answer pair is filtered out.
Assessing guesteon-answer pair alignment against the image. To determine if the information conveyed by the text T' is presented correctly in the image I, we use a VQA model based on PaLI- 17B [42] as follows. We reformulate each question and answer candidate pair (gg,az) into new yes-no predicate question of using the format "is {a,} true for {q;} } in this image?". For example. for the text "two girls are sitting on some grass", and the automatically induced question-answer pair { ("what are the girls sitting on?", "some grass"), ]. the reformulated question is "is on some grass true for what are the girls sitting on? in this image?". The VQA model is then invoked to answer the predicate question (g') over image I. We define the alignment score 8 as the probability of the model for answering "yes". We note that we also experimented with other answer alignment methods, e.g. ones that directly ask the generated question without formulating as a yes/no question. However, the yes-no approach worked best. More details can be found in appendix A.4
3.2 End-to-end VNLI Models
Another approach is to train end-to-end Visual NLI models (VNLI) that receive an image and text as input, and directly predict an alignment score. We do so by fine-tuning multimodal pretrained models while formatting the examples as yes/no questions using the prompt: "Does this image entail the description: [text]?", followed by a binary "yes" or "no" " answer. In inference time we measure the probabilities of predicting "yes" or "no", and use the relative ratio between the two as the alignment score. Specifically, we finetune BLIP2 7 and PaLI-17B 42 using a dataset comprising 110K text-image pairs labeled with alignment annotations. This includes 44K examples from COCO-Con, 3.5K from PickaPic-Con, 20K from COCO 12i and 40K from the training split of the SN dataset. We generate COCO-Con and COCO 12i based the COCO train split and PickaPic-Co with distinct set of images, to ensure that there is no overlap with samples in the SeeTRUE benchmark. More technical details and training hyperparameters described in appendix A.7
Experiments
4.1 Models and Metrics
We evaluate VQ2 and fine-tuned VNLI models based on PaLI and BLIP2 (Section 3 against several state-of-the tiee-of-thhearte multimodal models: (a) CLIP 15] and two larger versions CLIP RN50x64 and CLIP ViT-L 14 [43) (b) CoCa [22] (c) BLIP Large 6 (d) BLIP2 F 77. and (e) OFA Large 23 and (f) TIFA 44 First five models were typically trained with either a contrastive objective or an image-text matching objective that samples positive or negative caption-label pairs. TIFA, like VQ", employs a VQA model with generated question-answer pairs. However, TIFA contrasts textual and visual answer candidates provided by the model, while our method checks if the textual answer is accurate given the image.
We assess each method's 's ability detect misalignments in each dataset in SecTRUE. We use binary labeling scheme and report the Area Under the ROC Curve (ROC AUC) for each method. For Winoground,
existing metrics: (1) text score: accuracy in selecting the right caption for an image; (2) image score: accuracy in choosing the correct image given a caption; (3) group score: accuracy requiring all four image-caption pairs to be correct for successful example.
4.2 Results
We present our main results in Table Notably, our VQ² approach excels as the top- oop-eerfrmming zero-shot model across all datasets, surpassing other zero-shot baselines and even outperforming most of the fine-tuned models while achieving the highest score on the challenging Winoground dataset. This shows the robustness of the approach, which decomposes the alignment decision by generating multiple yes/no verification questions.
When looking at finetuned models, the PaI.I variant finetuned on all the available datasets outperforms all the rest with an average score of 82.9, achieving the best results on 3 out of 7 datascts. The

---

Table Main Results on SeeTRUE, split into zero-shot and end-to-end fine-tuned methods across the real and synthetic image-text eval-sets. The numbers in the table are ROC AUC. Note that TIFA and VQ2 require higher computational cost. They generate question-answer pairs and use VLM model, while the other models the image-tex directly.
Text & Images
Real Real
Real Syuthetie
Syathetic + Real
Synthctic Synthetic
Avg.
Model
Wineground
EditBench
COCO
COCO-Coa
Pickalse-Con
CLIP RN50:64
66.6
53.6
59.2
67.1
58.8
71.1
66.8
63.3
CLIP VIT-L14
65.8
53.3
60.5
62.1
58.8
70.7
66.8
62.6
COCA VIT-L14
COCA VIT-L14
2000-500
on (OCO)
70
53.1
66.2
68.3
66.2
76.5
67.2
66.8
BLIP
75.2
58.2
60.5
68
70.7
84.2
76.6
70.5
BLIP2
76.4
56.9
58.5
67.5
66.9
84.3
76.9
69.6
BLIP 2 COCO)
75.9
60
65.7
70
73.3
85.8
72.7
PALII
65.4
53.6
60.2
56.7
53.3
65.5
59.3
TIFA
58.0
73.4
67.8
72.0
VQ" (Ours)
sal--ve
OFA Large (470M)
80.5
53.3
77.6
70.9
67.5
75.4
69.5
70.7
BLJI2 (12R)
82.3
58.5
64.3
58.7
82.6
PALI (178)
95.1
61.7
82.8
65.5
77.7
91.2
83.7
79.7
Pall + Synthotic Data
94.2
61.8
86.8
77.2
83.2
91
85.9
82.9
Avg(VQ". PaLl+Syn)
93.9
63.5
87.8
78.4
85.1
87.3
84.1
Table 3: Results on the Winoground dataset, reporting text score, image score, and group score.
Model
Text Score
Imuge Score
Group Score
VQ (Ours)
47.00
42.20
30,50
PaL. (n SNLI-VE Synthetic Data)
38
28.75
Pal.1 (ft SNLL-VE)
45.00
41,50
28.70
BLIP2 (ft. COCO)
44.00
26.00
23,50
IAISlarge 45
VinVI. X4
37.75
17.75
14,50
TIFA
12.50
11.30
CLIP RN50x64
26.50
13.75
10.25
OFA Large (f... SNLI-VE)
27.70
14,30
9.00
COCA VIT-L.14 COCO)
28.25
11,50
8.25
Random
16.67
Humans
X4
89.50
88.50
85.50
SNLI-VE-only variant is behind with average score of 79.7, while achieving the highest scores for 2 out of 7 datasets. This shows that integrating synthetic training data leads to notable improvements on synthetic images on DrawBench (+4%), EditßBenc (+1 1.7%). COCO t2i (+5.5%), PickaPic-Con (+2.2%). Nevertheless, the inclusion of synthetic training data did not enhance performance on the COCO-Con dataset, comprised solely of natural images. This indicates that the variation in image types could be a contributing factor that calls for additional exploration. Notably, the last row shows a simple average between VQ² and our leading fine-tuned PaLI model, that produces higher performance, suggesting that they complement each other effectivvey.
(a) "the orange lollipop is and the red lollipop is surprised" Q: What is the orange lollipop feeling? A: sad
(b) "Somcone in stand- ing on a snowy hill"
(c) "A black apple green backpack"
Q: What is the person wearing?
Q: What color is the apple? A: black
A: blue hat
(1) Winoground
(2) CocoCon
DrawBench
Figure 4: Contradicting captions and the question/answer pairs with lower VQ2 alignment score,
indicating the contradiction reason.
7

---

DrawBench
COCO
00
66
Rea gpaags
Model
Model
Maan VQ"2 soore
VO
11.0
-55
Mean Pal. score
84
VG" R 0.90
Mean PaLl spore
PaLl
Pall
PALIR -
82
Pat
71.0
70
80--27
SD
50
500 Grgggnn
70.0
35
37
38
39
80
70
75
80
85
Image text alignment ratio
Image text alignment
Figure 5: Highly correlated VQ and PaI.I scores human rankings of text-to-image models Winoground Results. Shols: performance of the different methods on the challenging Winoground dataset, which requires strong visual reasoning and compositional understanding skills. Our zero-shot approach, VQ², achieves state-of-the-ee results on this dataset, surpassing other strong baselines, with group score of 30.5%. This again indicates that VQ approach that decomposes the alignment task into multiple question-ant answer pairs is promising path for image-text alignment.
Contradiction Generation. We assessed the VQ method's capacity to detect image-text contra- dictions, as shown in fig. + Contradiction generation relies on identifying the question anestion-answer pair with the lowest VQA score, signaling the least likely alignment between the image and the text. Three paper authors evaluated whether a particular contradiction (consisting of a question and an answer) accurately represents the primary discrepancy between the image and the text. The majority vote among the authors determined the final outcome, yielding the following accuracy rates; 88% for Coco-Con, 74% for DrawBench, and 80% for Winoground. This indicates that our method is capable of identifying these contradictions by investigating the structure and content the given caption and image. As result, method can achieve strong results, particularly datasets that require compositional understanding.
Comparing Generative Models. VQ2's ability to compare between generative mod- els is demonstrated the results of Draw- Bench and COCO-12i, which include gen- erated images from different models, to- gether with human quality ratings. fig. 5 shows that the VQ² and our fine-tuned PaLT ranking correlates very well with human ranking (R² > 0.92). In addition, since unlike human annotations, the VQ score consistent across datasets, it offers a way to evaluate dataset difficulty on an absolute scale.
Table 4: Comparison of human-labeled quality scores for top-ranked images with model breakdown
Dataset
Model
Random
CLIP
PaI.I
VQ²
SD 1.4
68.6
74.6
88.2
86.4
COCO
SD 2.1
71.3
81.2
84.5
87.3
SD 1.4
66.7
77.4
77.4
87.1
DrawBench
SD 2.1
59.0
78.0
87.0
82.0
Reranking Using Alignment Assessment. Alignment scores can also used for reranking can- didate generations, on top of evaluation. To demonstrate this, we re-rank the image candidate per prompt in the DrawBench and COCO-12i datasets. We do so using VQ and CLIP and measure human-labeled quality of the top-ranked image for each method. The results, presented in table 4 show that ranking with VQ² consistently achieves higher quality scores when compared to ranking with CLIP. One such example is shown in fig. where both VQ and our nop-performing fine-tuned PaLI model demonstrate superior ranking by placing the brown-and cats. This consistency between VQ² and PaLI highlights their alignment evaluation models' potential for enhancing sot-too-mmae systems, which contrasts with the divergent ranking exhibited by CLIP.
cats above the white-only

---

A brown and white cat is in suitcase
VQa score: 0.578
(1")
VQ' score: 0.845
(2*))
score: 0.731
(3*)
VO' score: 0.717
(4")
PoLi score: 0.992
(1")
Poli ft. score: 0,952 (2*9)
Poll 0,803 (34)
Paúl 0.437
(4")
similority 0.236 (4")
CLI similerity; 0.238 (3*)
CLIP similrity: 0.253 (19)
CLiP simiiority 0.25
(2*)
Figure 6: Four COCO-22 text-to-image model outputs ranked by VQSQR scores, correlating with top PaLI model. Image order and CLIP (RN50x64) similarity scores given, but not aligned with VQ2 ranks.
5
Related Work
Our work advances research in visual entailment (VE) 27 visual question answering (VQA) 46).
text-to-image alignment evaluation, and cross-task consistency for multi-modal models, with a focus on enhancing the semantic understanding of image-caption relationships.
Textual Entailment (TE) 26 evaluates the truthfulness of a textual hypothesis given a textual premise, providing a key benchmark for the semantic capabilities of neural network models [48
50 331. Recently TE has been adapted the multimodal domain Visual Entailment (VE) 227 to assess the semantic alignment between images and text. Vision-and-languaa models like CLIP 15
CoCa [22]. BLIP [6]. BLIP2 7 and OFA 23 often act as bag-of- oordsrds models, lacking deep comprehension of language compositionality [16]. Our approach addresses this by generating multiple questions probing diverse semantic aspects, thereby improving performance on challenging compositional tasks like Winoground (24) and unnatural images as in Drawbench [5]
Unlike DrawBench (5) and DALL-Eval [51 which depend human feedback and operate within a discrete set of alignments, our approach produces automated scores for broader range of text- image alignments, facilitating efficient evaluation of assion-and-lanuuaee models. Our approach also surpasses the recently proposed TIFA [44), which may be due to employing more question-answer pairs and tailored models for question generation and answering.
Several works have explored cross-task consistency in multi mottll models across various modalities. VQA studies have tackled inconsistencies and enhanced consistency using data augmentation and contrastive loss. NI.P researchers have improved consistency across tasks or within a single task by employing counterfactual instances or contrast sets 226 47]. Our research aligns with studies that evaluate natural text and images [52] ; however, extending the focus to synthetic images and texts, and
aligning with synthetic image understanding research (83)58) We introduce unique approaches to address the complexities of image-text alignment.
Another related effort is PickScore [29], which predicts human preferences for image quality and aesthetics by ranking or choosing between two images. In contrast, our methods independently score single image and focus specifically on image-text alignment.
Limitations
We recognize that in some cases, making a binary decision for whether a text and an image are aligned may be be difficult, also for humans. To tackle this limitation, provided human annotators with comprehensive guidelines, which resulted in a high inter-annotator agreement Pleiss-Kappa score of 0.722 with 80% of the
where all annotators agreed the entailment label).
Although many images in our datasets were obtained by others and not created by us, we made an effort to ensure that they do not contain harmful or potentially harmful content, such NSFW or biased imagery. During the annotation process, three individuals examined each image and indicated if it could be considered offensive. Additionally, two of the authors manually reviewed the images

---

for any harmful content. However, we understand that the perception of harmful or offensive content may vary among individuals and may be subject personal interpretation.
7 Conclusion
We addressed the task of image-texl alignment evaluation, which we find very close to the Visual Entailment (VNLI) task. We first introduced the SeeTRUE benchmark, which covers the mix of real and synthetic text and image pairs in text-to-image and image-to-text generation tasks, and includes challenging cases based on generated contradictions. We then proposed two methods, VQ and end-to-end VNLI, which outperform strong baselines on SeeTRUE and can serve as a starting point for future research the task.
In future work, we would like to employ our automatic evaluation models for guiding the training of text-to-image and image-to-text models towards more aligned outputs, following recent trends in text-to-text generation [59 66). For example, such models may be useful either for filtering training examples a reward when training models using reinforcement learning.
References
[1] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831 PMLR, 2021.
[2] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[3] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. 2022 ieee. CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 10674-10685, 2022.
[4] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10788 2022.
[5] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorcalistic text-to-image diffusion models with deep language understanding. arXiv preprint MrXiv:2205.11488 2022.
[6] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[7] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.125977 2023.
[8] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716-23736, 2022.
[9] Gary Mareus, Ernest Davis, and Scott Aaronson. very preliminary analysis of dall-e 2. arXiv preprint arXiv:2204. 13807, 2022.
[10] Nan Liu, Shuang Li, Yilun Du, Josh Tenenbaum, and Antonio Torralba. Learning to compose visual relations. Advances in Neural Information Processing Systems, 34:23166 13-23166 2021.
[11] Royi Rassin, Shauli Ravfogel, and Yoav Goldberg. Dalle-2 seeing double: flaws in word-to- concept mapping in text2image models. arXiv preprint arXiv:2210.1006 2022.
10

---

[12] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and- excite: Attention-basee based semantic guidance for eoxt-to-image diffusion models. arXiv preprint arXiv:2301. 13826, 2023.
[13] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 4035-4045, Brussels, Belgium, October- November 2018. Association for Computational Linguisties. doi: 10. 10.188533/1/D18-14333 14377 URL
attps:///aclanthoolle otg.ee8-og/D11--- o 1437
[14] Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, and Noah A. Smith. Transparent human evaluation for image captioning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3464-3478, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.
http:///2222...almmaaaa 254. URL
https://aclanthologgs org/ org/2022 2022. nww.a maiin. 254
[15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748 8763. PMLR, 2021.
[16] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference Learning Representations, 2022.
[17] Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All you may need for VQA are image captions. In NAACL, 2022.
[18] Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. Q²: Evaluating factual consistency in Noowledge-grondeer dialogues via question generation and question answering. arXiv preprint arXiv:2 18420220040 2021.
[19] Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. TRUE: Re- evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 3905-3920 Seattle, United States, July 2022. Association for Computational org/
Linguistics. doi: 18.18653/v 1/2022.naack. manellmmai.28777 287. URL
2022. www.aaalllla.mmmm - main. .87
[20] Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst, SummaC: Re-Visiting NLI-based Models for Inconsistency Detection Summarization. Transactions of the As- sociation for Computational Linguistics, 10:163-177, 02 2022. ISSN 2307-387X. doi:
10.1162/tac_ a 00453. URL //do:. 1ww.:0..11.2aaaac...
[21] Liyan Tang, Tanya Goyal, Alexander R Fabbri, Philippe Laban, Jiacheng Xu, Semih Yahvuz, Wojeiech Kryšcinski, Justin F Rousseau, and Greg Durrett. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. arXiv preprint arXiv:2205. 12854, 2022.
[22] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917. 2022.
[23] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai. Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying architectures, tasks, and modalities through simple
learning framework. In International Conference Machine PMLR, 2022.
Learning, pages
[24] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. ArXiv preprint, hbs//2044003162 2022. https::::: //arxiv. org. abs/
001622

---

[25] Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David Fleet, Radu Soricut, al. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. arXiv preprint arXiv: 2..i::2222.0000 069999, 2022.
[26] Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. Recognizing textual entailment: Rational, evaluation and srroacchss-rrattunn Natural Language Engineering, 16(1):105-10) 2010.
[27] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. arXiv preprini arXiv:1901.0677 2019.
[28] Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer ECsi 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[29] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset user preferences for text-to-image generation, 2023.
[30] Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378, 1971.
[31] Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models' local decision boundaries via contrast sets. arXiv preprint wrXiv:2004.02709 2020.
[32] Yonatan Bitton, Gabriel Stanovsky, Roy Schwartz, and Michael Elhadad. Automatic generation of contrast sets from scene graphs: Probing the compositional consistency of GQA. arXiv preprint arXiv:2103.0990 2021.
[33] Chuanrong Li, Lin Shengshuo, Zeyu Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert- Threlkeld. Singuisticalll-infoome transformations (LIT); A method for automatically generating contrast sets. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 126-135, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v http22...llabboooppp... 2020. 2ww.b.llccooxmlp1.....
URL httpss/ts:/// peattp:://clannthooogg. org/
[34] Shachar Rosenman, Alon Jacovi, and Yoav Goldberg. Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3702-3710 10. Online, November 2020. Association for Computational Linguistics. doi: URL https::/accaatthollgg.eoo// org 2220.2000.aomm emnlp-main 302
[35] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.025: 2022.
[36] Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to eval- uate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online, July 2020. Association for Computational Linguistics. doi: 10. 16653//1/0220.allmaai.. httts::/acllnttholoos wwwphhooogyooog/2//000 wwwp/020..002...../ acl main. 450
URL
[37] Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang. and Patrick Gallinari. QuestEval: Summarization asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18863/v 1/202 Lemnlp-main.522
URL https://
aclann sclanthology.org .rr. 2021, wttp:o//....mmmmmmmaaa..... emnlp-main 529
[38] Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
12

---

[39] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas, Novem- ber 2016. Association for Computational Linguistics. doi: 10. 10.18653/11/11- 1264. URL.
fttps:://aclanthologg. org/D16- 2eggDD16-264 1264
[40] Pranav Rajpurkar, Robin Jia, and Perey Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, July 2018, Association for Computational Linguistics. doi: 10. 10.18665////111--2222 URL
https://aclaantholegg. org. wwwg///--200 2124
[41] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slay Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, 2019. doi: 10.1162/tacl a 00276. URL https://aclantholle
org/ Q19-
[42] Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebas- tian Alexander Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLI: A jointly-scales multilingual language- immageegeiiaeeeeee model. In ICLR, 2023.
[43] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint ar arXiv:2010.1192: 2020.
[44] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang. Mari Ostendorf. Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable eext-to-imaee faithfulness evaluation with question answering. arXiv preprint arXiv:2303. 1897, 2023.
[45] Shuhuai Ren, Junyang Lin, Guangxiang Zhao, Rui Men, An Yang, Jingren Zhou, Xu Sun, and Hongxia Yang. Learning relation alignment for calibrated cross-modal retrieval. arXiv preprint arXiv:2105. 13868, 2021.
[46] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitniek, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[47] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1033. URL https://aclanttoolegs www.///////000.
[48]
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171 116616 Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10.18653/v1/N19. 1423. https://cca http://cclanthoooyy.ororrg
1423
[49] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), jan 2020.
[50] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton,
13

---

Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
[51] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases toxt-to-image generative transformers. arXiv preprint arXiv:2202.04053 2022.
[52] Adyasha Maharana, Amita Kamath, Christopher Clark, Mohit Bansal, and Aniruddha Kembhavi. Exposing and addressing cross- cross-task inconsistency in unified vision- lision-language models. arXiv preprint arXiv:2303. 2023.
[53] Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. arXiv preprint arXiv:22/2.100155 2022.
[54] Hang Li, Jindong Gu, Rajat Koner, Sahand Sharifzadeh, and Volker Tresp. Do dall-e and flamingo understand each other? arXiv preprint arXi::22/2.12249, 2022.
[55] Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! vision-and-languugg benchmark of synthetic and compositional images. arXiv preprint arXiv:2303.07274, 2023.
[56] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to- image models with human preference. arXiv preprint arXiv:2303. 14420, 2023.
[57] Ali Borji. Generated faces in the wild: Quantitative comparison of stable diffusion, midjourney and dall-e 2. arXiv preprint arXiv:2210.00580 2022.
[58) Andreas Stöckl. Evaluating synthetic image dataset generated with stable diffusion. arXiv preprint arXiv:2211.01777, 2022.
[59] Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. Increasing faithfulness in knowledge- grooudded dialogue with controllable features. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 704-718, Online, August 2021. Association for Computational Linguistics. doi: 10. 11653//11/222 lol- URL httpss://aclanachoooo...o. wwwp:oyyy.rr///1/o/////// acl-
[60] Roce Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. mFACE: Multilingual summarization with factual consistency evaluation, 2022.
[61] Norman P Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. A domain-specific supercomputer for training deep neural networks.
Communications of the ACM, 63(7):67-78, 2020.
[62] Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. Scaling up models and data with t5x and seqio, 2022. URL https://arxiv. org/abs/2203.17188
[63] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miine et al. Jax: composable transformations of python+ numpy programs. 2018
14

---

A Appendix
A.1 Dataset Supplementary Materials
1. Dataset documentation, metadata, and download instructions: anonymous
2. Intended uses: we hope SeeTRUE will be used by researchers to evaluate image-text matching models.
3. Author statement: We bear all responsibility in case of violation of right in using our benchmark.
4. Each dataset's license is described below. Our additional human annotations and geen- rated images are licensed under CC-BY 4.0 license httpp:://creat.vveccooo.ss org/ licenses/by/4. 0/ o/legalcolllll
5. Hosting & preservation: the dataset will be hosted in Huggingface Datasets, accessible and available for open research.
6. SecTRUE fields presented in table 5
Additional licensing details: do publish the datasets annotated in this work, and do re-publish the existing SNLI-VE and Winoground datasets. Full licenses:
1. MS COCO
28
2. EditBench
25
https:// wwW http:://ww.ppcch...gg//ooooo///. w
3. DrawBench 55: http:///w////ggo...errrroooo////
com/
gid=0
4. Fick-a-Pick 29
5. SNLI-VE 27
6. Winoground 24
Table 5: SeeTRUE Rows Examples
image
label
original dataset_ii
dataset_ source
imgl
A zebra to the right of fire hydrant.
text. 133 image 1228
drawbench
img?
A group of people standing next to hags of luggage.
text_ 105 105_image_11377 3786
coco_r2i
img3
a tiny figurine is surrounded by cell phones on a table.
editbench
A.2
Additional Examples
We added additional examples from (a) instances from SeeTRUE fig. 7 the VQ2 method fig.
to visualize the question-answer pairs; and (c) the ConGin technique fig. This will underscore the versatility of ConGen in handling both real and synthetic images, as well as the variety of contradictions it can generate.
A.3 Human Annotation Process
In order to provide reliable human labels for our datasets, we conducted an annotation process using the SecTRUE platform. The process comprised several steps, including setting qualification requirements, providing instructions, and evaluating annotator agreement.
We set the basic requirements for our annotation task as follows: above 98%, more than 5,000 approved HITss and annotator locations limited the US, UK, Australia, or New Zealand. We selected 5 examples from our dataset for qualification test and screened the annotators' results. fig. 10 displays sample of the Mechanical Turk user interface. The payment for the crowd-workers was 15-18 USD hourly.
percentage of approved assignments
15

---

image (inage)
label
original, datasett_tt (string)
dataset_ source (string)
(string)
(int64)
"A zebra to the right of fire hydrant.'
'text_133 tet___ne__mmgg_______ 1228"
"dzawbench"
"A group people standing next to bags of luggage."
1
Frext_155_imag__ie
"coco_t2i"
old lift overturned truck..
tries
0
Seatt_17__imag____1
"coco_t2i"
"Abraham Lincoln touches his while Geozge Washington..
0
"drawbench"
"A cat on the 1id of toilet looking pezturbed."
"text mrext_3_ingge_.
"coco _22
Figure 7: An example from SeeTRUE Huggingface Dataset.
A brown bird and a blue bear
Question
Answer (based on text)
VQA "yes" probability
VQ2 score
Aside from bear, what other animal
depicted in the picture?
brown bird
0.28
Aside from the brown bird, what other
blue bear
0.41
animal is
picture?
What is the blue animal?
bear
0.44
0.4
Aside from bear, what other animal
depicted?
bird
0.37
What color the bear?
blue
What color the bird in the picture?
brown
0.42
A wine glass top of a dog
Question
Answer (based on text)
VQA "yes" probability
VQ2 score
What is on top of dog?
Awin glass
0.82
wine glass is on top what animal?
dog
0.78
What the wine glass top of?
dog
0.77
What is on top of dog?
glass
0.74
0.78
On what part of the dog is the wine glass
0.64
located?
Where is the wine glass located?
of dog
0.88
What kind glass is on the dog?
wine
0.82
What top of dog?
glass top
0.79
Figure 8: Examples of the VQ2 method. The answers extracted from the text are named entities, nouns, and multi-word spans, such as adjectives and locations.
The instructions provided were as follows:
Evaluate the given image and text to determine if they match, selecting either "Yes" or "No". Some images may be synthetically generated by a text-to- sost-to-image model. To assess the match, mentally generate textual description for the image (no need to write it down) and compare this generated description to the given text. If the descriptions closely resemble each other, mark "Yes". If not, mark "No" " and provide feedback the specific issue causing the misalignment, focusing the primary issue if multiple misalignments are present. If you encounter an image or text that may be offensive due to bias, race, NSFW content, eic., mark the checkbox to indicate this issue.
Full agreement metrics are presented in table b As shown the table, the percentage of cases where all annotators agreed and the Fleiss- Fleiss-Kappa Kappa scores vary across the datasets, with COCO-Con
16

---

large white and pink rose planter surrounded by lush greenery large white and pink rose planter surrounded by dead greenery
red haired bowie is painted on the wall
blue haired bowie is painted on the wall
milk pouring into a coffee cup on a wooden table
milk pouring into a coffee cup on a plastic table
Figure 9: Examples from the ConGen method to generate contradictions. The top caption is the original, positive one, and the bottom one is the contradiction generated by ConGen.
a man with a mustache is sitting in front of some pine trees
Does the image present all the details described in the text correctly?
Yes
No
If No', what is the model feedback? (*main* misalignment)
Might be offensive
Figure 10: Annotation interface for determining whether given image-text pair are aligned.
exhibiting the highest level of agreement and Drawbench the lowest. These differences highlight the varying levels of complexity within the datasets.
Table 6: Agreement metrics for different datasets.
Dataset
Full
Drawbench
COCO 12
COCO-Con
PickaPic-Con
Items
8,527
1,968
2,586
1,992
1,981
% all agreed
80
76
78
86
77
Kapsa
0.72
0.66
0.68
0.81
0.69
A.4
Comparing VQ2 variants
VQ² consists of two main parts: generating question-answer pairs and assessing question-answer pair alignment against the image. We have experimented with different variants of the VQ² zero-shot method.
17

---

Assessing question-answer pair alignment methods Given a question-answer pair, we would like assess the question based on the image and compare the information the text. We experimented with several configurations for answer alignment:
1. Type A: Given a question-answer pair (gy az) generated from the text, we answer the question using VQA model and obtain answer based the image af VQA(gj, T)- We compare a pair of (yg, uz) with pair of (q, af) using an Natural Language Inference (NLI) model, where the pair based the text serves as the premise and the other as the hypothesis. We define the alignment score 8, as the probability of the NLI model for answering "entailed".
2. Type B: As done in type A, we answer the question using a VQA model and obtain the answer af VQ A(g;; I). We the VQA model again compare the the answers and determine whether they are the same. The question is formulated as "Is {az} = {af) in this image?". We define the alignment score sj as the probability of the VQA model for answering "yes".
3. Type C: We reformulate each question and answer candidate pair (gjz a,) into new yes-no predicate question of using the format "is {aj} true for {/r} in this image?". The VQA model is then invoked to answer the predicate question (1') over image I. We define the alignment score 8y the probability of the model for answering "yes".
Generating question-ansen wnnweer pairs To produce question-answa pairs from the text, we first extract informative spans in the text T. We extract as answer candidates all named entities and noun phrases T using spaCy. question- answer pairs to assess the alignment between the text and the image. Thus, extend the answer candidates by adding multi-word spans, such as adjectives ("black and white") and location "in the air". We use the extended answer candidate extraction in all of our experiments. The number of question-answer pairs is dependent on the length of the text T. In EditBench, the text contains 12.5 words average, and it results in average of 34.9 question-answer pairs with extended answer candidates.
noticed that for short text T, this method doesn't produce enough
Table 7 summarize the results of the VQ² variants the EditBench dataset. Our zero-shot VQ² method is VQ² type C, it outperforms other configurations and is more efficient, since it requires single run of the VQA model for the question-answer pair assessment.
Table 7: Comparing VQ² configurations all EditBench categories
Method
Models used for assessment
EditBench Material
Color
Count
Shape
Size
Mcan
VQ² (A)
VQA & NLI
75.7
63.6
71.9
67.6
75.9
70.9
VQa (B)
VQA & VQA
80.2
72.3
75.6
73.5
75.4
75.4
VQ² (C) w.o. multi-word answers
VQA
77.2
71.5
73.6
71.9
75.5
73.9
VQ2 (C)
VQA
78.5
73.5
76.9
71.7
78.2
75.8
A.5
Evaluating Contradictions Generated by GPT4
To assess GPT4's capability in generating contradictions, we compared the outputs from GPT4- driven ConGen approach against PaL---aseed method. We selected set of 100 images from both COCO-Con and PickaPic-Con databases. For each image, we obtained 100 positive and 100 negative captions using PaLM. Subsequently. by utilizing the identical prompt with GPT4 we generated an additional 100 negative captions. These captions were randomly assigned labels 'A' and 'B'.
For the evaluation process, we engaged three independent evaluators from Amazon Mechanical Turk. They were presented with an image, its caption, and the two negative caption candidates. Their task was to discern whether each candidate accurately contradicted the depicted image. A snapshot of this user interface can seen in fig.
The final judgment was based on the majority vote of the workers. The results showed a close match in performance between the two models: PaI.M achieved 77% while GPT4 reached 76%. This suggests our ConGen approach works effectively with various large-scale language models.
18

---

Cantradictiore giafle with is nodk oo around roci and coring moot.
Comedily coninatets trage Cltseerr conttacke the image
Contradiotion: gifffe its neck stretched
C Comoucly contradict Bl raage DoDwsn't contrack) the image
giratte
eating grass.
The Amazon Mechanical Turk user interface on determining whether a contradicting caption generated by GPT4 or PaLM is correct.
Figure 11:
A.6 Comparing VQ² with PaLI vs. BLIP2
We compare PaLI and BLIP2 for the VQ² method in table & While PaLI yields higher scores, it's pivotal to recognize their distinet evaluation metrics. PaLl's score indicates the probability of "yes" response, whereas BLIP2 gives definitive answer without probability. This difference can influence the application suitability of each model.
PaLI
BLIP2
Drawbench 82.6
60.5
Winoground 63.5
58.2
COCÓCON 87.1
88.5
Table 8: Comparison of VQ² results for BLIP2 and PaLI (ROC AUC scores). Note the difference in evaluation: PaLI's VQ² score represents the probability of "yes" response, while BLIP2 offers definitive answer without probability.
A.7 Reproducibility
To fine-tune BLIP2, we adjust only the Q-former parameters of the model using the Adam optimizer. We train the model for two epochs and designate 10% of the training set as a validation set early stopping and use learning rate selection between (le-5, 5e-5). A single training took 5 hours linux server with one A6000 GPU. All experiments took <2 days.
Zero-shot VQ²: For 10,000 text- imate pairs, the inference time of every step is as follows. Answer candidate generation: when using extended answer candidates about day. Otherwise, 12 hours. Question generation and filtering: When using extended answer candidates, about days. otherwise, 1 day. The last step only takes few minutes.
Aspect
PaI.I
VQ²
BLIP2
Inference Time
500ms per image-
40 seconds per image (full pipeline)
750ms per image-text pair (as measured source) 12B parameters
text pair 17B parameters
Model Parameters
T5-XXL 11B pa-
rameters + PaLI 17B
Hardware Requirements
Four v4 chips [61
16 TPU v3 cores 4
GPU with 24GB reported in HuggingFace Pytorch
v4 chips 61
Framework
T5X 63
62 on JAX
T5X
62
on JAX
63
Table 9: Computational Costs Summary
19


Reviews:
[{"review_id": "RPCvXu25kS", "reviewer": "Reviewer_j32y", "summary": "This paper introduces SeeTRUE, a benchmark for evaluating image-text alignment, encompassing a diverse range of both real and synthetic images and text. The authors proposes two innovative approaches to evaluating alignment: VQ2, which relies on question generation and visual question resolution, and VNLI, which relies on fine-tuning substantial multimodal language models. The suggested methods perform better on a variety of alignment tests than earlier methods, particularly in difficult scenarios involving complicated composition or strange images. The study also shows how these techniques may rerank generated image candidates and pinpoint particular misalignments. This provides alignment evaluation methods for image-to-text and text-to-image models.", "questions": "-Regarding the Dependency on the LLM, the methods largely depends on the quality of the LLM and language-and-vision model. Does this mean that the proposed methods would perform poorly if the model is not good enough? Can you provide a comparison of how these methods perform with different LLM and VLM models?\n\n-On the Novelty of Generating Contradicting Captions, Since the generation of contradicting captions is not a new concept, how does the proposed negative mining with LLM methods in this field? Is there an additional aspect of the method that brings unique contributions to the field of text-image alignment?\n\n-Can these methods be adapted for other tasks that involve the interaction of text and image", "limitations": "The author addressed the limitations of this work. The authors could discuss the limitation of prompting LLM in more detail, as well as suggest ways to mitigate it.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper introduces comprehensive benchmark, SeeTRUE covers real and synthetic text and image pairs from a variety of different tasks, allowing for a more thorough evaluation of text-image alignment models. This could be a solid contribution on this field.\n- The paper presents two new text-image alignment evaluation techniques, VQ2 and VNLI. these methods outperform previous ones, especially in complex or unnatural images.\n- The paper is clearly written, and easy to follow\n- The author shared part of their codes", "weaknesses": "- Because performance is dependent on the LLM, the comparison doesn't seem fair. \n- Also, as Generating Contradicting Captions (even with LLM) isn't a new concept, there isn't much insight to be gained from this paper.\n( for example, Momeni et al. https://arxiv.org/abs/2304.06708  )\n- There aren't many examples. Even the appendix is not sufficient. It would be better if there were more examples.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the contribution of the SeeTRUE benchmark to allow a more thorough evaluation of text-image alignment methods, and for recognizing our approaches as innovative. We will now address your comments and questions.\n\n### Dependency on LLM/VLM Quality + LLM Limitations [weakness 1 and question 1 + limitation]\n\nOur ConGen method does depend on an LLM, but with the increasing availability of robust LLMs like GPT4, Stable Beluga, Vicuna, Falcon, LLAMA2 etc. our methodology can be applied by others. \n\nBased on your feedback, we explored the capability of GPT-4 in producing Contradicting Captions. We conducted an evaluation on Amazon Mechanical Turk, engaging three annotators to ascertain if the contradiction candidates correctly contradicts the image. The results are on par with our original model, indicating that our ConGen method is compatible and effective with other leading language models. The full analysis is presented in Appendix A.5 in the revised paper. \n\nAs per VLM models, our results (Table 2) illustrate that the VNLI model based on PaLI excels over the BLIP-2-based model, where both are trained on the SNLI-VE dataset. Additionally, the quality of VNLI can be further improved with more extensive fine-tuning.\n\nWe also added the results of using VQ$^2$ with BLIP-2 instead of PaLI in Appendix A.6. Regarding the QG and QA models used in the V$Q^2$ method, in the Q$^2$ work (https://arxiv.org/abs/2104.08202) Table 8 presents results with T5-small instead of T5-XXL (11B) and the results are inferior.\n\n### Generating Contradiction Captions [weakness 2 and question 2]\n\nWhile prior works like BLIP2 (https://arxiv.org/abs/2301.12597) and ALIGN (https://arxiv.org/abs/2107.07651) employed hard negative mining at the embeddings layer using vector similarity, our method leverages LLMs.\n\nWe appreciate the reviewer pointing out the concurrent work. Notably, it was published in the same month as the NeurIPS submission deadline. While that study emphasizes video-text alignment using verb perturbations, our focus is broader, encompassing transformations of objects, relationships, and attributes. \n\nFinally, our approach extends to synthetic images created by text-to-image models, while previous works focused on natural images or videos.\n\n### Inclusion of More Examples [weakness 3]\n\nBased on your feedback, we've enriched Appendix A.2 with additional examples from (a) instances from SeeTRUE; (b) the VQ$^2$ method; and (c) the ConGen technique. This will underscore the versatility of ConGen in handling both real and synthetic images, as well as the variety of contradictions it can generate."}, {"review_id": "ykdksKpSF4", "reviewer": "Reviewer_prMg", "summary": "The authors propose a benchmark and two methods for evaluating fine-grained and complex image-text alignment. Their benchmark involves multiple distracter captions and images involving both real and synthetic images and captions. They propose two methods - one evaluates image-text alignment by asking multiple entailed questions, and another is a large Vision-Language model fine-tuned on their benchmark image-caption data.", "questions": "What are the train, val, and test splits of SeeTRUE? If I understand correctly, the VQ2 models are not trained and are just evaluated on the test split. However, the VNLI is trained on the train splits of SeeTRUE and then evaluated on the test splits. Is that correct?\n\nHow many questions are asked for a given image-text pair in the VQ2 method? ", "limitations": "Authors discuss limitations sufficiently", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Evaluating by visual entailment is neat since it is interpretable. \nThey also show that they outperform a contemporary benchmark TIFA that does the same. \nThey release a comprehensive benchmark, a combination of available data and annotations they collect, which will be valuable to the community for evaluating vision-language models. ", "weaknesses": "Sometimes certain design choices are not well motivated and make the reader wonder why such a choice was adopted. For example, when generating qa pairs, the question is answered using a t5 Language model first to filter some qa pairs. Is this to ensure the answer semantically makes sense for such a question in general? If so, writing out clear motivation before explaining what is being done might be good. \n\nThe benchmark seems to be a collection of simply more human-annotated image-text data. Motivating why sometimes synthetic images or text is required and what are new ways one can use this to evaluate their models (that current benchmarks lack) would be nice. For instance, with the contrastive captions, one can evaluate compositionality. However, we can already do this with Winoground; what extra does this benchmark give us? \n\nSome other relevant image-text benchmarks that evaluate compositionality using similar ideas of distractor captions are CREPE (https://arxiv.org/abs/2212.07796) and COLA (https://arxiv.org/abs/2305.03689). It might be good to add in a discussion of how this benchmark relates.  \n\n(minor) Even though synthetic images and texts are human evaluated and filtered, the datasets may be biased to having images that current generation models can already generate well, limiting the application of evaluating image generation models on the dataset on edge case prompts.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our approaches and the SeeTRUE benchmark as valuable to the community for evaluating vision-language models. We will address your comments and questions.\n\n### Clarity on $VQ^2$ Design Choices [weakness 1 and question 2]\n\nWe've elaborated on the motivations underlying $VQ^2$ in Section 3.1. Further, Appendix A.3 offers insights into other $VQ^2$ variants we explored.\n\nOur strategy for question-answer pair generation is based on established works like $Q^2$ and $VQ^2A$. We leverage a Question Generation (QG) model, and to refine the output, a Question Answering (QA) model is deployed against the text, thereby filtering irrelevant or low-quality questions.\n\n### Benchmark's Value as a Comprehensive Dataset [weakness 2&3 and question 1]\n\nOur benchmark's unique blend of natural & synthetic images and text differentiates it from Winoground, CREPE, and COLA. This composition permits more robust system evaluations. While Winoground adeptly assesses compositionality, our approach, with its contrasting captions, hones in on finer details such as color, objects, and image composition.\n\nNotably, advanced chatbots like Bing Chat now serve users by either sourcing existing images or generating new ones “on the fly”. This evolution underscores the importance of real text and prompts in contemporary evaluations. Furthermore, there's a shift towards synthetic/generated captions in academia, highlighted by studies like \"Improving Multimodal Datasets with Image Captioning\" (https://arxiv.org/abs/2307.10350). Such research showcases how machine-generated captions can even outperform human annotations, thus elevating performance across diverse tasks.\n\nOur SeeTRUE benchmark, particularly with synthetic datasets like DrawBench and EditBench, presents genuine challenges to image-text alignment methodologies. By using various text-to-image models, several data sources, and diverse prompts, we aim for a rich image-text example distribution. The inclusion of human-annotated pairs ensures our benchmark challenges text-to-image models with nuanced misalignments.\n\n### Details on Training & Testing [question 1]\n\nOur primary objective for SeeTRUE was to establish a high quality, diverse benchmark for image-text alignment. As you've rightly noted, while the $VQ^2$ method undergoes evaluation on the test split, we utilize SNLI-VE train and validation splits for the VNLI model. Our training set further benefits from additional data, detailed in section 3.2 and Appendix A.4."}, {"review_id": "z4WrqSUwmP", "reviewer": "Reviewer_xP6K", "summary": "This paper addresses the problem of text-image alignment evaluation by proposing a benchmark named SeeTRUE and two alignment metrics named VQ^2 and end-to-end VNLI. \n\nSeeTRUE covers a variety of real and synthetic images and captions. The synthetic captions, which are generated by large language models (LLMs), preserve most words from the original captions but convey a contradictory meaning. The synthetic images are generated by text-to-image generative models. The real image and captions are collected from existing datasets. In addition to the image-text pairs, SeeTRUE also provides binary human annotation of the alignment between these pairs.\n\nVQ^2 first generates question-answer pairs based on the caption and then assesses question-answer pair alignment against the image. End-to-end VNLI directly fine-tunes a visual NLI model to determine whether an image entails a caption.\n\nExperiments show that the proposed image-text alignment metrics outperform state-of-the-art (SOTA) vision-language models on the SeeTRUE benchmark. The VQ^2 metric also exhibits (1) a high correlation with humans in the evaluation of text-to-image generative models and (2) strong potential as a reranking technique to select generated images.\n", "questions": "N/A", "limitations": "Please refer to the “Weakness” section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed SeeTRUE overcomes two limitations found in existing benchmarks: a primary emphasis on real images and a lack of challenging negative captions. SeeTRUE provides a valuable testbed for evaluating image-text alignment methods.\n* The proposed two image-text alignment metrics are well-designed and exhibit very promising results, which can facilitate the evaluation and development of image captioning and text-to-image generation models.\n* The experiments are comprehensive and well-designed.\n* The paper is well-written and easy to follow.\n", "weaknesses": "The current manuscript still has three problems (which do not warrant rejection): \n* SeeTRUE primarily focuses on the challenging scenario where negative captions differ from positive captions by only a few words. However, it remains unclear whether the proposed VQ^2 and End-to-end VNLI models, specifically tailored for this setting, can still surpass the strong baselines in the standard scenario where negative captions describe completely different images.\n* Considering the utilization of three models—namely, an answer generation model (T5-XXL), a QA model (T5-XXL) and a VQA model (PaLI-17B)—VQ^2 might incur higher computational costs compared to the baseline methods. It would be beneficial to provide a detailed account of the computational expenses of these methods in Table 2.\n* It seems that the results in Table 7 of Appendix A3 are missing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that our methods can facilitate the evaluation and development of image captioning and text-to-image generation models. \n\n### Evaluation Scenarios with Different Negative Captions [weakness 1]\n\nSeeTRUE's diverse architecture encompasses seven distinct test sets. Three of them, namely Winoground, COCO-Con, and PickaPic-Con, operate on a \"few-words\" change dynamic. Conversely, the remaining datasets, such as SNLI-VE, DrawBench, EditBench, and COCO t2i, explore alternative evaluation angles. For instance, SNLI-VE extracts text descriptions from *entirely distinct images*, while COCO t2i offers a *broader semantic spectrum* due to its text-image generation capability. These variations ensure SeeTRUE’s capacity to evaluate alignment over a wide range of text perturbations, and not just simple lexical changes.\n\n### Computational Costs of $VQ^2$ vs. VNLI [weakness 2]\n\nIndeed, $VQ^2$’s sequential pipeline can present more computational overhead compared to simpler models like VNLI. However, this modular approach is advantageous for a few reasons:\n1. **Robustness**: As a zero-shot model, $VQ^2$ steers clear of shallow heuristics during training, ensuring a comprehensive evaluation.\n2. **Interpretability**: The $VQ^2$ model can effectively pinpoint the questions that majorly contribute to misalignments. This is evident in Figure 4.\n3. **Data Generation Capability**: $VQ^2$ doubles as a synthetic data generation tool. Typically employed offline, it can generate pertinent questions and answers for datasets consisting of images and texts. As we work on optimizing its latency, we're confident of further enhancing its efficiency.\n\nWe are continuously working to refine the computational efficiency of $VQ^2$, ensuring its alignment with real-world applications.\n\n### Table 7 appendix missing results [weakness 3]\n\nWe appreciate you pointing this out. We have since addressed and rectified this oversight.\n"}, {"review_id": "i2vRn9ECcy", "reviewer": "Reviewer_UX4d", "summary": "This paper aims to develop a method for evaluating the level of semantic alignment between text and image. In order to achieve this, the authors construct datasets of image and text pairs, and collect human judgments to determine if the text and image are semantically aligned. The paper proposes two methods for estimating alignment between text and image. The first method generates questions and answers from text and checks whether a Visual Question Answering (VQA) model provides consistent answers to the questions. The second method is an end-to-end classifier that predicts if the image entails the text. Experimental results demonstrate that the proposed methods outperform existing multimodal models.", "questions": "- Can the authors provide more details about the PaLM instruction used to generate contradicting captions? Was the prompt in Figure 2 utilized to rewrite captions?", "limitations": "The paper discusses two limitations:\n1. The difficulty in judging alignment between a text and an image even for humans.\n2. The challenge of filtering offensive content from the dataset.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A significant contribution of this paper is the collection of human judgments on text and image alignment. This will aid in the development and validation of alignment evaluation methods in future research.\n- The annotation pipeline is carefully designed, and the details are throughly described.", "weaknesses": "- The VQ^2 method depends on the sampling of questions and answers, which may result in output variation due to chance. The effects of sampling are not discussed in the paper.\n- Language models used to generate contradicting captions may introduce biases in the datasets. For example, generated captions might have typical wording patterns that are difficult for humans to detect. Validating synthesized datasets is challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the value of our human image-text judgments, that will aid in the development and validation of alignment evaluation methods. \n\n### Variability in $VQ^2$ Sampling [weakness 1]\n\nOur strategy for question-answer pair generation is based on established works like $Q^2$ and $VQ^2A$. We choose answer candidates by extracting noun phrases using the SpaCy toolkit so they are not random. For the QG we do beam-search, not sampling, but generating 5 questions per answer and filter out low quality question and answer pairs. \n\n### Validating contradictions generated by the LLM [weakness 2]\n\nFor our SeeTRUE test set, the labels are a product of a majority consensus among three annotators. This yielded a high inter-annotator agreement, as documented in Table 6. Such consistency among annotators signifies a shared understanding of the semantics inherent in the generated captions. We agree that generating the contradictions with different LLMs would reduce biases of a single LLM. In light of your and reviewer j32y feedback, we tested GPT4 for Generating Contradicting Captions, which yielded comparable results to our initial model. We believe that the combination of several LLMs and VLMs in the ConGen methodology reduces biases introduced by a single LLM.\n\nWhen considering the training set, we acknowledge that it contains a proportion of auto-labels. However, as depicted in Table 6, any noise introduced is minimal. More crucially, even when such noise is present, Table 2 shows that fine-tuning on this synthetic dataset leads to enhanced performance on SeeTRUE, which employs human-annotated labels.\n\n### PaLM instruction [question 1]\nYou're right. The prompt illustrated in Figure 2 is precisely what we deployed to produce contradictions. This was done by running the prompt with multiple few-shot examples, like the three presented in Figure 2. By altering specific elements (e.g., \"knife\" to \"spoon\" or \"rainforest\" to \"desert\"), we applied this to the five input captions using the LLM. This procedure is adaptable and can integrate other LLMs like GPT4 or Stable Beluga, among others. Once the contradictions were generated, we employed an NLI model to rank each one, selecting the “most contradicting” candidate which is identified by the lowest NLI score.\n"}]

Author rebuttal:
We appreciate the reviewers highlighting the comprehensiveness of our SeeTRUE benchmark and its potential contribution to the field (j32y). The noted effectiveness of our visual entailment evaluation and advancements in negative captions generation, particularly in addressing prior benchmark limitations, were recognized (prMg & xP6K). The emphasis on our significant human judgments collection (UX4d) reinforces our dedication to the text-image alignment domain. 

Below, we address each reviewer's comments and questions in detail.

Meta-review: This paper proposes a new benchmark for image-text alignment which provides more challenging negative examples than most prior work. To help ensure accurate labels, the authors also included human judgments that help alleviate some problems with ambiguous cases that can arise due to sparse labels.  The authors also present some results with unnatural images that a lot of prior work ignores.  The paper was evaluated by four reviewers that uniformly recommended acceptance, and the AC finds no reason to overturn their recommendation.
Decision: Accept (poster)