Title: What You See is What You Read? Improving Text-Image Alignment Evaluation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Vision-and-language

Image-text alignment

Text-to-image generation

Image-to-text generation

Multi-modal models

Synthetic images

Meta-evaluation

Visual-question-answering
Abstract: Automatically determining whether a text and a corresponding image are semantically aligned is a significant challenge for vision-language models, with applications in generative text-to-image and image-to-text tasks. In this work, we study methods for automatic text-image alignment evaluation. We first introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets from both text-to-image and image-to-text generation tasks, with human judgements for whether a given text-image pair is semantically aligned. We then describe two automatic methods to determine alignment: the first involving a pipeline based on question generation and visual question answering models, and the second employing an end-to-end classification approach by finetuning multimodal pretrained models. Both methods surpass prior approaches in various text-image alignment tasks, with significant improvements in challenging cases that involve complex composition or unnatural images. Finally, we demonstrate how our approaches can localize specific misalignments between an image and a given text, and how they can be used to automatically re-rank candidates in text-to-image generation.

What You See is What You Read? Improving Text-Image Alignment Evaluation

Abstract

Automatically determining whether a text and a corresponding image are semantically aligned is a significant challenge for vision-language models, with applications in generative text-to-image and image-to-text tasks. In this work, we study methods for automatic text-image alignment evaluation. We first introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets from both text-to-image and image-to-text generation tasks, with human judgements for whether a given text-image pair is semantically aligned. We then describe two automatic methods to determine alignment: the first involving a pipeline based on question generation and visual question answering models, and the second employing an end-to-end classification approach by finetuning multimodal pretrained models. Both methods surpass prior approaches in various text-image alignment tasks, with significant improvements in challenging cases that involve complex composition or unnatural images. Finally, we demonstrate how our approaches can localize specific misalignments between an image and a given text, and how they can be used to automatically re-rank candidates in text-to-image generation. 1

SeeTRUE

The cat is resting on the

chair on the porch of the

house.

Real

Images

a big boat is

going down

a small river

VQ 2 -  Zero-Shot evaluation

End to End

Q: Who is

resting on the

chair on the

The dog is resting

on the chair on

the porch of the

house.

VNLI

fine-tune

The dog is resting on the

chair on the porch of the

house.

An SUV and a man are

going in opposite

directions.

A man is chasing an SUV

that is going in the same

direction as him.

A yellow and black bus

cruising through the

rainforest.

A yellow and black bus

cruising through the

desert.

Synthetic

Images

Real Text

Synthetic Text

The cat is resting on the

chair on the porch of the

house.

The dog is resting on the

chair on the porch of the

house.

VQA

A: a dog

Pr(A|Q, I)

0.07

1 Introduction

The recent success and proliferation of multi-modal large language models (LLMs) for text-to-image and image-to-text generation [1-8] make such technology increasingly useful for a wide range of creative applications. However, such models still struggle in generating semantically-aligned imagetext pairs; in text-to-image generation, models do not cope with complex specifications [9, 10] or fail to map words in the prompt to visual entities [11, 12]. In image captioning, object hallucination is a long-standing challenge [13] with generated captions still being inferior to human-written ones [14].

Given the above, the task of automatically determining whether a given text-image pair is semantically aligned is highly important, as it is useful both for evaluating and for improving text-to-image and image-to-text models. However, existing evaluation approaches are still far from ideal; common methods like CLIP [15] or BLIP [6, 7] are based on encoding the image and text as fixed-size embeddings, making it hard to model complex semantics [16]. In addition, while the task is relevant both to text-to-image and image-to-text generation, it is usually studied in silo while considering only one of the applications, thus impeding progress.

In this work, we promote a comprehensive approach to evaluating image-text alignment. We introduce SeeTRUE, a diverse evaluation suite which includes a wide range of image-text pairs with human judgments that determine if the image and text are semantically aligned. SeeTRUE encompasses both real and synthetic images and text, allowing the assessment of text-image alignment models' generalization capabilities across various tasks and 31,855 labeled examples from diverse sources. As part of constructing SeeTRUE, we also introduce a novel method for generating contradicting captions from existing ones by prompting a large language model with tailored instructions.

We present two approaches for automatic image-text alignment evaluation. The first, VQ 2 , utilizes question generation and visual question answering by generating questions related to the text [17] and ensuring that the correct answer is obtained when asking these questions with the provided image. The second method, Visual Entailment 2 (VNLI), involves directly fine-tuning a large pretrained multimodal model to predict if a given image-text pair is semantically aligned. Both strategies are inspired by recent studies on evaluating factual consistency between two texts [18-21].

We conduct comprehensive experiments on SeeTRUE, demonstrating that both our VQ 2 and VNLI methods outperform a wide range of strong baselines, including various versions of CLIP [15], COCA [22], BLIP [6, 7], and OFA [23]. While previous work showed that vision-and-language models tend to exhibit sub-optimal 'bag-of-words' behavior [16], the VQ 2 method particularly excels on datasets with compositional challenges, achieving state-of-the-art results on the Winoground dataset [24] e.g. by improving the group score from 16% to 30.5%. Our methods also demonstrate improved performance when evaluating synthetic images (e.g. on DrawBench [5] and EditBench [25]). Finally, we showcase how VQ 2 can identify specific sources of misalignment for a given text-image pair and how our methods can re-rank generated image candidates for a given prompt.

To summarize, our contributions are as follows: (1) We introduce the SeeTRUE benchmark for metaevaluation of image-text alignment. (2) We introduce a novel method to generate contradicting image captions from given captions with LLMs. (3) We suggest two reference-free metrics for image-text alignment evaluation: VQ 2 , based on question generation and visual question answering, and VNLI, based on fine-tuning large multimodal language models. (4) We conduct extensive evaluation of the above approaches against strong baselines, demonstrating superior performance over multiple datasets. (5) We release our evaluation suite, models and code to foster future work.

2 SeeTRUE: A Comprehensive Text-Image Alignment Benchmark

We begin by introducing SeeTRUE, a diverse benchmark for meta-evaluation of image-text alignment methods, covering the 4-way combinations of real and synthetic text-and-image pairs. It addresses limitations in current benchmarks, which mainly focus on natural images and often lack challenging negative captions. SeeTRUE allows to better assess the generalization abilities of text-image alignment models across various tasks.

Defining how image-text alignment is assessed has a direct impact on the construction of evaluation datasets. As images can display more details than described in their caption or text prompt, we define

Real Text + Real Images

Real Text + Synthetic Images

Synthetic +

Real

Synthetic +

Synthetic

Dataset Name

SNLI-VE

DrawBench

EditBench

COCO t2i

COCO-Con

PickaPic-Con

# Test Examples

17,901

1,600

1,968

3,827

2,586

1,992

1,981

%Positive / Total

33.3%

50%

55.7%

36.9%

63.6%

52.7%

44.1%

Labeled in this work?

7

7

glyph[check]

glyph[check]

glyph[check]

glyph[check]

glyph[check]

Image

Text

the player

swings his bat

the heavy

oncoming

traffic is

contrasted

with the light

outgoing

traffic

A blue cup

and a green

cell phone

a few pink

candles and

some cream

on top of a

cake.

A person on a

snow board

high up in the

air.

A giraffe

leaned over in

a plush field

next to some

cows

a doctor

wearing a

white coat in

the middle of

a street

Human Label

True

True

False

True

False

False

True

image-text alignment as the case where all the details described in the text are accurately represented within the image. Inspired by the Textual Entailment task [26] which judges for two pieces of text whether one (the 'hypothesis') can be inferred given the other (the 'premise'), our definition maps the image to the premise and the text to the hypothesis, resulting in the task of predicting whether the information in the text can be inferred from the given image.

2.1 Datasets

We describe the datasets included in our benchmark, with a high-level overview in Table 1.

Real text and real images. For pairs of human-written text and real (non-generated) images, we include the SNLI-VE [27] and Winoground [24] datasets. SNLI-VE is a widely adopted VNLI dataset containing an image, a text, and a label of the alignment between the two - entailment, contradiction, or neutral. Winoground is a challenging dataset for compositional understanding, where each example includes two images and two text captions, where the task is to match each text to its corresponding image. The captions only differ in a few words, which should result in distinct visual interpretations. For example, 'some plants surrounding a lightbulb' vs. 'a lightbulb surrounding some plants'.

Real text and synthetic images. For datasets that represent text-to-image generation tasks we use EditBench [25] which offers prompts and images generated by various text-to-image models given those prompts, accompanied by alignment ratings. To encourage more diversity in the data, we also create new datasets by generating images using Stable Diffusion models [3] (V1.4 and V2.1) and Imagen [5] by prompting them with COCO [28] captions and text prompts from DrawBench [5], creating the COCO text-to-image ('COCO t2i') and the DrawBench text-to-image datasets.

Synthetic text and real images. This category includes a new dataset which we name COCO-Con. COCO-Con is generated using a novel automatic method which we describe in detail in Section 2.3. Specifically, we generate synthetic contradicting captions for COCO images based on their original captions by prompting a large language model, and verify the resulting captions with human raters.

Synthetic text and synthetic images. We utilize PickaPic [29], a source of user-generated and ranked synthetic images. We create synthetic captions using BLIP2 [7] and employ our automatic contradiction generation method (Section 2.3) to produce unaligned captions. This category evaluates synthetic text that is generated by image captioning models, e.g. for improving textual image search.

We note that some of the datasets are only used for testing (e.g., Winoground, DrawBench, EditBench) while others include both training and test sets (e.g., SNLI-VE, COCO t2i, COCO-Con, PickaPicCon). This allows us to investigate different training configurations and their effect on performance.

(a) SeeTRUE

A yellow and black

bus cruising through

the rainforest

T2I

ConGen

Caption

a silver sports car driving on the street with

trees and mountains in the background

LLM: 'Provide a caption that contradicts all context captions by

making a minimal change in the input caption'

A knife sitting next

to carrots on top of

a cutting board.

a dodge viper sports car driving

down a road in the desert

the new viper sports car

driving on a desert road

Captions

0.45

0.03

0.01

Entailment

Score

A knife sitting next

to carrots on top of

a cutting board.

spoon

A yellow and black

bus cruising through

the rainforest

desert

(b) ConGen

a silver sports car driving on the street with

trees and mountains in the background

a dodge viper sports car driving

down a road in the desert

the new viper sports car

driving on a desert road

silver

dirt

ford mustang

houses

NLI model: Select the generated caption with the lowest entailment score

A sandwich is

next to a yellow

stuffed animal.

ConGen

Human Eval

(

)

(i)

(ii)

(iii)

2.2 Human Annotation and Evaluation

To standardize the labeling scheme across datasets, we follow TRUE [19] and use binary annotations for alignment/misalignment. In datasets with three-way annotations (e.g. Entailment, Contradiction, Neutral) we convert the labels to binary labels by collapsing all non-entailment/non-alignment labels to a single negative label.

Some datasets, such as COCO-Con and PickaPic-Con, start with automatically generated labels, while others lack annotations entirely (e.g. DrawBench). To make sure we have high quality labels we conduct human annotation for all test examples in such datasets. We ask three crowd-workers from Amazon Mechanical Turk (AMT) to evaluate whether a given image-text pair is aligned, by answering the question: 'Does the image present all the details described in the text correctly?' with 'Yes' or 'No'. If the answer is 'No', the workers are also requested to describe the main misalignment to enhance the annotation quality. While the random chance of agreement is 25%, the annotators reached consensus in 80% of cases. Furthermore, we measured a Fleiss-Kappa [30] score of 0.722, showing a good level of agreement between the annotators. Full annotation details, AMT user interface example, and agreement numbers per dataset can be found in appendix A.3.

The datasets we annotated include DrawBench, COCO t2i, COCO-Con and PickaPic-Con, with statistics presented in Table 1. These datasets vary in their positive/negative distribution, with COCO t2i having the highest percentage of positives (63.6%) and DrawBench having the lowest (36.9%). The agreement with the auto-label is 94% for COCO-Con and 77% for PickaPic-Con. To prevent the inclusion of offensive images, particularly those that are synthetically generated, annotators are asked to mark any images that may be considered offensive and these were discarded.

2.3 ConGen: Generating Contradicting Captions by Prompting LLMs

We propose an automatic method for generating unaligned captions from existing, aligned imageand-text pairs, with the goal of creating challenging examples for evaluation and training. Our method is inspired by the concept of contrast sets: given an original example with a corresponding label, we create a minimally perturbed example where the perturbation changes the corresponding

A black apple and

a green backpack.

VQ

2

A

Besides a green backpack, what else is in

the picture?

High quality question-answer pairs

(based on the text)

A black

apple

VQA

VQ

2

score:

0.68

Validating the QA pairs

(based on the image)

What is the fruit in the picture?

apple

What item is green and has a strap across it?

backpack

What color is the backpack?

green

No

0.845

Yes

0.895

Yes

0.860

Yes

0.938

Yes

0.155

Final

Score

(a)

(b)

(c)

(d)

label [31-34]. Contrast sets address the issue of supervised models exploiting data artifacts in i.i.d. train/test splits to achieve high test scores, while their performance degrades significantly on samples outside their training distribution.

To create contrast sets for image-text alignment, we go over the text captions from the image-text pairs in the COCO and PickaPic datasets, covering both natural and synthetic images. For each caption we instruct PaLM [35], a large language model, to generate several contradicting captions via few-shot inference with 7 positive and 8 negative examples. For instance, for the caption ' a knife sitting next to carrots on top of a cutting board ', the model replaces the word knife with spoon (see Fig. 2, left). We then use a Natural Language Inference (NLI) model [18] to score whether the generated caption is indeed contradicting the original, and select the generated caption with the highest contradiction score. Figure 2 illustrates this process. Human annotators verified that the resulting contradicting captions are of high quality, with 94% agreement with human labels in COCO and 77% agreement with human labels in PickaPic (more details in section 2.2).

3 Methods

Using our SeeTRUE benchmark, we would like to reassess the performance of multimodal alignment approaches. In this section we introduce two image-text alignment methods. In Section 4 we will compare their performance against established, previously published methods.

3.1 V Q 2 : Zero-Shot Alignment via Question Generation and Visual Question Answering

Inspired by recent work on factual consistency evaluation in text-to-text tasks [36, 18, 37], we propose a zero-shot approach for automatically evaluating image-text alignment based on question generation and question answering. Figure 3 provides an overview of the method. The motivation is to extract question-answer pairs, which capture the important details of the text, and then to validate that they are presented correctly in the image. For a given image-text pair { I, T } , we first extract a set of candidate answer spans { a j } N j =1 from the given text T . Then, we use a question generation (QG) model to generate a question for each answer candidate q j = QG ( a j , T ) . Each generated question-answer pair ( q j , a j ) is scored with a question answering (QA) model, and if QA ( q j , a j , T ) returns a low score, we filter out the corresponding pair. This results in a subset of M question-answer pairs { ( q j , a j ) } j M =1 .

Each generated question-answer pair ( q j , a j ) is then independently validated based on the image I using a visual question answering (VQA) model, obtaining an answer alignment score s j = V QA ( q j , a j , I ) (more details on how this score is computed are given in 3.1). The overall alignment score for a image-text pair, denoted as the V Q 2 score, is the average over all s j scores for all the generated ( q j , a j ) pairs. We next describe each step in more detail.

Generating question-answer pairs. We follow the V Q 2 A method [17] to generate question and answer pairs given an image caption in three steps. The purpose is to generate high quality questionanswer pairs, which capture the most important details of the text. First, answer spans are extracted from text T using SpaCy [38], based on Part-of-Speech (POS) and dependency parse tree annotations. Then, for each answer span, a question q j is generated given the answer span and the full caption as input using a T5-XXL model fine-tuned on SQuAD1.1 [39]. Finally, each candidate question-answer pair ( q j , a j ) is validated by answering q j on T using a QA model, which is trained by fine tuning a

T5-XXL model on SQuAD2.0 [40] and Natural Questions [41]. Finally, we match the output answer a ′ j to the expected answer a j using token-level F1 comparison. As suggested in [17], if the answer comparison F1 score is lower than 0.54 the question-answer pair is filtered out.

Assessing question-answer pair alignment against the image. To determine if the information conveyed by the text T is presented correctly in the image I , we use a VQA model based on PaLI17B [42] as follows. We reformulate each question and answer candidate pair ( q j , a j ) into a new yes-no predicate question q ′ j using the format 'is { a j } true for { q j } in this image?' . For example, for the text 'two girls are sitting on some grass' , and the automatically induced question-answer pair { 'what are the girls sitting on?' , 'some grass' }, the reformulated question is 'is on some grass true for what are the girls sitting on? in this image?'. The VQA model is then invoked to answer the predicate question ( q ′ j ) over image I . We define the alignment score s j as the probability of the model for answering 'yes'. We note that we also experimented with other answer alignment methods, e.g. ones that directly ask the generated question without formulating it as a yes/no question. However, the yes-no approach worked best. More details can be found in appendix A.4.

3.2 End-to-end VNLI Models

Another approach is to train end-to-end Visual NLI models (VNLI) that receive an image and text as input, and directly predict an alignment score. We do so by fine-tuning multimodal pretrained models while formatting the examples as yes/no questions using the prompt: 'Does this image entail the description: {text}?', followed by a binary 'yes' or 'no' answer. In inference time we measure the probabilities of predicting 'yes' or 'no', and use the relative ratio between the two as the alignment score. Specifically, we finetune BLIP2 [7] and PaLI-17B [42] using a dataset comprising 110K text-image pairs labeled with alignment annotations. This includes 44K examples from COCO-Con, 3.5K from PickaPic-Con, 20K from COCO t2i and 40K from the training split of the SNLI-VE dataset. We generate COCO-Con and COCO t2i based on the COCO train split and PickaPic-Con with a distinct set of images, to ensure that there is no overlap with samples in the SeeTRUE benchmark. More technical details and training hyperparameters are described in appendix A.7.

4 Experiments

4.1 Models and Metrics

We evaluate V Q 2 and fine-tuned VNLI models based on PaLI and BLIP2 (Section 3) against several state-of-the-art multimodal models: (a) CLIP [15] and two larger versions - CLIP RN50x64 and CLIP ViT-L 14 [43], (b) CoCa [22], (c) BLIP Large [6], (d) BLIP2 FlanT5-XXL [7], and (e) OFA Large [23], and (f) TIFA [44]. First five models were typically trained with either a contrastive objective or an image-text matching objective that samples positive or negative caption-label pairs. TIFA, like VQ 2 , employs a VQA model with generated question-answer pairs. However, TIFA contrasts textual and visual answer candidates provided by the model, while our method checks if the textual answer is accurate given the image.

We assess each method's ability to detect misalignments in each dataset in SeeTRUE. We use a binary labeling scheme and report the Area Under the ROC Curve (ROC AUC) for each method. For Winoground, we use existing metrics: (1) text score : accuracy in selecting the right caption for an image; (2) image score : accuracy in choosing the correct image given a caption; (3) group score : accuracy requiring all four image-caption pairs to be correct for a successful example.

4.2 Results

We present our main results in Table 2. Notably, our V Q 2 approach excels as the top-performing zero-shot model across all datasets, surpassing other zero-shot baselines and even outperforming most of the fine-tuned models while achieving the highest score on the challenging Winoground dataset. This shows the robustness of the VQ 2 approach, which decomposes the alignment decision by generating multiple yes/no verification questions.

When looking at finetuned models, the PaLI variant finetuned on all the available datasets outperforms all the rest with an average score of 82.9, achieving the best results on 3 out of 7 datasets. The

00

(a) 'the orange lollipop is sad and

Winoground

(b) 'Someone in a blue hat standing on a snowy hill'

Q: What is the person wearing? A: blue hat (2) CocoCon

(c) 'A black apple and a green

Q: What color is the apple? A:

(3) DrawBench

35

36

37

38

39

Image-text alignment ratio

70.0

70.5

71.0

71.5

72.0

Mean VQ^2 score

Imagen

SD v1.4

SD v2.1

DrawBench

60

65

70

75

80

85

90

95

Image-text alignment ratio

78

80

82

84

86

Mean VQ^2 score

Imagen

SD v1.4

SD v2.1

Real Images

COCO t2i

50

51

52

53

54

55

Mean PaLI score

Model

VQ²

VQ² R² = 1.00

PaLI

PaLI R² = 0.92

65

70

75

80

Mean PaLI score

Model

VQ²

VQ² R² = 0.98

PaLI

PaLI R² = 0.95

Contradiction Generation. We assessed the VQ 2 method's capacity to detect image-text contradictions, as shown in fig. 4. Contradiction generation relies on identifying the question-answer pair with the lowest VQA score, signaling the least likely alignment between the image and the text. Three paper authors evaluated whether a particular contradiction (consisting of a question and an answer) accurately represents the primary discrepancy between the image and the text. The majority vote among the authors determined the final outcome, yielding the following accuracy rates: 88% for Coco-Con, 74% for DrawBench, and 80% for Winoground. This indicates that our method is capable of identifying these contradictions by investigating the structure and content of the given caption and image. As a result, our method can achieve strong results, particularly on datasets that require compositional understanding.

Comparing Generative Models. VQ 2 's ability to compare between generative models is demonstrated on the results of DrawBench and COCO-t2i, which include generated images from different models, together with human quality ratings. fig. 5 shows that the VQ 2 and our fine-tuned PaLI ranking correlates very well with human ranking ( R 2 > 0 . 92 ). In addition, since unlike human annotations, the VQ 2 score is consistent across datasets, it offers a way

to evaluate dataset difficulty on an absolute scale.

Reranking Using Alignment Assessment. Alignment scores can also be used for reranking candidate generations, on top of evaluation. To demonstrate this, we re-rank the image candidate per prompt in the DrawBench and COCO-t2i datasets. We do so using VQ 2 and CLIP and measure the human-labeled quality of the top-ranked image for each method. The results, presented in table 4, show that ranking with VQ 2 consistently achieves higher quality scores when compared to ranking with CLIP. One such example is shown in fig. 6, where both VQ 2 and our top-performing fine-tuned PaLI model demonstrate superior ranking by placing the brown-and-white cats above the white-only cats. This consistency between VQ 2 and PaLI highlights their alignment evaluation models' potential for enhancing text-to-image systems, which contrasts with the divergent ranking exhibited by CLIP.

A brown and white cat is in a suitcase

VQ 2 score: 0.878 (1 st ) PaLI ft. score: 0.992     (1 st ) CLIP similarity: 0.236   (4 th )

VQ 2 score: 0.731 (3 rd ) PaLI ft. score: 0.803     (3 rd ) CLIP similarity: 0.253   (1 st )

VQ

2

score: 0.717           (4

th

)

PaLI ft. score: 0.437     (4

th

)

CLIP similarity: 0.25     (2

nd

)

5 Related Work

Our work advances research in visual entailment (VE) [27], visual question answering (VQA) [46], text-to-image alignment evaluation, and cross-task consistency for multi-modal models, with a focus on enhancing the semantic understanding of image-caption relationships.

Textual Entailment (TE) [26, 47] evaluates the truthfulness of a textual hypothesis given a textual premise, providing a key benchmark for the semantic capabilities of neural network models [4850, 35]. Recently, TE has been adapted to the multimodal domain as Visual Entailment (VE) [27] to assess the semantic alignment between images and text. Vision-and-language models like CLIP [15], CoCa [22], BLIP [6], BLIP2 [7] and OFA [23] often act as bag-of-words models, lacking a deep comprehension of language compositionality [16]. Our approach addresses this by generating multiple questions probing diverse semantic aspects, thereby improving performance on challenging compositional tasks like Winoground [24] and unnatural images as in Drawbench [5].

Unlike DrawBench [5] and DALL-Eval [51] which depend on human feedback and operate within a discrete set of alignments, our approach produces automated scores for a broader range of textimage alignments, facilitating efficient evaluation of vision-and-language models. Our approach also surpasses the recently proposed TIFA [44], which may be due to employing more question-answer pairs and tailored models for question generation and answering.

Several works have explored cross-task consistency in multi-modal models across various modalities. VQA studies have tackled inconsistencies and enhanced consistency using data augmentation and contrastive loss. NLP researchers have improved consistency across tasks or within a single task by employing counterfactual instances or contrast sets [26, 47]. Our research aligns with studies that evaluate natural text and images [52]; however, extending the focus to synthetic images and texts, and aligning with synthetic image understanding research [53-58]. We introduce two unique approaches to address the complexities of image-text alignment.

Another related effort is PickScore [29], which predicts human preferences for image quality and aesthetics by ranking or choosing between two images. In contrast, our methods independently score a single image and focus specifically on image-text alignment.

6 Limitations

We recognize that in some cases, making a binary decision for whether a text and an image are aligned may be be difficult, also for humans. To tackle this limitation, we provided human annotators with comprehensive guidelines, which resulted in a high inter-annotator agreement (Fleiss-Kappa score of 0.722 with 80% of the cases where all annotators agreed on the entailment label).

Although many images in our datasets were obtained by others and not created by us, we made an effort to ensure that they do not contain harmful or potentially harmful content, such as NSFW or biased imagery. During the annotation process, three individuals examined each image and indicated if it could be considered offensive. Additionally, two of the authors manually reviewed the images

for any harmful content. However, we understand that the perception of harmful or offensive content may vary among individuals and may be subject to personal interpretation.

7 Conclusion

We addressed the task of image-text alignment evaluation, which we find very close to the Visual Entailment (VNLI) task. We first introduced the SeeTRUE benchmark, which covers the mix of real and synthetic text and image pairs in text-to-image and image-to-text generation tasks, and includes challenging cases based on generated contradictions. We then proposed two methods, VQ 2 and end-to-end VNLI, which outperform strong baselines on SeeTRUE and can serve as a starting point for future research on the task.

In future work, we would like to employ our automatic evaluation models for guiding the training of text-to-image and image-to-text models towards more aligned outputs, following recent trends in text-to-text generation [59, 60]. For example, such models may be useful either for filtering training examples or as a reward when training models using reinforcement learning.

References

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. 2022 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10674-10685, 2022.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789 , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 , 2022.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35: 23716-23736, 2022.

Gary Marcus, Ernest Davis, and Scott Aaronson. A very preliminary analysis of dall-e 2. arXiv preprint arXiv:2204.13807 , 2022.

Nan Liu, Shuang Li, Yilun Du, Josh Tenenbaum, and Antonio Torralba. Learning to compose visual relations. Advances in Neural Information Processing Systems , 34:23166-23178, 2021.

Royi Rassin, Shauli Ravfogel, and Yoav Goldberg. Dalle-2 is seeing double: flaws in word-toconcept mapping in text2image models. arXiv preprint arXiv:2210.10606 , 2022.

Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-andexcite: Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.13826 , 2023.

Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 4035-4045, Brussels, Belgium, OctoberNovember 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1437. URL https://aclanthology.org/D18-1437 .

Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, and Noah A. Smith. Transparent human evaluation for image captioning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 3464-3478, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.254. URL https://aclanthology.org/2022.naacl-main.254 .

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations , 2022.

Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All you may need for VQA are image captions. In NAACL , 2022.

Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. Q 2 : Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. arXiv preprint arXiv:2104.08202 , 2021.

Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. TRUE: Reevaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 3905-3920, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.287. URL https://aclanthology.org/ 2022.naacl-main.287 .

Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization. Transactions of the Association for Computational Linguistics , 10:163-177, 02 2022. ISSN 2307-387X. doi: 10.1162/tacl_a_00453. URL https://doi.org/10.1162/tacl_a_00453 .

Liyan Tang, Tanya Goyal, Alexander R Fabbri, Philippe Laban, Jiacheng Xu, Semih Yahvuz, Wojciech Kry´ sci´ nski, Justin F Rousseau, and Greg Durrett. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. arXiv preprint arXiv:2205.12854 , 2022.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning , pages 23318-23340. PMLR, 2022.

Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. ArXiv preprint , abs/2204.03162, 2022. URL https://arxiv.org/abs/ 2204.03162 .

Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. arXiv preprint arXiv:2212.06909 , 2022.

Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. Recognizing textual entailment: Rational, evaluation and approaches-erratum. Natural Language Engineering , 16(1):105-105, 2010.

Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706 , 2019.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023.

Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin , 76(5):378, 1971.

Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models' local decision boundaries via contrast sets. arXiv preprint arXiv:2004.02709 , 2020.

Yonatan Bitton, Gabriel Stanovsky, Roy Schwartz, and Michael Elhadad. Automatic generation of contrast sets from scene graphs: Probing the compositional consistency of GQA. arXiv preprint arXiv:2103.09591 , 2021.

Chuanrong Li, Lin Shengshuo, Zeyu Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert-Threlkeld. Linguistically-informed transformations (LIT): A method for automatically generating contrast sets. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP , pages 126-135, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.blackboxnlp-1.12. URL https://aclanthology.org/ 2020.blackboxnlp-1.12 .

Shachar Rosenman, Alon Jacovi, and Yoav Goldberg. Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 3702-3710, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.302. URL https://aclanthology.org/2020.emnlp-main.302 .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 5008-5020, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.450. URL https://aclanthology.org/2020.acl-main.450 .

Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. QuestEval: Summarization asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6594-6604, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.529. URL https:// aclanthology.org/2021.emnlp-main.529 .

Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1264. URL https://aclanthology.org/D16-1264 .

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-2124. URL https://aclanthology.org/P18-2124 .

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:452-466, 2019. doi: 10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026 .

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Alexander Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLI: A jointly-scaled multilingual language-image model. In ICLR , 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897 , 2023.

Shuhuai Ren, Junyang Lin, Guangxiang Zhao, Rui Men, An Yang, Jingren Zhou, Xu Sun, and Hongxia Yang. Learning relation alignment for calibrated cross-modal retrieval. arXiv preprint arXiv:2105.13868 , 2021.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425-2433, 2015.

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1075. URL https://aclanthology.org/D15-1075 .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423 .

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21(1), jan 2020.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton,

Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.

Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. arXiv preprint arXiv:2202.04053 , 2022.

Adyasha Maharana, Amita Kamath, Christopher Clark, Mohit Bansal, and Aniruddha Kembhavi. Exposing and addressing cross-task inconsistency in unified vision-language models. arXiv preprint arXiv:2303.16133 , 2023.

Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. arXiv preprint arXiv:2212.10015 , 2022.

Hang Li, Jindong Gu, Rajat Koner, Sahand Sharifzadeh, and Volker Tresp. Do dall-e and flamingo understand each other? arXiv preprint arXiv:2212.12249 , 2022.

Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images. arXiv preprint arXiv:2303.07274 , 2023.

Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-toimage models with human preference. arXiv preprint arXiv:2303.14420 , 2023.

Ali Borji. Generated faces in the wild: Quantitative comparison of stable diffusion, midjourney and dall-e 2. arXiv preprint arXiv:2210.00586 , 2022.

Andreas Stöckl. Evaluating a synthetic image dataset generated with stable diffusion. arXiv preprint arXiv:2211.01777 , 2022.

Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. Increasing faithfulness in knowledge-grounded dialogue with controllable features. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 704-718, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.58. URL https://aclanthology.org/2021.acl-long.58 .

Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. mFACE: Multilingual summarization with factual consistency evaluation, 2022.

Norman P Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. A domain-specific supercomputer for training deep neural networks. Communications of the ACM , 63(7):67-78, 2020.

Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. Scaling up models and data with t5x and seqio, 2022. URL https://arxiv. org/abs/2203.17189 .

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. Jax: composable transformations of python+ numpy programs. 2018.

A Appendix

A.1 Dataset Supplementary Materials

Dataset documentation, metadata, and download instructions: anonymous .

Intended uses: we hope SeeTRUE will be used by researchers to evaluate image-text matching models.

Author statement: We bear all responsibility in case of violation of right in using our benchmark.

Each dataset's license is described below. Our additional human annotations and geenrated images are licensed under CC-BY 4.0 license https://creativecommons.org/ licenses/by/4.0/legalcode .

Hosting & preservation: the dataset will be hosted in Huggingface Datasets, accessible and available for open research.

SeeTRUE fields are presented in table 5.

Additional licensing details: we do publish the datasets we annotated in this work, and do not re-publish the existing SNLI-VE and Winoground datasets. Full licenses:

MS COCO [28]: https://cocodataset.org/#termsofuse

EditBench [25]: https://research.google/resources/datasets/editbench/ , https://www.apache.org/licenses/LICENSE-2.0

DrawBench [5]: https://imagen.research.google/ , https://docs.google.com/ spreadsheets/d/1y7nAbmR4FREi6npB1u-Bo3GFdwdOPYJc617rBOxIRHY/edit# gid=0

Pick-a-Pick [29]: https://huggingface.co/datasets/yuvalkirstain/pickapic_ v1

SNLI-VE [27]: https://github.com/necla-ml/SNLI-VE

Winoground [24]: https://huggingface.co/datasets/facebook/winoground

A.2 Additional Examples

We added additional examples from (a) instances from SeeTRUE fig. 7; (b) the V Q 2 method fig. 8 to visualize the question-answer pairs; and (c) the ConGen technique fig. 9. This will underscore the versatility of ConGen in handling both real and synthetic images, as well as the variety of contradictions it can generate.

A.3 Human Annotation Process

In order to provide reliable human labels for our datasets, we conducted an annotation process using the SeeTRUE platform. The process comprised several steps, including setting qualification requirements, providing instructions, and evaluating annotator agreement.

Weset the basic requirements for our annotation task as follows: a percentage of approved assignments above 98%, more than 5,000 approved HITs, and annotator locations limited to the US, UK, Australia, or New Zealand. We selected 5 examples from our dataset for a qualification test and screened the annotators' results. fig. 10 displays a sample of the Mechanical Turk user interface. The payment for the crowd-workers was 15-18 USD hourly.

The instructions provided were as follows:

Evaluate the given image and text to determine if they match, selecting either 'Yes' or 'No'. Some images may be synthetically generated by a text-to-image model. To assess the match, mentally generate a textual description for the image (no need to write it down) and compare this generated description to the given text. If the descriptions closely resemble each other, mark 'Yes'. If not, mark 'No' and provide feedback on the specific issue causing the misalignment, focusing on the primary issue if multiple misalignments are present. If you encounter an image or text that may be offensive due to bias, race, NSFW content, etc., mark the checkbox to indicate this issue.

Full agreement metrics are presented in table 6. As shown in the table, the percentage of cases where all annotators agreed and the Fleiss-Kappa scores vary across the datasets, with COCO-Con

large white and pink rose planter surrounded by lush greenery large white and pink rose planter surrounded by dead greenery

a red haired bowie is painted on the wall

a blue haired bowie is painted on the wall

milk pouring into a coffee cup on a wooden table milk pouring into a coffee cup on a plastic

table

amanwithamustacheissittinginfrontofsomepinetrees

Doestheimagepresentallthedetailsdescribedinthetextcorrectly?

If'No,whatis themodel feedback?(*main*misalignment)

exhibiting the highest level of agreement and Drawbench the lowest. These differences highlight the varying levels of complexity within the datasets.

A.4 Comparing V Q 2 variants

V Q 2 consists of two main parts: generating question-answer pairs and assessing question-answer pair alignment against the image. We have experimented with different variants of the V Q 2 zero-shot method.

Assessing question-answer pair alignment methods Given a question-answer pair, we would like to assess the question based on the image and compare it to the information in the text. We experimented with several configurations for answer alignment:

Type A: Given a question-answer pair ( q j , a j ) generated from the text, we answer the question using a VQA model and obtain an answer based on the image a I j = V QA ( q j , I ) . We compare a pair of ( q j , a j ) with a pair of ( q j , a I j ) using an Natural Language Inference (NLI) model, where the pair based on the text serves as the premise and the other as the hypothesis. We define the alignment score s j as the probability of the NLI model for answering 'entailed'.

Type B: As done in type A, we answer the question using a VQA model and obtain the answer a I j = V QA ( q j , I ) . We use the VQA model again to compare the the two answers and determine whether they are the same. The question is formulated as 'Is { a j } == { a I j } in this image?' . We define the alignment score s j as the probability of the VQA model for answering 'yes'.

Type C: We reformulate each question and answer candidate pair ( q j , a j ) into a new yes-no predicate question q ′ j using the format 'is { a j } true for { q j } in this image?' . The VQA model is then invoked to answer the predicate question ( q ′ j ) over image I . We define the alignment score s j as the probability of the model for answering 'yes'.

Generating question-answer pairs To produce question-answer pairs from the text, we first extract informative spans in the text T . We extract as answer candidates all named entities and noun phrases in T using spaCy. We noticed that for short text T , this method doesn't produce enough question-answer pairs to assess the alignment between the text and the image. Thus, we extend the answer candidates by adding multi-word spans, such as adjectives ("black and white") and location "in the air". We use the extended answer candidate extraction in all of our experiments. The number of question-answer pairs is dependent on the length of the text T. In EditBench, the text contains 12.5 words on average, and it results in average of 34.9 question-answer pairs with extended answer candidates.

Table 7 summarize the results of the V Q 2 variants on the EditBench dataset. Our zero-shot V Q 2 method is V Q 2 type C, it outperforms other configurations and it is more efficient, since it requires a single run of the VQA model for the question-answer pair assessment.

A.5 Evaluating Contradictions Generated by GPT4

To assess GPT4's capability in generating contradictions, we compared the outputs from a GPT4driven ConGen approach against our PaLM-based method. We selected a set of 100 images from both COCO-Con and PickaPic-Con databases. For each image, we obtained 100 positive and 100 negative captions using PaLM. Subsequently, by utilizing the identical prompt with GPT4, we generated an additional 100 negative captions. These captions were randomly assigned labels 'A' and 'B'.

For the evaluation process, we engaged three independent evaluators from Amazon Mechanical Turk. They were presented with an image, its caption, and the two negative caption candidates. Their task was to discern whether each candidate accurately contradicted the depicted image. A snapshot of this user interface can be seen in fig. 11.

The final judgment was based on the majority vote of the workers. The results showed a close match in performance between the two models: PaLM achieved 77% while GPT4 reached 76%. This suggests our ConGen approach works effectively with various large-scale language models.

A Contradiction:A giraffe with its neck coiled around rocks and eatingmeat.

CorrectlycontradictstheimageODoesn'tcontradicttheimage

BContradiction:Agiraffewithitsneckstretched overrocksand eatingsand.

CorrectlycontradictstheimageODoesn'tcontradicttheimage

Agiraffewithitsneckstretchedoverrocksandeatinggrass.

A.6 Comparing V Q 2 with PaLI vs. BLIP2

We compare PaLI and BLIP2 for the VQ 2 method in table 8. While PaLI yields higher scores, it's pivotal to recognize their distinct evaluation metrics. PaLI's score indicates the probability of a 'yes' response, whereas BLIP2 gives a definitive answer without probability. This difference can influence the application suitability of each model.

A.7 Reproducibility

To fine-tune BLIP2, we adjust only the Q-former parameters of the model using the Adam optimizer. We train the model for two epochs and designate 10% of the training set as a validation set for early stopping and use learning rate selection between {1e-5, 5e-5}. A single training took 5 hours on a linux server with one A6000 GPU. All experiments took <2 days.

Zero-shot V Q 2 : For 10,000 text-image pairs, the inference time of every step is as follows. Answer candidate generation: when using extended answer candidates - about 1 day. Otherwise, 12 hours. Question generation and filtering: When using extended answer candidates, about 2 days, otherwise, 1 day. The last step only takes a few minutes.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to develop a method for evaluating the level of semantic alignment between text and image. In order to achieve this, the authors construct datasets of image and text pairs, and collect human judgments to determine if the text and image are semantically aligned. The paper proposes two methods for estimating alignment between text and image. The first method generates questions and answers from text and checks whether a Visual Question Answering (VQA) model provides consistent answers to the questions. The second method is an end-to-end classifier that predicts if the image entails the text. Experimental results demonstrate that the proposed methods outperform existing multimodal models.\n\nSTRENGTHS:\n- A significant contribution of this paper is the collection of human judgments on text and image alignment. This will aid in the development and validation of alignment evaluation methods in future research.\n- The annotation pipeline is carefully designed, and the details are throughly described.\n\nWEAKNESSES:\n- The VQ^2 method depends on the sampling of questions and answers, which may result in output variation due to chance. The effects of sampling are not discussed in the paper.\n- Language models used to generate contradicting captions may introduce biases in the datasets. For example, generated captions might have typical wording patterns that are difficult for humans to detect. Validating synthesized datasets is challenging.\n\nQUESTIONS:\n- Can the authors provide more details about the PaLM instruction used to generate contradicting captions? Was the prompt in Figure 2 utilized to rewrite captions?\n\nLIMITATIONS:\nThe paper discusses two limitations:\n1. The difficulty in judging alignment between a text and an image even for humans.\n2. The challenge of filtering offensive content from the dataset.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of text-image alignment evaluation by proposing a benchmark named SeeTRUE and two alignment metrics named VQ^2 and end-to-end VNLI. \n\nSeeTRUE covers a variety of real and synthetic images and captions. The synthetic captions, which are generated by large language models (LLMs), preserve most words from the original captions but convey a contradictory meaning. The synthetic images are generated by text-to-image generative models. The real image and captions are collected from existing datasets. In addition to the image-text pairs, SeeTRUE also provides binary human annotation of the alignment between these pairs.\n\nVQ^2 first generates question-answer pairs based on the caption and then assesses question-answer pair alignment against the image. End-to-end VNLI directly fine-tunes a visual NLI model to determine whether an image entails a caption.\n\nExperiments show that the proposed image-text alignment metrics outperform state-of-the-art (SOTA) vision-language models on the SeeTRUE benchmark. The VQ^2 metric also exhibits (1) a high correlation with humans in the evaluation of text-to-image generative models and (2) strong potential as a reranking technique to select generated images.\n\nSTRENGTHS:\n* The proposed SeeTRUE overcomes two limitations found in existing benchmarks: a primary emphasis on real images and a lack of challenging negative captions. SeeTRUE provides a valuable testbed for evaluating image-text alignment methods.\n* The proposed two image-text alignment metrics are well-designed and exhibit very promising results, which can facilitate the evaluation and development of image captioning and text-to-image generation models.\n* The experiments are comprehensive and well-designed.\n* The paper is well-written and easy to follow.\n\nWEAKNESSES:\nThe current manuscript still has three problems (which do not warrant rejection): \n* SeeTRUE primarily focuses on the challenging scenario where negative captions differ from positive captions by only a few words. However, it remains unclear whether the proposed VQ^2 and End-to-end VNLI models, specifically tailored for this setting, can still surpass the strong baselines in the standard scenario where negative captions describe completely different images.\n* Considering the utilization of three models—namely, an answer generation model (T5-XXL), a QA model (T5-XXL) and a VQA model (PaLI-17B)—VQ^2 might incur higher computational costs compared to the baseline methods. It would be beneficial to provide a detailed account of the computational expenses of these methods in Table 2.\n* It seems that the results in Table 7 of Appendix A3 are missing.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nPlease refer to the “Weakness” section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a benchmark and two methods for evaluating fine-grained and complex image-text alignment. Their benchmark involves multiple distracter captions and images involving both real and synthetic images and captions. They propose two methods - one evaluates image-text alignment by asking multiple entailed questions, and another is a large Vision-Language model fine-tuned on their benchmark image-caption data.\n\nSTRENGTHS:\nEvaluating by visual entailment is neat since it is interpretable. \nThey also show that they outperform a contemporary benchmark TIFA that does the same. \nThey release a comprehensive benchmark, a combination of available data and annotations they collect, which will be valuable to the community for evaluating vision-language models.\n\nWEAKNESSES:\nSometimes certain design choices are not well motivated and make the reader wonder why such a choice was adopted. For example, when generating qa pairs, the question is answered using a t5 Language model first to filter some qa pairs. Is this to ensure the answer semantically makes sense for such a question in general? If so, writing out clear motivation before explaining what is being done might be good. \n\nThe benchmark seems to be a collection of simply more human-annotated image-text data. Motivating why sometimes synthetic images or text is required and what are new ways one can use this to evaluate their models (that current benchmarks lack) would be nice. For instance, with the contrastive captions, one can evaluate compositionality. However, we can already do this with Winoground; what extra does this benchmark give us? \n\nSome other relevant image-text benchmarks that evaluate compositionality using similar ideas of distractor captions are CREPE (https://arxiv.org/abs/2212.07796) and COLA (https://arxiv.org/abs/2305.03689). It might be good to add in a discussion of how this benchmark relates.  \n\n(minor) Even though synthetic images and texts are human evaluated and filtered, the datasets may be biased to having images that current generation models can already generate well, limiting the application of evaluating image generation models on the dataset on edge case prompts.\n\nQUESTIONS:\nWhat are the train, val, and test splits of SeeTRUE? If I understand correctly, the VQ2 models are not trained and are just evaluated on the test split. However, the VNLI is trained on the train splits of SeeTRUE and then evaluated on the test splits. Is that correct?\n\nHow many questions are asked for a given image-text pair in the VQ2 method?\n\nLIMITATIONS:\nAuthors discuss limitations sufficiently", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SeeTRUE, a benchmark for evaluating image-text alignment, encompassing a diverse range of both real and synthetic images and text. The authors proposes two innovative approaches to evaluating alignment: VQ2, which relies on question generation and visual question resolution, and VNLI, which relies on fine-tuning substantial multimodal language models. The suggested methods perform better on a variety of alignment tests than earlier methods, particularly in difficult scenarios involving complicated composition or strange images. The study also shows how these techniques may rerank generated image candidates and pinpoint particular misalignments. This provides alignment evaluation methods for image-to-text and text-to-image models.\n\nSTRENGTHS:\n- This paper introduces comprehensive benchmark, SeeTRUE covers real and synthetic text and image pairs from a variety of different tasks, allowing for a more thorough evaluation of text-image alignment models. This could be a solid contribution on this field.\n- The paper presents two new text-image alignment evaluation techniques, VQ2 and VNLI. these methods outperform previous ones, especially in complex or unnatural images.\n- The paper is clearly written, and easy to follow\n- The author shared part of their codes\n\nWEAKNESSES:\n- Because performance is dependent on the LLM, the comparison doesn't seem fair. \n- Also, as Generating Contradicting Captions (even with LLM) isn't a new concept, there isn't much insight to be gained from this paper.\n( for example, Momeni et al. https://arxiv.org/abs/2304.06708  )\n- There aren't many examples. Even the appendix is not sufficient. It would be better if there were more examples.\n\nQUESTIONS:\n-Regarding the Dependency on the LLM, the methods largely depends on the quality of the LLM and language-and-vision model. Does this mean that the proposed methods would perform poorly if the model is not good enough? Can you provide a comparison of how these methods perform with different LLM and VLM models?\n\n-On the Novelty of Generating Contradicting Captions, Since the generation of contradicting captions is not a new concept, how does the proposed negative mining with LLM methods in this field? Is there an additional aspect of the method that brings unique contributions to the field of text-image alignment?\n\n-Can these methods be adapted for other tasks that involve the interaction of text and image\n\nLIMITATIONS:\nThe author addressed the limitations of this work. The authors could discuss the limitation of prompting LLM in more detail, as well as suggest ways to mitigate it.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}