Title: Are Diffusion Models Vision-And-Language Reasoners?
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion model

automatic evaluation

vision-and-language

compositionality
Abstract: Text-conditioned image generation models have recently shown immense qualitative success using denoising diffusion processes. However, unlike discriminative vision-and-language models, it is a non-trivial task to subject these diffusion-based generative models to automatic fine-grained quantitative evaluation of high-level phenomena such as compositionality.
Towards this goal, we perform two innovations. First, we transform diffusion-based models (in our case, Stable Diffusion) for any image-text matching (ITM) task using a novel method called DiffusionITM.
Second, we introduce the Generative-Discriminative Evaluation Benchmark (GDBench) benchmark with 7 complex vision-and-language tasks, bias evaluation and detailed analysis.
We find that Stable Diffusion + DiffusionITM is competitive on many tasks and outperforms CLIP on compositional tasks like like CLEVR and Winoground.
We further boost its compositional performance with a transfer setup by fine-tuning on MS-COCO while retaining generative capabilities. 
We also measure the stereotypical bias in diffusion models, and find that Stable Diffusion 2.1 is, for the most part, less biased than Stable Diffusion 1.5.
Overall, our results point in an exciting direction bringing discriminative and generative model evaluation closer. We will release code and benchmark setup soon.

Are Diffusion Models Vision-And-Language Reasoners?

Abstract

Text-conditioned image generation models have recently shown immense qualitative success using denoising diffusion processes. However, unlike discriminative vision-and-language models, it is a non-trivial task to subject these diffusion-based generative models to automatic fine-grained quantitative evaluation of high-level phenomena such as compositionality. Towards this goal, we perform two innovations. First, we transform diffusion-based models (in our case, Stable Diffusion ) for any image-text matching (ITM) task using a novel method called DiffusionITM . Second, we introduce the Generative-Discriminative Evaluation Benchmark (GDBench) benchmark with 7 complex vision-and-language tasks, bias evaluation and detailed analysis. We find that Stable Diffusion + DiffusionITM is competitive on many tasks and outperforms CLIP on compositional tasks like like CLEVR and Winoground. We further boost its compositional performance with a transfer setup by fine-tuning on MS-COCO while retaining generative capabilities. We also measure the stereotypical bias in diffusion models, and find that Stable Diffusion 2.1 is, for the most part, less biased than Stable Diffusion 1.5. Overall, our results point in an exciting direction bringing discriminative and generative model evaluation closer. We are releasing code and benchmark setup. 1

1 Introduction

Text-to-image generation is rapidly advancing. Generated images are not only highly realistic in various styles, but also reflect the compositional structure of open-ended text prompts [Chang et al., 2023, Saharia et al., 2022, Li et al., 2022b]. In this work, we evaluate language-conditioned generative image models on discriminative tasks to shed light on their fine-grained understanding of vision and language. A generative objective trains a model to understand how various objects and parts compose together, and often brings non-trivial emergent capabilities with it such as latent interpolation of composite concepts [Brock et al., 2019, Rombach et al., 2022]. On the other hand, discriminative vision-and-language models need only focus on the minimal information required to solve their discriminative task, which could often be spurious correlations that don't generalize [Agrawal et al., 2016]. Such erroneous understanding is then exposed downstream on image-text matching tasks purposefully designed to catch it, such as image/text retrieval using Winoground [Thrush et al., 2022], ARO [Yuksekgonul et al., 2023], or ImageCoDe [Krojer et al., 2022]. Following these benchmarks' releases, there has been a growing focus in the vision-and-language community to fix these problems.

Vikram Voleti

"two dogs play with

an orange toy in

W

tall grass"

Diffusion

→Eo(x, t, w)1

[lle(x, t, w) - ell2

Model

x

Score

ITM

Diffusion

E(x,t)—llee(x, t) -ell2

Model

80

79.4

Model

70

74.1

76.1

Diffusion ITMwithStable Diffusion 1.5

60

59.5

59.2

61.5

65.7

63.1

62.5

68.6

DiffusionITMwithStableDiffusion2.1

504030

52.7

55.3

50.7

49.8

46.1

41.6

20

23.9

24.8

31

31.6

26

30.1

29

32.3

10

13.9

15.7

10.7

8.3

7.5

6

0

Flickr Text

Flickr Image

AROVG Attr.

ARO VG Rel.

COCOOrder

CLEVR

Pets

SVOVerb

SVO Obj

ImageCoDestatic

ImageCoDevideo

Winogroundgroup

Task

We hypothesize that a generative model trained to synthesize compositional data is capable of understanding the complexities required to solve hard image-text-matching tasks. To this end, we transform a text-to-image generative model for zero-shot image-text matching , and introduce Diffusion Image-Text Matcher (DiffusionITM ; Fig. 1). In this work, we use Stable Diffusion (SD) [Rombach et al., 2022] as the text-to-image model, but any other diffusion model could be used. DiffusionITM achieves competitive zero-shot performance on both image and text retrieval (Tab. 1).

The naive approach for image retrieval given a text prompt would be to pick the image for which SD gives the least noise prediction error, and vice versa for text retrieval. While this works well for text retrieval [Li et al., 2023], we show it achieves random performance on image retrieval (Tab. 1). Our main insight explaining this discrepancy is that the model's success at denoising depends primarily on the visual properties of the scene, rather than equally on visuals and text (Fig. 3). Therefore, if such a model has to select among several images given a text prompt, it will have the lowest noise prediction error for a visually familiar image regardless of the text prompt. To address this, we compute the error relative to the unconditional (i.e. no text) error (Fig. 1). Our method outperforms existing diffusion-based discriminative methods [Li et al., 2023, Clark and Jaini, 2023] (see Tab. 1).

Since the original generative pretraining objective of noise prediction is far from the ultimate downstream task of image-text-matching, we explore a novel discriminative finetuning scheme with hard negative image-text-pairs on MS-COCO [Lin et al., 2014]. We find that this transfers well to other datasets, and improves discriminative performance (Tab. 1) as well as generated images from DrawBench prompts [Saharia et al., 2022].

Finally, we present the GDBench to foster research progress on image generation. Our DiffusionITM method enables a new automatic, fine-grained, and downstream way to evaluate diverse skills in text-conditioned image generation. Thus, once we cast an image generation model as a discriminator, we can draw from the rich history of diagnostic datasets and detailed analyses in the vision-andlanguage literature to analyze the model's reasoning ability. Evaluation of image generation has been notoriously difficult, and recent proposals advocate using secondary model probes on generated images to test for fine-grained ability [Hu et al., 2023, Cho et al., 2022]. In contrast, we directly evaluate diffusion models on downstream tasks without requiring any probes. In the same spirit as GLUE [Wang et al., 2018], we systematically select 7 image-text-matching tasks covering diverse reasoning skills from traditional image retrieval to diagnostic tasks (i.e. compositionality). In addition, GDBench also includes a bias evaluation dataset which we use for social bias analysis.

GDBench allows head-on comparison between generative models, as well as with discriminative models like CLIP [Radford et al., 2021]. Our results are as follows:

Stable Diffusion (SD) vs. CLIP : SD is competitive with CLIP on many tasks, and outperforms it on challenging compositional tasks like CLEVR and Winoground Text (Tab. 1). Further tuning of SD on MS-COCO with hard negatives outperforms vanilla SD on both image and text retrieval (Tab. 2). SD has lower stereotypical bias than CLIP (Tab. 3).

Stable Diffusion (SD) 1.5 vs 2.1 : We observe varying degrees of progress, but overall SD 2.1 outperforms SD 1.5 (Fig. 2). SD 2.1 is less biased than SD 1.5, contrary to the common trend of increased bias in larger, stronger models [Nadeem et al., 2021].

Image generation: Remarkably, improved discriminative performance via finetuning on MS-COCO also leads to improved high-level understanding of image generation. We find higher image-textalignment on DrawBench [Saharia et al., 2022] prompts compared to vanilla SD (Appendix B).

2 Related Work

Vision-And-Language Understanding: Widely popular tasks for vision-and-language understanding are often framed as discriminative tasks, such as image retrieval [Miech et al., 2021], VQA [Agrawal et al., 2015] or REC [Yu et al., 2016]. We focus on image-text-matching (ITM) tasks (assigning a score to an image-text-pair) due to their general applicability and simplicity. Noisy image-text pairs are used for large-scale ITM training using contrastive learning [Radford et al., 2021], and has been traditionally tackled either via dual encoders like CLIP [Radford et al., 2021], or models with richer cross-modal interaction like BLIP [Li et al., 2022a]. At the same time, ITM allows probing models in a controlled manner with simple metrics [Thrush et al., 2022, Hendricks and Nematzadeh, 2021], as opposed to more complex metrics on generative tasks [Hessel et al., 2021, Saharia et al., 2022]. There has been a growing interest on diagnostic benchmarks for compositionality that use hard negatives [Yuksekgonul et al., 2023, Thrush et al., 2022, Krojer et al., 2022, Hendricks and Nematzadeh, 2021, Lewis et al., 2022, Parcalabescu et al., 2022]. This rich literature can now be transferred to the world of image generation with our GDBench benchmark, enabling fine-grained analysis and benchmarking of text-conditioned image generation.

Repurposing Text-Conditioned Diffusion Models: Text-conditioned diffusion has shown remarkable advancements not only in image quality but also image-text alignment [Rombach et al., 2022, Saharia et al., 2022]. A natural next question is how to leverage these abilities for other tasks [Burgert et al., 2022]. Two concurrent studies use similar methods to our diffusionITM in a more restricted setting: Diffusion Classifier [Li et al., 2023] performs image classification using SD by selecting the class with the lowest noise prediction error; Clark and Jaini [2023] puts more focus on timestep-weighting, and uses Imagen [Saharia et al., 2022] instead of SD. However, they only show competitive results for text retrieval, emphasizing image classification as a special case. Many vision-and-language reasoning tasks are also framed as image retrieval [Krojer et al., 2022, Hendricks and Nematzadeh, 2021, Thrush et al., 2022]. In this work, we tackle the broader scope of vision-and-language understanding, generalize our method to image retrieval, and study hard-negative fine-tuning and transfer.

Evaluation of Image Generation: Image generation is traditionally evaluated along the two axes of image quality and image-text alignment, using metrics based on individual examples. The recently proposed TIFA metric [Hu et al., 2023] relies on an additional VQA model to answer a set of LLMgenerated questions about the generated image. More traditional metrics include FID [Heusel et al.,

2017] for image quality; CLIPScore [Hessel et al., 2021, Kim et al., 2022] for image-text alignment based on CLIP-embedding; object-centric metrics [Hinz et al., 2020, Cho et al., 2022] leveraging object detectors such as DETR [Carion et al., 2020]; and caption-based metrics [Hong et al., 2018] like BLEU on captions of the generated image. In contrast, GDBench is not a metric on individual examples, but rather a holistic evaluation framework. GDBench does not require another large model (e.g. VQA) for evaluation, and can be run on many diverse datasets.

Bias in Image Generation Models: It is well known that LLMs learn and amplify harmful biases present within text training corpora [Caliskan et al., 2017]. Due to the lack of automatic evaluation techniques for generative models, bias investigation has mostly focused on discriminative visionand-language (VL) models [Srinivasan and Bisk, 2022, Janghorbani and De Melo, 2023]. Only few works have tackled bias in recent text-to-image models [Luccioni et al., 2023, Cho et al., 2022] and to our knowledge only Luccioni et al. [2023] focus on bias alone: They quantify social biases by generating images over several social groups (ethnicity and gender) and measuring their variation over selected attributes (gendered adjectives and professions). They found that SD 1.4 and 2.0 are biased towards groups 'associated with whiteness and masculinity' across target attributes, and that SD 2.0 was more biased than 1.4. While their methods are thorough and work for black-box systems, the evaluation is quite time consuming and manual.

3 Our Approach to Image-Text Matching with Diffusion Models

3.1 Diffusion Image-Text Matching: Overcoming the modality asymmetry for image retrieval

We present our method Diffusion Image-Text Matching (ITM) . Our goal is to assign a score to an image( x )-text( w ) pair ( x , w ) which is broadly useful for downstream applications. We provide ( x , w ) to the diffusion model and task it to 'edit' the image according to the text. Our main intuition is if the image is not described by the text, a lot of edits are needed to fit the text, in which case it gets a low score, and vice-versa. See Appendix C for visualization.

Text-conditioned Diffusion: The objective of diffusion models is to denoise an image x by predicting the noise ϵ added to its clean version, conditioned on text w and noise level t :

Intuitively, the predicted noise is farther from the true noise when the image-text do not fit together. To transform this for ITM tasks, the sample with the lowest L2-distance of predicted and true noise could be used to select among a set of image-text-pairs. Li et al. [2023] and Clark and Jaini [2023] (concurrent works) have focused primarily on text retrieval (with classification as a special case), where the model selects from a number of texts (or class names for classification):

However, naively applying this in practice would imply sampling a different ϵ for each pair ( x , w ) to reduce the variance in L2-distance. Li et al. [2023] a) sample many (hundreds!) noise-timestep pairs ( ϵ, t ) uniformly, and b) crucially keep the sampled ( ϵ, t ) constant across different w when calculating Equation 2. Finally, the guidance scale is kept at 0, thereby discarding unconditional noise prediction. This could be repurposed to perform image retrieval by iterating over images instead of text:

We observe that while this results in SD as a strong text retriever, it achieves random chance on all image retrieval tasks! Interestingly, Li et al. [2023] observed that a Bayesian posterior from a discrete-label class-conditional generative model can be approximated using the analysis:

Li et al. [2023] use pairwise samples between pairs of labels c i and c j , to produce a low complexity approximate computation reminiscent of paired differences tests in statistics. Their analysis is based on a marginal across all possible discrete label captions c with respect to an image x . In

37.5

38.0

38.5

39.0

39.5

40.0

40.5

Denoising Error

Image 1

Image 2

random captions

caption 1

unconditional

contrast, in our approach, we replace the marginal in the denominator with simply a sample from the unconditional diffusion model for p ( x ) , and operate in log space. The intuition behind our approach is that taking an integral over all possible caption strings ( w ) is equivalent to simply the unconditional, since the 'caption' dimension is (implicitly) marginalized out. Fig. 3 illustrates another view of this intuition that for a correct text-image pair (Image1-Caption1), the denoising error is visibly smaller than for all mismatched text-image pairs for the same image (Image1). Moreover, for an incorrect but similar image (Image2), the denoising error for Caption1 is close to random, but could be less than that of Image1-Caption1 in absolute value. The incorrect Image2 would be selected regardless of the text since it is visually easier to denoise. Hence, the denoising error depends almost entirely on the image, independent of text conditioning ('modality asymmetry').

This analysis naturally leads to the intuitive solution of normalizing the error by the unconditional (no text) error. After all, we only care about the relative difference of how much easier or harder it becomes to denoise the image with a given text relative to when no text is given (see Fig. 1):

3.2 HardNeg-DiffusionITM : Tuning with compositional hard negatives and transfer

Our goal is to transform diffusion-based models for discriminative image-text-matching (ITM). However, the denoising diffusion objective only considers positive image-text pairs, and the large pre-training corpus LAION [Schuhmann et al., 2021] contains many noisy/simple examples, not conductive to complex linguistic reasoning. In contrast, models specifically dedicated to vision-andlanguage reasoning such as CLIP [Radford et al., 2021] or BLIP [Li et al., 2022a] were pre-trained with negatives and, in the case of BLIP, had access to high-quality curated image-text data. Previous baselines [Li et al., 2023] ditched the generative objective and turned it fully discriminative by finetuning a ResNet on top of the frozen mid-layer features of the U-Net for each dataset separately. We instead adopt parameter-efficient finetuning with LORA layers [Hu et al., 2022] that are added to the cross-attention from U-Net to the text, so as not to deviate too far from pretraining representations.

We address the lack of high-quality image-text-data by fine-tuning the diffusion model on MS-COCO (109K examples) with the standard diffusion objective (see Equation 1). As MS-COCO contains diverse high-quality image-text pairs, we finetune only once, and evaluate using GDBench tasks. This could be thought of as a second limited pre-training.

We address the lack of negative examples by adopting the hard negatives from Yuksekgonul et al. [2023] on MS-COCO: swapped text elements of the same part-of-speech (e.g. 'Men keep watch on a herd of goats' → 'Goats keep watch on a herd of men'), and CLIP-based image hard negatives. The naive approach would be to minimize the noise prediction error on positive pairs ( x , w pos ) , and maximize for negative pairs ( x , w neg ) . However, if this inverse loss were applied unhinged with potentially infinite gains, it would lead to the model predicting non-sense for everything. Therefore we threshold the hard-negative error at a relative scaling factor λ of the value of the positive error:

We choose relative thresholding since we want to ensure that the model never deviates too much from its original objective of noise prediction from positive prompts. Hence L neg is clipped between

[ -L pos , L pos ] . Unlike CLIP, we cannot include a large number of hard negatives in our batches since diffusionITM encodes image and text together. The resulting model, HardNeg-DiffusionITM , is still evaluated in a zero-shot fashion on the target evaluation tasks, i.e., how well does DiffusionITM trained on MS-COCO transfer to target tasks.

4 Data: The GDBench Benchmark

There is a fundamental need to measure downstream performance of diffusion-based generative models on a wide range of vision-and-language reasoning tasks, to facilitate quick improvement and progress tracking. This has worked in the past, i.e. with the NLP GLUE benchmark [Wang et al., 2018]. With this motivation, we introduce GDBench , a benchmark of eight diverse imagetext-matching (ITM) tasks to explore many types of vision-and-language reasoning. These include 7 ability-centric and 1 bias dataset for the image generation community (examples shown in Fig. 4). Additionally, most GDBench tasks have further fine-grained scores on sub-phenomena without requiring manual inspection, as is the case with evaluating generative models [Saharia et al., 2022]. ITM as a reasoning benchmark offers simplicity and a surprising amount of diversity. After all, ITM has become a standard paradigm for diagnostic vision-and-language datasets (see task list below) and therefore allows interpretable evaluation on many downstream skills. GDBench is similar to the spirit of downstream evaluation of generative models such as in TIFA evaluation [Hu et al., 2023] which makes use of a secondary model like VQA on the generated images and shows that downstream performance correlates with image quality. Whereas, we stay closer to the generative realm and evaluate generative models on several discriminative tasks without the need for secondary models. Below we introduce each dataset, asking 'What phenomena does it cover that others do not?'

Flickr30K [Young et al., 2014] is a well-established open-ended image and text retrieval dataset, captioning diverse scenes involving people. Note that we changed the task setup to reduce computation overhead: A model has to retrieve among the 10 most similar examples based on the CLIP embedding space. While this might not be fair towards our CLIP baselines, we chose this setup primarily to study if SD could be useful a second-stage slow retriever, after a fast retriever like narrows down the candidates [Miech et al., 2021]. Both Winoground [Thrush et al., 2022] and ARO [Yuksekgonul et al., 2023] are diagnostic benchmarks for compositionality. Winoground is carefully curated at the cost of only 400 examples and many SOTA models have not reached significantly beyond random chance. ARO is automatically generated on top of Flickr30K, MS-COCO, and others with only compositional hard text negatives. ImageCoDe [Krojer et al., 2022] is an image retrieval task focusing on highly similar images with complex pragmatic captions crowdsourced from a guessing game. SVO [Hendricks and Nematzadeh, 2021] disentangles performance along different parts-of-speech by pairing images that differ only in subject, object or verb. Lewis et al. [2022] introduced a diagnostic controllable benchmark based on simple synthetic CLEVR 2 images of 3D shapes, thereby isolating various phenomena like attribute binding or spatial relations. Finally, we include Pets [Parkhi et al.,

(b) Accuracy of DiffusionITM and baselines on GDBench text retrieval tasks .

2012] as a smaller quick-to-run image classification dataset. In contrast to linguistically complex GDBench tasks, Pets covers the complementary skill of fine-grained recognition (37 animals). 3

Measuring Bias: Bias evaluation is essential as large-scale models increasingly suffer from harmful biases that can impact downstream tasks. DiffusionITM allows automatic, quantitative bias evaluation with bias datasets intended for discriminative vision-and-language models. We utilize the dataset from Janghorbani and De Melo [2023] and investigate three different types of social biases: religious, nationality, and sexual orientation. Quantitatively, bias is present when there is a substantially stronger association of one target group to pleasant attributes compared to unpleasant attributes over another group, as measured by some notion of distance or similarity. The target groups are represented by sets of images X I , Y I and the attributes are captured by sets of words A T , B T . Bias is measured by the following normalized association score, d , called the effect size : 4

and σ ( · , · ) is our proposed DiffusionITM score, or, in the case of CLIP, cosine similarity.

More concretely, in the case of Religion, the image sets might be X = Christians, Y = Muslims and the attribute word sets would be A = { 'joy', 'trust', 'good', ... } , B = { 'corrupt', 'vulgar', 'bad', ... } . Here, a positive effect size means that it is biased towards Christians and a negative effect size means it is biased towards Muslims. We provide more details under limitations in Appendix A.

5 Experiments and Results

Our main two findings are summarized in Tab. 1: First, zero-shot DiffusionITM achieves performance near CLIP on image retrieval (Tab. 1a), overcoming the close-to-random performance of Diffusion Classifier [Li et al., 2023]. Second, our best hard negative transfer-finetuning strategy improves performance across the board, on both image and text retrieval.

Hyperparameters: Based on ablations in Li et al. [2023], we adopt most of their setup but aim for more simplicity: Timesteps t are sampled uniformly from [0 , 1000] , guidance scale is kept at 0, but we drop the complicated procedure that iteratively prunes classes after a number of noise-timestep samples ( ϵ, t ) . Instead we keep samples constant at 250 for the main zero-shot experiments in Tab. 1 and reduce it to a much more feasible number of 10 samples for other experiments. 6 This is aligned with our goal that GDBench should be easily adopted by researchers to evaluate their image generation method. As shown in Appendix Fig. 9, performance is not at its maximum with few samples but the trends can be studied in the same way when comparing models along different tasks. We adopt the common CLIP RN50x64 baseline and OpenCLIP ViT-L/14 for a fair comparison since SD 2.1's text backbone is from the latter. We fine-tune on the MS-COCO hard negative training set [Yuksekgonul et al., 2023] with lr = 1 e -4 , λ = 1 . 0 and batchsize 112. We select a checkpoint after 8 epochs based on hard negative validation. Runtime: With 10 noise samples per image-text-pair evaluation on Flickr30K Text Retrieval validation takes 68 minutes on a single NVIDIA RTX A6000 GPU (compared to around 4 minutes with OpenCLIP ViT-L/14). We point to Li et al. [2023] for an in-depth analysis of runtime. We emphasize that we envision future stronger SD-based discriminators as slow retrievers that are applied to a small set of candidates provided by a fast retriever [Miech et al., 2021], as well as the benefit of automatic evaluation.

DiffusionITM performance: Our DiffusionITM significantly outperforms Diffusion Classifier on all image retrieval tasks (Tab. 1a) while CLIP still outperforms DiffusionITM . However on text retrieval, DiffusionITM even outperforms CLIP on some tasks (Tab. 1b), most notably compositionality-focused Winoground Text and CLEVR by a large margin. Here DiffusionITM is clearly behind CLIP on only two ARO subtasks (Flickr and COCO Order).

HardNeg-DiffusionITM performance: We find that HardNeg-DiffusionITM trained on MS-COCO transfers well to all tasks, outperforming DiffusionITM (Tab. 1). In Tab. 2 we disentangle the effect of using higher quality data and various additional hard negatives. We highlight three insights: 1) Despite fine-tuning on only a single dataset, we observe gains across almost all tasks without any negatives ( NoNeg ) explicable by the general high-quality MS-COCO dataset compared to more noisy pre-training data. 2) Using hard negatives for only one modality ( HardNegTxt/RandNegTxt vs. HardNegImg ) only improves the respective retrieval performance while showing occasional drops in the other. 7 3) We therefore combine all three types of hard negatives, allowing us to work with one model, HardNeg-DiffusionITM , rather than multiple models specific to each modality retrieval.

Stable Diffusion 1.5 vs. 2.1 Performance: 2.1 improves on the majority of GDBench tasks except for ARO. With GDBench 's diverse tasks, we can study if later versions of Stable Diffusion improve in all skill-dimensions or just some of them (see Fig. 2). Interestingly SD 2.1 does not show significant gains over SD 1.5 on all of them. Most notable is ARO (compositionality): We see only minor improvements (VG tasks) or slight drops (Order tasks). At the same time, we do see a jump on Winoground Text from 29% to 32.3% and on other less adversarial tasks such Flickr30K or SVO.

Bias: Both CLIP and Stable Diffusion exhibit bias towards the dominant groups, namely Christians, Americans, and Heterosexuals (Tab. 3) with Stable Diffusion 2.1 displaying the least bias. Almost all scores are statistically significant for p < 0 . 01 , with the exception of the Jewish-Muslim for both SD versions (Tab. 3) and some of the Buddhist scores (Tab. 5). Version 2.1 has overall lower effect sizes (average absolute effect size of 0.65 in 2.1 vs. 0.79 in 1.5), suggesting that it is less biased than version 1.5 for this metric (Tab. 3). This goes against the trend of increased bias in stronger models [Nadeem et al., 2021]. On the other hand, Luccioni et al. [2023] found that Stable Diffusion 1.4 is less biased than 2.0. Further investigation is needed to draw a strong conclusion as to whether the 2.x versions are more or less biased than the 1.x versions due to this discrepancy. It is important to note that there was a major weakening of the safety filter between version 2.0 and 2.1, which may have affected the diversity in the training set and as such, model bias.

Analysis: Wefind higher image-text-alignment of images generated by HardNeg-DiffusionITM model based on human judgement. Although our method improves discriminative performance, does it also result in more compositional image generation? Crucially our finetuning on MS-COCO preserved the generative capabilities despite directly modifying the noise prediction. We therefore compare image-text-alignment of DiffusionITM against HardNeg-DiffusionITM on DrawBench [Saharia et al., 2022] and find promising results: From 105 complex prompts, we found that HardNeg is closer to the text almost twice as often as the zero-shot model. Similarly, HardNeg finetuning also shows slightly better image-text-alignment than NoNeg finetuning. For more DrawBench details see (Appendix B) and other analyses (Appendix F). One might ask: how complementary are the skills learned in a generative vs. a discriminative vision-and-language model? We quantify this via the overlap of correctly predicted examples. Our hypothesis: Even if DiffusionITM might have lower performance on a task, its correct predictions may still cover new examples that discriminative models fail to capture. On three datasets, we compare DiffusionITM and two discriminative models (CLIP and BLIP) that were trained differently enough to expect varying predictions. However we find no evidence for

our hypothesis and perhaps even signs of an opposite trend (Fig. 8). We speculate that this points towards the big role the text encoder behind an text-to-image model plays for vision-and-language understanding. After all, Stable Diffusion relies on a frozen CLIP text encoder.

6 Post-submission: The intriguing case of Stable Diffusion XL

After the paper submission, Stable Diffusion XL (SDXL) [Podell et al., 2023] was released so it was a reasonable next step to include it in our comparison of different SD versions (see Fig. 2). We expected SDXL to outperform previous versions, based on our upwards trend from 1.5 to 2.1 as well as findings in Podell et al. [2023]. Surprsingly, SDXL reaches significantly lower scores, below 2.1 scores on everything except three ARO subtasks, and even below 1.5 on most tasks (App. F Fig. 10 show exact numbers). Moreover, the top predictions of SDXL have the same exact scores (i.e. two images being ranked first) far more often than in SD 2.1. We confirmed that our results are not coming from a bug in our implementation and hope that future work can shed more light on quantifying the higher-level abilities of SDXL. In the meantime we offer a preliminary interpretation here: It is possible that the capabilities of image generation and image editing, specifically providing a new prompt on a partially noised image, are not always correlated. In light of this, we also offer two broader interpretations for the validity of DiffusionITM as a new evaluation methodology: Either our proposed evaluation is not generally applicable to all forms of vision-and-language skills in text-to-image models and instead measures more nuanced aspects such as image editing or text sensitivity. Alternatively, this anomaly is evidence that our evaluation is precisely working as intended and exposes flaws that might otherwise taken longer to detect via quantitative evaluation or other metrics such as FID or TIFA.

7 Conclusion and Future Directions

In this paper, we introduce DiffusionITM and GDBench . DiffusionITM allows us to transform any diffusion-based generative model to perform image-text matching tasks. This in turn leads to an exciting new paradigm of evaluating text-to-image models on vision-and-language reasoning using diagnostic benchmarks. We also improve vision-and-language understanding of diffusion models with a novel hard negative finetuning strategy ( HardNeg ). GDBench allows one to study many different types of vision and language reasoning, ranging from: compositional (ARO, Winoground) and visual fine-grained reasoning (ImageCoDe), to elements of spatial/attribute binding (CLEVR). Through our experiments, we find that our proposed DiffusionITM approach shows vision-and-language reasoning capability through increased performance on the several evaluations, as well as provides for head-on automatic comparison among generative models. We conclude that Stable Diffusion performs competitively to CLIP, and performs better on compositional tasks. We also conclude that SD 2.1 is less biased than SD 1.5. We hope that this line of work demonstrates that high-quality diffusion methods for generating images performs well on vision-and-language reasoning. We see that the simple task of image-text-matching allows us to test many types of reasoning capabilities by carefully selecting the appropriate negatives. We encourage future research to explore this paradigm on other models families like Masked Generative Transformers [Chang et al., 2023], and with stronger backbone text encoders. While we found that improved discriminative performance translates into better image-text-alignment in generated images, this should be explored in more detail. We discuss detailed limitations (especially bias-related) in Appendix A

References

Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. Vqa: Visual question answering. International Journal of Computer Vision , 123:4-31, 2015.

Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 1955-1960, 2016.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=B1xsqj09Fm .

Ryan Burgert, Kanchana Ranasinghe, Xiang Li, and Michael S Ryoo. Peekaboo: Text to image diffusion models are zero-shot segmentors. arXiv preprint arXiv:2211.13224 , 2022.

Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science , 356(6334):183-186, April 2017. ISSN 0036-8075. doi: 10.1126/science.aal4230.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16 , pages 213-229. Springer, 2020.

Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 , 2023.

Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. arXiv preprint arXiv:2202.04053 , 2022.

Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero-shot classifiers. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models , 2023. URL https://openreview.net/forum?id=c09lxGznVa .

Lisa Anne Hendricks and Aida Nematzadeh. Probing image-language transformers for verb understanding. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 3635-3644, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.findings-acl.318. URL https://aclanthology.org/2021.findings-acl.318 .

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Joseph Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. ArXiv , abs/2104.08718, 2021.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/ 8a1d694707eb0fefe65871369074926d-Paper.pdf .

Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic object accuracy for generative text-toimage synthesis. IEEE transactions on pattern analysis and machine intelligence , 2020.

Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7986-7994, 2018.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id= nZeVKeeFYf9 .

Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897 , 2023.

Sepehr Janghorbani and Gerard De Melo. Multi-modal bias: Introducing a framework for stereotypical bias assessment beyond gender and race in vision-language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 1717-1727, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023.eacl-main.126 .

Jin-Hwa Kim, Yunji Kim, Jiyoung Lee, Kang Min Yoo, and Sang-Woo Lee. Mutual information divergence: A unified metric for multimodal generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=wKd2XtSRsjl .

Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. Image Retrieval from Contextual Descriptions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3426-3440, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long. 241. URL https://aclanthology.org/2022.acl-long.241 .

Martha Lewis, Qinan Yu, Jack Merullo, and Ellie Pavlick. Does clip bind concepts? probing compositionality in large image models. arXiv preprint arXiv:2212.10537 , 2022.

Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. arXiv preprint arXiv:2303.16203 , 2023.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pretraining for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022a.

Zhiheng Li, Martin Renqiang Min, Kai Li, and Chenliang Xu. Stylet2i: Toward compositional and high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18197-18207, 2022b.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014.

Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias: Analyzing societal representations in diffusion models, 2023.

Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. An empirical survey of the effectiveness of debiasing techniques for pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1878-1898, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. acl-long.132. URL https://aclanthology.org/2022.acl-long.132 .

Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers, March 2021. URL http: //arxiv.org/abs/2103.16553 . arXiv:2103.16553 [cs].

Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 5356-5371, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.416. URL https://aclanthology.org/2021.acl-long. 416 .

Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. VALSE: A task-independent benchmark for vision and language models centered on linguistic phenomena. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8253-8280, Dublin, Ireland, May 2022. Association for

Computational Linguistics. doi: 10.18653/v1/2022.acl-long.567. URL https://aclanthology. org/2022.acl-long.567 .

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. IEEE Conference on Computer Vision and Pattern Recognition , 2012.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL http://arxiv.org/abs/2103.00020 . arXiv:2103.00020 [cs].

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, June 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv , abs/2205.11487, 2022.

Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. It is all about where you start: Text-to-image generation with seed selection. arXiv preprint arXiv:2304.14530 , 2023.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Tejas Srinivasan and Yonatan Bisk. Worst of both worlds: Biases compound in pre-trained vision-and-language models. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP) , pages 77-85, Seattle, Washington, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.gebnlp-1.10. URL https: //aclanthology.org/2022.gebnlp-1.10 .

Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A Corpus for Reasoning about Natural Language Grounded in Photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 6418-6428, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1644. URL https://www.aclweb.org/anthology/P19-1644 .

Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality, April 2022. URL http://arxiv.org/abs/2204.03162 . arXiv:2204.03162 [cs].

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-5446. URL https://aclanthology.org/W18-5446 .

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , 2:67-78, 2014. doi: 10.1162/tacl_a_00166. URL https://aclanthology.org/Q14-1006 . Place: Cambridge, MA Publisher: MIT Press.

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. ArXiv , abs/1608.00272, 2016.

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/ forum?id=KRLUvxh8uaX .

Kankan Zhou, Eason Lai, and Jing Jiang. VLStereoSet: A study of stereotypical bias in pre-trained vision-language models. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 527-538, Online only, November 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022. aacl-main.40 .

A Limitations

Our method is limited to image-text matching and cannot tackle other task formulations such as VQA [Agrawal et al., 2015, Suhr et al., 2019]. Additionally, taking more noise samples for a given image-text pair leads to better performance up to a certain point (see Fig. 9). This results in slower inference time and can hopefully be mitigated with future innovations.

Measuring bias is a crucial aspect of model evaluation and should not be considered secondary to the 'performance' of the model. Unfortunately, the dataset from Janghorbani and De Melo [2023] does not include gender bias but other datasets such as Zhou et al. [2022] could also be incorporated. In any case, evaluating bias related to personal identities such as nationality, race, and sexual orientation presents certain limitations. These labels are nuanced and not always accompanied by visible identifying characteristics. Consequently, image datasets depicting these groups have limited capacity to fully represent these demographics and intersectional identities. While we recognize that assigning a bias score based on these limited resources might not be entirely accurate, it is a vital first step in the right direction. 8 It is also important to note that the bias evaluation has only positive predictive power, meaning that a large effect size is an indication of bias, but a low one does not necessarily verify that the model is fair. Moreover, the bias effect size (Eq 8) may sometimes be unreliable [Meade et al., 2022]. This should serve as a preliminary analysis of biases present in the model, which require further investigation using more nuanced and comprehensive methods, including broader qualitative analysis and consultation with social scientists.

B DrawBench Evaluation

As described in Sec. 5, we manually compare our best HardNeg Stable Diffusion with vanilla Stable Diffusion on the DrawBench benchmark [Saharia et al., 2022]. It contains 200 curated challenging prompts along 11 categories. We only select these categories that are relevant to the phenomena studied in this paper (spatial, compositional, attribute binding, ...) and drop categories such as Text (i.e. A storefront with 'Hello World' written on it. ). This leaves us with 6 categories and 104 prompts:

Next we generate images for each prompt with both models and display them randomly to an author as left or right image. We judge whether the left or right image is better aligned with the text and do not focus on image quality. Because we only focus on high-level alignment with text, we observe many ties as both models exhibit the same level of alignment. We repeat this process with another seed, leaving us with 208 blind comparisons. Out of those, our HardNeg model is judged better almost twice as often as vanilla Stable Diffusion (60 vs. 33). The rest (115) are ties.

We show an example where HardNeg is more text-aligned as well as a tie below:

Since HardNeg is not strictly better than NoNeg in our experiments (see Tab. 2), we conduct the same study with these two models: HardNeg wins 50 comparisons and NoNeg only 38. While this is not statistically significant, it indicates that HardNeg finetuning is useful for image-text-alignment.

C Visualizing image editing with input optimization and standard denoising

Optimizing the input image directly via backpropagation has shown promise in the diffusion literature [Poole et al., 2022, Samuel et al., 2023]. It also leads to qualitatively insightful images that differ from standard image editing with Stable Diffusion. Concretely, we optimize the input image (i.e. the latent z ) with Adam (lr=0.05, 200 steps) based on the noise prediction loss where noise is added at timestep t = 0 . 5 . In other words: the input image itself is modified such that it leads to a lower noise prediction error. Note that this was generated with Stable Diffusion 1.5 before we adopted 2.1 into our experiments.

Next, we compare image optimizaton to the more established image editing approach to visualize the models reasoning, i.e. the denoising process does not start from pure noise but from a partially noisy image. Below, we show different strength factors between 0.4 and 0.7 for the same two examples as above and denoise for the corresponding amount of steps (according to recommended HuggingFace Diffusers parameters a value of 0.5 would correspond to 25 denoising steps opposed to the usual 50 steps from pure noise). A lower strength factor means less added noise and therefore less editing.

D Bias Evaluation Details

E CLEVR Detailed Performance

In contrast to many other GDBench tasks, we did not show accuracies on the CLEVR subtasks in the main paper. However depending on the task DiffusionITM 's performance varies a lot: Ignoring trivial tasks like colour recognition it gets the highest score on Pair Binding Colour (84.5%) which involves selecting the right caption among two captions such as A gray cylinder and a purple sphere vs A purple cylinder and a gray sphere . On the other hand on spatial it is close to random chance (i.e. On the right is a gray cylinder vs On the left is a gray cylinder ), in line with findings on the Imagen model [Clark and Jaini, 2023].

F Further analyses and plots

Diminishing returns of increasing the number of noise-timestep samples? We show in Fig. 9 that accuracy hits a plateau on Flickr30K text retrieval around a sample size of 100. We used 250 in main results Tab. 1 since we didn't test this on all datasets.

Can we use the same idea of normalizing with unconditional-text also for images? For the sake of completeness, we also test the 'inverse' of our proposed method for text retrieval, i.e. 'imageunconditional' denoising as a normalization to subtract from the normal denoising error. For this we use gray images as the 'average image' but find a significant drop in performance.

What happens if we apply the hard negative loss without threshold λ in Eq. (7) ? Not thresholding the negative loss (7) leads to more improvements on ARO (i.e. performance jumps to 63.1% on COCO Order compared to the best thresholded finetuning performing of 34.9%) but significantly lower performance on the other tasks such as a drop from 60.9% zero-shot on Pets (Tab. 2) to 53.2%. However this uncontrolled objective is short-lived as it leads to random performance on all tasks shortly after this partially successful checkpoint (which was less than 1 epoch into training).

(a) Flickr30K Text Retrieval

(b) ARO - Flickr30K Order

(c) ARO - VG Attribution

191

548

119

698

194

9

259

Diffusion ITM

CLIP

DiffusionITM

CLIP

BLIP

BLIP

157

297

306

245

209

198

354

249

158

Diffusion ITM

CLIP

Diffusion ITM

BLIP

CLIP

BLIP

259

434

203

639

264

58

579

324

Diffusion ITM

CLIP

Diffusion ITM

CLIP

BLIP

BLIP

0.9

0.8

1R@5

@1and

0.7

M

R

0.6

0.5

R@1

R@5

0

100

200

300

400

500

Sample size

50403

80

70

60

55.3

52.7

59.2

59.5

56.3

49.8

50.7

52.3

65.7

61.5

56.5

62.5

63.1

71.2

68.6

65.6

74.1

72.6

67.8

79.4

76.1

73.1

Model

·Diffusion ITM with Stable Diffusion 2.1

StableDiffusionXL

· Diffusion ITM with Stable Diffusion 1.5

46.1

41.6

41.9

35.8

36.3

24.8

23.9

33.9

31.6

31

38.2

30.1

26

26.9

32.3

29

27

20

10

15.7

13.9

13.6

75

8.3

6

10.7

9

0

FlickrText

Flickr Image

ARO VG Attr.

ARO VG Rel.

COCOOrder

CLEVR

Pets

SVOVerb

sVo Obj

ImageCoDe static

ImageCoDe video

Task

G More technical details

Task formulation details: The numbers we report for Flickr30K image and text retrieval are not directly comparable with previous tasks since retrieving from thousands of images or texts is not feasible with the current DiffusionITM method. Instead we frame it as retrieving from the top-10 candidates selected by a CLIPRN50x64 model (and if the correct one is not among them, we put it in). For all other tasks the setup is the same.

Hard negative finetuning: We adopt the exact list of negatives used in Yuksekgonul et al. [2023] which includes several text hard negatives (subsection 3.2 and several image hard negatives per image-text pair. The images are the nearest neighbor images based on CLIP embeddings.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Diffusion-ITM, a new method that directly adapts diffusion-based models to image-text matching tasks without retraining. Additionally, the authors collected a new benchmark called Generative-Discriminative Evaluation Benchmark (GDBench), which includes seven complex vision-and-language tasks and bias evaluation. The results show that Diffusion-ITM performs competitively on multiple tasks and surpasses CLIP. This paper underscores the significance of jointly considering discriminative and generative models and provides a new benchmark for future work.\n\nSTRENGTHS:\n* The paper presents a new method to adapt diffusion-based models for image-text matching tasks without retraining. This innovation has practical value to apply. The finding of the relative difference between the with and without text conditions (Fig. 3) is pretty interesting.\n* The authors thoroughly evaluate the proposed method on various vision-and-language tasks, offering insights into its performance and potential biases.\n* The introduction of new benchmark for image generation models over the discriminative tasks provides a useful tool for the research community and enables comparative analysis.\n* This paper is well written and organized with clear logic to read.\n\nWEAKNESSES:\nWeaknesses:\n* The Stable Diffusion includes a pre-trained CLIP text encoder and has itself been pre-trained on numerous image-text pairs. Therefore, Stable Diffusion has sufficiently learned the joint distribution of images and texts. The new finding is interesting but not surprising. \n* The performance of the proposed method on more challenging benchmarks, like Winogroud, is relatively poor, suggesting room for potential improvement.\n* The paper lacks theoretical analysis—since diffusion models are generative, applying them to a discriminative task would benefit from more theoretical insight.\n* This paper ignores time cost/efficiency analysis. The Stable Diffusion inference is slow. According to Eq. 5, how to assign the t in your method? How many steps are needed and why?\n* Minor issues, such as missing punctuation after equations, formatting errors, and a redundant space at the start of line 141, need to be addressed.\n\n\nQuestion:\n* Can the Stable Diffusion method be extended to more challenging image-text tasks like Visual Question Answering (VQA)?\n\nQUESTIONS:\nPlease see weakness\n\nLIMITATIONS:\nPlease see weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRecently diffusion-based text-to-image generation models have evolved rapidly, but it’s still challenging to evaluate them quantitatively in an efficient way.\n\nThis paper smartly converts evaluations of Stable Diffusion based image generation tasks as simpler image-text matching tasks (e.g. image-text retrieval), and proposes an image-text benchmark (GDBench) including 8 carefully-chosen tasks to measure generative model’s fine-grained properties (e.g. compositionality) and fairness. CLIP and Stable Diffusion models are evaluated on this new benchmark.\n\nOverall, this paper bridges discriminative and generative model evaluations, which could be inspiring to the image generation community and improve the quantitative evaluations of generative models.\n\nSTRENGTHS:\nTackling a challenging problem, i.e., efficient quantitative evaluation of image generation models, with a simple yet effective method (predicting the noise of the diffusion for image-text alignment).\n\nProposing an image-text benchmark (GDBench), covering diverse dimensions for measuring image generation (e.g. semantics, compositionality, fairness), which is beneficial for the image generation community.\n\nWell written and easy to read.\n\nWEAKNESSES:\nThe biggest concern on my side is the lack of enough evidence to show such a ITM eval aligns well with SD model quality, which IMO can’t be strongly supported by the evals to compare CLIP and SD models on retrieval numbers. If possible, I would suggest at least to prepare a few SD models with different capability (e.g. trained with different numbers of steps), and show they have different numbers on GDBench; bonus: give some qualitative examples for these models conditioned on the same text.\n\nQUESTIONS:\nAs in “Weaknesses”: how to prove the proposed method can measure the capability of SD models?\n\nTo measure discriminative vision-language models, both classification and retrieval tasks are usually used. Besides “pets”, could we also add some other common 0-shot classification tasks as well? E.g., INET (most common) or ObjectNet (to measure robustness).\n\nminor question/suggestion: this paper’s citation seems to use a different format (e.g. “[name year]”) than others (e.g. “[number]”)? The latter is more convenient to read, especially for prints.\n\nLIMITATIONS:\nThis paper has addressed model bias in one of GDBench tasks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how to use a pre-trained text-to-image generative diffusion model to do discriminative tasks like image and text matching. Building upon previous works [Li et al. 2023, Clark and Jaini 2023], this paper introduces two technical contributions: 1) Unconditional normalization largely improves image retrieval; 2) Tuning the diffusion model with hard negatives on better-quality data like MSCOCO. The paper also introduces a benchmark to evaluate different diffusion models by image and text matching performance, and specifically studies the biases of two versions of Stable Diffusion model.\n\nSTRENGTHS:\n1. The two technical contributions (unconditional normalization & tuning on MS COCO) are effective and well ablated.\n\n2. The introduced benchmark could be useful for the community to evaluate the prompt-following ability and biases new diffusion models.\n\n3. The paper studied bias in state-of-the-models, which should draw more attention to the community.\n\nWEAKNESSES:\n1. The generative-for-discriminative methods typically take many feedforwards of the diffusion model to evaluate a single sample. Although the paper briefly mentions its slow runtime at the end of the Appendix, I think the computational cost is an important factor of the proposed algorithm and should be rigorously presented and studied in the main text, especially when the paper proposes a benchmark - A clear study on the computational cost would be important to others who plan to evaluate on this benchmark.\n\n2. The GDBench covers several hard image-and-text matching datasets. However, I think results on other standard discriminative datasets would be helpful for the audience to understand the advantage of the proposed algorithm and compare to previous works. For example, how does HardNeg-DiffusionITM work on ImageNet compare to Diffusion Classifier [Li et al. 2023] / DiffusionITM (Ours) in Table 2(b)? I understand ImageNet is a 1000-way classification and would be a long time to run for this kind of methods (therefore studying runtime is important as mentioned above), but Diffusion Classifier [Li et al. 2023]  has already reported their performance on ImageNet and an apple-to-apple comparison would be very helpful. Same applied to the CLEVR dataset. Another question is how tuning on MSCOCO works for image-text matching on MSCOCO?\n\n3. I am not sure what is the use case to introduce MSCOCO tuning into Diffusion Classifier. It indeed improves the performance of image and text matching compare to the plain one, but all these methods are too slow (or not good enough) to be practically applied. On the other hand, tuning the weights of diffusion models prevents it from serving as a tool for evaluating diffusion models. Then in what case do we need MSCOCO tuning?\n\nQUESTIONS:\n1. I am not sure how much MSCOCO HardNeg is clearly better than MSCOCO NoNeg - To me, MSCOCO NoNeg seems to be even better than HardNeg quantitatively, or at least on par with in Table 2. How do the authors land with HardNeg as the default setting?\n\n2. Why is Winground's random accuracy (25%) higher than the algorithms' accuracy in Table 1(a)?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the discriminative capabilities of diffusion models measured by image-text matching. A new matching score computation enables text-based image retrieval beyond simply text retrieval in existing works. A new benchmark, augmented from existing image-text benchmarks, is proposed for researchers to evaluate current diffusion models.\n\nSTRENGTHS:\n1. DiffusionITM enables image retrieval by making simple and minimal changes to existing text retrieval methods. The method generates good results on simpler datasets and performs better than random on challenging ones.\n2. The proposed benchmark covers diverse aspects but is also optimized for lightweight testing. Differences in a few methods have been demonstrated with the benchmark.\n\nWEAKNESSES:\n1. Evaluation on Flickr with CLIP retrieved negatives. This process might introduce bias towards or against CLIP based image diffusion models. For example, in Table 1a, CLIPRN50x64 obtains an accuracy of 71.9. According to line 545, does it mean that 28.1% of the cases CLIP does not select the correct positive? In order to fix this, maybe an evaluation against the original model-free Flickr retrieval metric is needed, e.g. measuring if CLIPRN50x64 or CLIP based method has an advantage/disadvantage compared with other methods, measured on the new simplified set vs. the original set. Similarly, the CLIP-based HardNeg training might be exploiting the bias, besides learning useful discrimination skills.\n2. It is preferred to compare different kinds of diffusion model variants in the proposed benchmark and show some insights there. For example, one might see an advantage of T5 based text encoder in grammar/text related metrics, or one could study pixel based diffusion vs. latent diffusion etc. This comparison is challenging in obtaining the models but can provide more value to the community.\n3. The current method makes an assumption that DiffusionITM, based on eq. 2 and 5, is a great way of ITM and will be the case for a while. However, given the field advancing so rapidly, researchers might discover a better zero-shot matching method that could infer quickly and outperform CLIP consistently for example. Then, at that time, why would researchers not adopt the original retrieval tasks, e.g. on Flickr with all images, but adopt GDBench with 10 images?\n4. It is claimed that the proposed tuning strategy keeps the generation capability. Examples are shown in Figure 5 and positive human evaluation is shown, but is it possible to evaluate existing standard metrics and see the difference? FID might not apply as the tuning happens on COCO. One might even see better generation quality with discriminative tuning applied.\n5. How is GDBench different from an ensemble of image-text matching evaluation metrics, except that it is made light-weight, e.g. measuring 10 samples. It seems DiffusionITM is a general enough method that can be evaluated on theoretically any image-text matching tasks. From another perspective, is it possible to show correlation of the GDBench metrics with real compositional generation quality, e.g. measured by human evaluation.\n6. Given the limited samples in some of the evaluation datasets, what are the standard deviation for different runs? From Figure 9, it looks noisy.\n\nQUESTIONS:\nTable 1 typo: Difussion.\nAnd see the weaknesses section for questions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes novel techniques for improving the performance of text-to-image diffusion models on zero-shot image-text matching tasks. They first propose subtracting the unconditional denoising error $\\|\\epsilon - \\epsilon_\\theta(x_t, t)\\|_2^2$, which reduces the problem in image retrieval where one image is a priori far more likely under all captions. The also propose using a hard negative finetuning loss to increase the denoising error on hard negatives, which improves its discriminative (and generative) capabilities. Finally, they assemble  GDBench, a benchmark of 7 existing datasets that they use to quantify different aspects of text-image alignment performance. They find that their proposed model significantly improves on prior work that uses text-to-image diffusion models (Diffusion Classifier), and is competitive with CLIP ResNet50x64. They also benchmark model bias and find that Stable Diffusion 2.1 is less biased than Stable Diffusion 1.5 and CLIP RN50x64.\n\nSTRENGTHS:\n- The paper proposes 2 novel modifications that significantly improve performance over Diffusion Classifier (a prior work on using diffusion models for text-image matching tasks). \n  - The hard negative finetuning method in particular is intuitive yet highly useful. It is also evaluated extremely thoroughly in its effect on both discriminative and generative behavior. \n- Authors also gather a set of 8 image-text-matching tasks into a proposed GDBench benchmark for easily measuring and comparing the performance of different ITM methods. \n- Authors do a thorough analysis of the bias of Diffusion ITM, across versions of Stable Diffusion, as well as in comparison to a few CLIP models. \n- Extensive ablations and experiments on variants of HardNeg, comparison between CLIP, BLIP, and Diffusion ITM, and the relationship between the number of timesteps and the performance of the method.\n\nWEAKNESSES:\n- Computational: The proposed method runs in $O(n)$, where $n$ is the number of candidate images or text captions to choose from. This becomes highly impractical for real retrieval applications, compared to approaches like CLIP where fast approximate nearest neighbor solutions work extremely efficiently. \n- Baselines: while the method does outperform previous work on using diffusion models (e.g., Diffusion Classifier), it doesn't compare against the strongest discriminative baselines, namely OpenCLIP Vit-H/14. This is fair since Stable Diffusion 2 actually uses this model as its text encoder. The paper currently only compares against CLIP ResNet50x64, which is quite weak in comparison.\n\nQUESTIONS:\n**Main questions/issues**: \n- How well does OpenCLIP ViT-H/14 do on these ITM tasks? This is the most relevant discriminative baseline since SD 2 uses its text encoder. \n- The Flickr30K benchmark is biased against the CLIP model used to produce it. The CLIP model is used to produce the top 10 candidates, and *the correct candidate is added if it's not among the top 10*. If the model being evaluated is CLIP, it has to choose between 9-10 hard negatives, whereas other models will receive a boost from a weak ensembling effect with CLIP. I'd suggest another way of creating the 10 candidates without correlation to any model output. \n- Equation 6: is there a major typo here? This appears to still minimize $\\mathcal L_{neg}$.\n- L 246: Hard negative finetuning uses $\\lambda=-1$? Why is it negative when it's only used for $clip(\\mathcal L_{neg}, |\\lambda \\mathcal L_{pos}|)$? Does this mean that $\\mathcal L_{neg}$ is clipped to the range $[-\\mathcal L_{pos}, \\mathcal L_{pos}]$? Something is wrong here.\n\n**Minor questions/comments**: \n- Table 1b: Where does the 37.5 score come from with Diffusion Classifier/DiffusionITM on Winoground text? The Diffusion Classifier paper reports 34.0. \n- Page 6, footnote 2: This is not too important, since the prompt is fixed across methods, but Pets prompt template used in the CLIP paper is \"a photo of a {}, a type of pet.\" This probably increases scores across the board for all methods. \n- L240-241: \"we drop the complicated procedure that iteratively prunes classes after a number of noise-timestep samples\" -- Diffusion Classifier doesn't use its adaptive strategy for its ITM experiments either, so its evaluation strategy matches DIffusionITM here.  \n- Appendix G (runtime) mentions that \"batch size has diminishing returns in terms of runtime after batchsize=4.\" However, based on the Diffusion Classifier github, using FlashAttention + FP16 + `torch.inference_mode()` + batchsize=32 achieves about 80 evals/s on an A6000. I'd suggest double checking your implementation. Computation speed should be an issue overall with this method, but not to this level.\n\nLIMITATIONS:\nYes, authors address the main limitations of their paper and method.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}