Are Diffusion Models Vision-And-Language Reasoners?
Authors: Benno Krojer
Elinor Poole-Dayan
Vikram Voleti
Christopher Pal
Siva Reddy
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion model
automatic evaluation
vision-and-language
compositionality
TL;DR: We propose the first method to transform a diffusion model for image-text-matching tasks which allows a new form of automatic evaluation.

Abstract: Text-conditioned image generation models have recently shown immense qualitative success using denoising diffusion processes. However, unlike discriminative vision-and-language models, it is a non-trivial task to subject these diffusion-based generative models to automatic fine-grained quantitative evaluation of high-level phenomena such as compositionality.
Towards this goal, we perform two innovations. First, we transform diffusion-based models (in our case, Stable Diffusion) for any image-text matching (ITM) task using a novel method called DiffusionITM.
Second, we introduce the Generative-Discriminative Evaluation Benchmark (GDBench) benchmark with 7 complex vision-and-language tasks, bias evaluation and detailed analysis.
We find that Stable Diffusion + DiffusionITM is competitive on many tasks and outperforms CLIP on compositional tasks like like CLEVR and Winoground.
We further boost its compositional performance with a transfer setup by fine-tuning on MS-COCO while retaining generative capabilities. 
We also measure the stereotypical bias in diffusion models, and find that Stable Diffusion 2.1 is, for the most part, less biased than Stable Diffusion 1.5.
Overall, our results point in an exciting direction bringing discriminative and generative model evaluation closer. We will release code and benchmark setup soon.

Are Diffusion Models Vision-And-Language
Reasoners?
Benno Krojer
Elinor Poole-Dayan McGill University
Vikram Voleti
Mila & McGill University
Mila & University of Montréal Stability AI
benno. penno.kroje eereerrlllll.m.e. quebec
Christopher Pal
Siva Reddy
Mila & Polytechnique Montréal
Mila & McGill University
Canada CIFAR AI Chair ServiceNow Research
Facebook CIFAR AI Chair
ServiceNow Research
Abstract
Text-conditioned image generation models have recently shown immense qualita- live success using denoising diffusion processes. However, unlike discriminative jision-and-languagl models, it is non-trivial task to subject these diffusion- generative models automatic fine-grained quantitative evaluation of high-level phenomena such as compositionality. Towards this goal, we perform two innova- tions. First, we transform diffusion-based models our case, Stable Diffusion) for any image-text matching (ITM) task using novel method called Diffusiio-- ITM. Second, we introduce the Gennerative-Discriinnnde (GDBench) benchmark with complex visionaand-langugg tasks, bias evaluation and detailed analysis. We find that Stable Diffusion + Diffusion/TM is competitive many tasks and outperforms CLIP on compositional tasks like like CLEVR and Winoground. We further boost its compositional performance with a transfer setup by fine-tuning MS-COCO while retaining generative capabilities. We also mea- sure the stereotypical bias in diffusion models, and find that Stable Diffusion 2.1 is, for the most part, less biased than Stable Diffusion 1.5. Overall, our results point in an exciting direction bringing discriminative and generative model evaluation closer. We releasing code and benchmark setup. !
Evaluation Benchmark
Introduction
Text-to-image generation is rapidly advancing. Generated images not only highly realistic in various styles, but also reflect the compositional structure open-ended text prompts [Chang et al., 2023, Saharia al., 2022, Li et al., 2022b). In this work, evaluate sagguac-coonitionee generative image models on discriminative tasks to shed light on their fine-grained understanding of vision and language. A generative objective trains model understand how various objects and parts compose together, and often brings non-trivial emergent capabilities with it such as latent interpolation of composite concepts [Brock al., 2019, Rombach et al., 2022]. On the other hand, discriminative siiio-andd-anguusee models need only focus on the minimal information required to solve their discriminative task, which could often be spurious correlations that don't generalize [Agrawal et al., 2016]. Such erroneous understanding is then exposed downstream on image-text matching tasks purposefully designed catch it, such as image/text retrieval using Winoground [Thrush et al., 2022], ARO [Yuksekgonul et al., 2023], or ImageCoDe [Krojer et al., 2022]. Following these benchmarks' releases, there has been growing focus in the siiio-and--anguuee community to fix these problems.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

"two dogs play with orange tall grass"
W
Diffusion Model
Eg(X, f, w)
11e 4, w) - =el2
X
Score ITM
Diffusion
Ea(X, f)
||Eo(x, 1) =l12
Model
E
Figure I: DiffusionITM allows us to apply diffusion model to any image-text-nee mmage-fex--matching task. It overcomes the asymmetry between image and text retrieval that previously led to random performance on image retrieval via unconditional normalization: An image is selected based the lowest noise prediction error when conditioned on the text (upper part figure) which is normalized by the noise prediction
without eext-oodiiioning (lower part). With this general method the image generation community can benchmark their models on complex vision-and-langunee tasks such as Winoground [Thrush et al., 2022] or ImageCoDe [Krojer et al., 2022].
80
Model
Difresion ITM with suble Dffiuste 1.5 Diftusion Subble Diffusion
60
Arcunn
40 His
20
BuB
Flicks Text
Plicks Imsaee
ARO Rel. ANDO VG Atts.
Flicke Chierr
CLEVE
Pert
SVD Verb
SVD Sobj
Imager suite
Imaientioo offee video
Whaegrooad group
Wileog poodgroddd
Wipegrood tmag
Task
Figure 2: Progress Stable Diffusion from 1.5 to 2.1 on GDBench tasks. GDBench allows fine- grained comparison of models.
We hypothesize that generative model trained to synthesize compositional data is capable of understanding the complexities required to solve hard image-text-mant metching tasks. To this end, we transform a text-to-image generative model for zero-shot image-text matching. and intro- duce Diffission Image-Text Matcher (DiffusionITM: Fig. 1). In this work, use Stable Diffusion (SD) [Rombach et al., 2022] as the text-to-image model, but any other diffusion model could be used. DiffusionITTM achieves competitive zero-shot performance both image and text retrieval (Tab. I).
The naive approach for image retrieval given a text prompt would be pick the image for which SD gives the least noise prediction error, and vice versa for text retrieval. While this works well for text retrieval [Li et al., 2023], we show achieves random performance on image retrieval (Tab. 1). Our main insight explaining this discrepancy is that the model's 's success at denoising depends primarily on the visual properties of the scene, rather than equally on visuals and text (Fig. 3). Therefore, such a model has to select among several images given text prompt, it will have the lowest noise prediction error for a visually familiar image regardless of the text prompt. To address this, we compute the eтror relative to the unconditional (i.e. text) eтror (Fig. I). Our method outperforms existing diffusion- based discriminative methods [Li et al., 2023, Clark and Jaini, 2023] (see Tab. 1).
Since the original generative pretraining objective of noise prediction is far from the ultimate downstream task of image-text-matching we explore a novel discriminative finetuning scheme with hard negative image-text-paie on MS-COCO [Lin et al., 2014]. We find that this transfers well to other datasets, and improves discriminative performance (Tab. 1) as well generated images from DrawBench prompts (Saharia et al., 2022).

---

Finally, we present the GDBench to foster research progress on image generation. Our DiffusiinITTM method enables a new automatic, fine-grain graengrridd and downstream way evaluate diverse skills in text-conditioned image generation. Thus, once we cast an image generation model as discriminator, we can draw from the rich history of diagnostic datasets and detailed analyses in the vision-and- language literature to analyze the model's 's reasoning ability. Evaluation of image generation has been notoriously difficult, and recent proposals advocate using secondary model probes on generated images to test for fine-grained ability [Hu et al., 2023, Cho et al., 2022]. In contrast, we directly evaluate diffusion models on downstream tasks without requiring any probes. In the same spirit as GLUE [Wang et al., 2018], systematically select 7 image- mett-mt tasks covering diverse reasoning skills from traditional image retrieval to diagnostic tasks (i.e. compositionality) In addition, GDBench also includes a bias evaluation dataset which we use for social bias analysis.
GDBench allows head-on comparison between generative models, as well as with discriminative models like CLIP [Radford et al., 2021]. Our results are as follows:
Stable Diffusion (SD) vs. CLIP: SD is competitive with CLIP on many tasks, and outperforms on challenging compositional tasks like CLEVR and Winoground Text (Tab. 1). Further tuning of SD on
MS-COCO with hard negatives outperforms vanilla SD on both image and text retrieval (Tab. 2). SD has lower stereotypical bias than CLIP (Tab. 3).
Stable Diffusion (SD) 1.5 2.1: We observe varying degrees of progress, but overall SD 2.1 outperforms SD 1.5 (Fig. 2). SD 2.1 is less biased than SD 1.5, contrary to the common trend of increased bias in larger, stronger models [Nadeem et al., 2021].
Remarkably, improved discriminative performance via finetuning on MS-COCO also leads to improved high-level understanding of image generation. We find higher image-text- alignment on DrawBench [Saharia et al., 2022] prompts compared to vanilla SD (Appendix B).
Image generation:
2 Related Work
Uision-And-Languaee Language Understanding: Widely popular tasks for vision vision-and--enn and-language understanding are often framed discriminative tasks, such as image retrieval [Miech et al., 2021]. VQA [Agrawal et al.. 2015] or REC [Yu et al., 2016]. We focus on score to an image-text-pain due their general applicability and simplicity. Noisy image-text pairs are used for large-scale ITM training using contrastive learning [Radford et al., 2021], and has been traditionally tackled either via dual encoders like CLIP [Radford et al., 2021], or models with richer cross- interaction like BLIP [Li et al., 2022a]. At the same time, ITM allows probing models in controlled manner with simple metrics [Thrush et al., 2022, Hendricks and Nematzadeh, 2021], as opposed to more complex metrics on generative tasks [Hessel et al., 2021, Saharia et al., 2022]. There has been growing interest on diagnostic benchmarks for compositionality that use hard negatives [Yuksekgonul et al., 2023, Thrush et al., 2022, Krojer et al., 2022. Hendricks and Nematzadeh, 2021, Lewis et al., 2022, Parcalabescu et al., 2022]. This rich literature can now be transferred to the world of image generation with our GDBench benchmark, enabling fine-grained analysis and benchmarking of text-conditioned image generation.
(ITM) tasks (assigning
Diffusion Models: Text-conditioned diffusion has shown remark- able advancements not only in image quality but also image- text aligmment [Rombach et al., 2022, Saharia et al., 2022]. A natural next question is how to leverage these abilities for other tasks [Burgert et al., 2022]. Two concurrent studies use similar methods to our diffusion//TY in a more restricted set- ting: Disfussion Classifier [Li et al., 2023] performs image classification using SD by selecting the class with the lowest noise prediction error; Clark and Jaini [2023] puts more focus on simestep-weighting and uses Imagen [Saharia et al., 2022] instead of SD. However, they only show competitive results for text retrieval, emphasizing image classification as a special case. Many vision-and- and-language rea- soning tasks are also framed as image retrieval [Krojer et al., 2022, Hendricks and Nematzadeh, 2021, Thrush et al., 2022]. In this work, we tackle the broader scope of vision-and- and-aaguag understanding, generalize our method to image retrieval, and study hard-negative fine-tuning and transfer.
Repurposing
Evaluation of Image Generation: Image generation is traditionally evaluated along the two axes of image quality and image-text text alignment, using metrics based on individual examples. The recently proposed TIFA metric [Hu et al., 2023) relies on an additional VQA model to answer a set of L.M-- generated questions about the generated image. More traditional metrics include FID [Heusel et al.,
3

---

2017] for image quality; CLIPScore [Hessel et al., 2021, Kim et al., 2022] for image-text alignment based on -LIP-embedding; object-centric metrics [Hinz al., 2020, Cho et al., 2022] leveraging object detectors such as DETR [Carion et al., 2020]; and caption-based metrics [Hong et al., 201 88 like BLEU on captions of the generated image. In contrast, GDBench is not metric on individual examples, but rather holistic evaluation framework. GDBench does not require another large model (e.g. VQA) for evaluation, and can be run on many diverse datasets.
Bias in Image Generation Models: It is well known that LLMs learn and amplify harmful biases present within text training corpora [Caliskan et al., 2017]. Due to the lack of automatic evaluation techniques for generative models, bias investigation has mostly focused on discriminative vision- and-language (VL) models [Srinivasan and Bisk, 2022, Janghorbani and De Melo, 2023]. Only few works have tackled bias in recent text-to-image models [Luccioni et al., 2023, Cho et al., 2022] and to our knowledge only Luccioni et [2023] focus on bias alone: They quantify social biases by generating images over several social groups (ethnicity and gender) and measuring their variation over selected attributes (gendered adjectives and professions). They found that SD 1.4 and 2.0 are biased towards groups "associated with whiteness and masculinity" across target attributes, and that SD 2.0 was more biased than 1.4. While their methods are thorough and work for black-box systems, the evaluation is quite time consuming and manual.
3 Our Approach to Image-Tex: Text Matching with Diffusion Models
3.1 Diffusion Image-Text Matching: Overcoming the modality asymmetry for image retrieval
We present our method Diffusion Image-Text Matching (ITM). Our goal is assign score to an image(x))tex((w) pair (x, w) which is broadly useful for downstream applications. We provide (x, w) to the diffusion model and task it "edit" the image according the text. Our main intuition is if the image is not described by the text, a lot of edits are needed to fit the text, in which case it gets a low score, and vice-versa. See Appendix C for visualization.
Text-conditioned Diffusion: The objective of diffusion models is to denoise an image x by predicting the noise added to its clean version, conditioned on text w and noise level t:
Diffusion loss: Ex,e,t
[||e- en (x, I, w)
Intuitively, the predicted noise is farther from the true noise when the image-text do not fit together. To transform this for ITM tasks, the sample with the lowest L2-distance of predicted and true noise Li et al. [2023] and Clark and Jaini [2023]
could be used to select among a set of image-text-pane (concurrent works) have focused primarily on text retrieval (with classification as special case), where the model selects from a number of texts (or class names for classification):
Text retrieval:
arg min EF, [[le e
(2)
w
However, naively applying this in practice would imply sampling a different for each pair (x, w) to reduce the variance in 1.2-distance. Li et al. [2023] a) sample many (hundreds!) noise-timestep pairs (e, 1) uniformly, and b) crucially keep the sampled (c. I) constant across different w when calculating Equation 2. Finally, the guidance scale is kept O, thereby discarding unconditional noise prediction. This could be repurposed to perform image retrieval by iterating over images instead of text:
Naive image retrieval:
arg min E., [||e - co (x, t, w) [[
We observe that while this results in SD as a strong text retriever, it achieves random chance on all image retrieval tasks! Interestingly, Li et al. [2023] observed that Bayesian posterior from a discrete-label conditionl generative model can be approximated using the analysis:
[|ce - co (X1, ct)]] + const.
((c))o(x(cc)
exp
E,,e
Po (c: x) (,p(c))p(x cj)
(4)
Σ, exp
-B,c
||c co (xx, c)) const.)
Li et al. [2023] use pairwise samples between pairs of labels e; and cj, to produce a low complexity approximate computation reminiscent of paired differences tests statistics. Their analysis is based on marginal across all possible discrete label captions with respect to image In

---

Image 2
random captions caption unconditional
Image
Denaising Error
Figure 3: Denoising losses for two similar Flickr30K images Imagel and Image2, and Caption I of Image1. The absolute denoising error of
is smaller than that of I hence Diffusion Classifier would have erroneously picked Image? for Caption Whereas, the difference between the errors for Imagel-Caption and Imagel- -unnenndtiini is greater than between Image2-Caption and Image2-Unconditional so our approach would correctly pick Imagel.
contrast, in our approach, we replace the marginal in the denominator with simply a sample from the unconditional diffusion model for p(x), and operate in log space. The intuition behind our approach is that taking an integral over all possible caption strings (w) is equivalent to simply the unconditional, since the "caption" dimension is (implicitly) marginalized out. Fig. 3 illustrates another view of this intuition that for a correct text-image pair than for all mismatched text-image pairs for the same image (Imagel). Moreover, for incorrect but similar image (Image2), the denoising error for Caption is close to random, but could be less than that of Image-CCaption in absolute value. The incorrect Image2 would be selected regardless of the text since is visually easier to denoise. Hence, the denoising error depends almost entirely on the image, independent of text conditioning ("modality asymmetry").
)), the denoising error visibly smaller
This analysis naturally leads to the intuitive solution of normalizing the error by the unconditional (no text) error. After all, only care about the relative difference of how much casier harder becomes to denoise the image with a given text relative to when no text is given (see Fig. 1):
Image retrieval:
arg min E,,, [[lle co(x, t, w) - ||c (x, +)]]
(5)
3.2 HardNeg- DitffusioffiiiMMM: Tuning with compositional hard negatives and transfer
Our goal is to transform diffusion-basee models for discriminative image-text-matching (ITM). However, the denoising diffusion objective only considers positive image-text pairs, and the large pre- vre-training corpus LAION [Schuhmann et al., 2021] contains many noisy/simple examples, not conductive complex linguistic reasoning. contrast, models specifically dedicated vision-and- language reasoning such CLIP [Radford et al., 2021] or BI.IP [Li et al., 2022a] were pre-trained with negatives and, in the case of BLIP, had access to high-quality curated image-text data. Previous baselines [Li et al., 2023] ditched the generative objective and turned it fully discriminative by finetuning a ResNet on top of the frozen mid-layer features of the U-Net for each dataset separately. We instead adopt parameter-efficient finetuning with LORA layers [Hu al., 2022] that are added to the cross-attention from U-Net to the text, so as not to deviate too far from pretraining representations.
We address the lack of high-quality image-text dext-data by fine-tuning the diffusion model on MS-COCO (109K examples) with the standard diffusion objective (see Equation 1). As MS-COCO contains diverse high-qualit image-text pairs, we finetune only once, and evaluate using GDBench tasks. This could be thought of as second limited pre-training.
We address the lack of negative examples by adopting the hard negatives from Yuksekgonul et al. [2023] MS-COCO: swapped text elements of the same part-of-speech (e.g. "Men keep watch a herd of goats" "Goats keep watch on a herd of men"), and CLIP-based image hard negatives. The naive approach would be to minimize the noise prediction error on positive pairs (x, W and maximize for negative pairs (x, Wneg). However, if this inverse loss were applied unhinged with potentially infinite gains, would lead the model predicting non-sense for everything. Therefore we threshold the hard-negativ eιror at relative scaling factor À of the value of the positive error:
L = Lpos + clip (Lnegy (Xp where
pos = E Ex,c,t [||c (x t, W pl?) Lneg = -xxeet [[|c- co co(xtt, Wneg)ll
We choose relative thresholding since we want to ensure that the model never deviates too much from its original objective of noise prediction from positive prompts. Hence Eney is clipped between
5

---

Flickr30K: lody in rilin on reonter (efl)) and CLIP- basod hard negative (right)
ARO: munn wearing the whirt (positive) and The shirt is wearing the mui (hard negative)
planto of riglish nooker spaniel
lorge sphere and small cylinder (pusilive and A small
sphere and large rylinder (nega- tive)
SVO: Person rtanding window sttting window
ImageCCDe::
There yellow bonh hondy (lefl) and hard negutive (right
Wingground: Some planty surround- lighttuull fighnioll swrroonding some plants (right)
Biax Christian (leff), Jewish (right). unhiased model should "pleusant" and "unplemeant words equaily both groups.
(right
Figure 4: Examples from the datasets in GDBench.
-- Cpos po Unlike CLIP, we cannot include a large number of hard negatives in our batches since diffusionITM encodes image and text together. The resulting model, HardNeg- Daaiieeg-iffusio/TMMM still evaluated in a zero-shot fashion on the target evaluation tasks, i.e., how well does Diffusion/TM trained on MS-COCO transfer to target tasks.
Data: The GDBench Benchmark
There is a fundamental need to measure downstream performance of diffusion-based generative models on a wide range vision-and-lang and-language reasoning tasks, to facilitate quick improvement and progress tracking. This has worked in the past, ie. with the NLP GLUE benchmark [Wang et al., 2018]. With this motivation, we introduce GDBench, a benchmark of eight diverse image- text-matching (ITM) tasks explore many types of vision-and-land ladian-aa reasoning. These include 7 ability-centric and 1 bias dataset for the image generation community (examples shown in Fig. 4). Additionally, most GDBench tasks have further fine- ame-grained scores sub-phenomena without requiring manual inspection, as is the case with evaluating generative models (Saharia et al., 2022]. ITM reasoning benchmark offers simplicity and surprising amount of diversity. After all, ITM has become a standard paradigm for diagnostic Dision-and-languaal datasets (see task list below) and therefore allows interpretable evaluation many downstream skills. GDßench is similar to the spirit of downstream evaluation of generative models such as in TIFA evaluation [Hu et al., 2023] which makes use of secondary model like VQA on the generated images and shows that downstream performance correlates with image quality. Whereas, we stay closer to the generative realm and evaluate generative models on several discriminative tasks without the need for secondary models. Below we introduce each dataset, asking "What phenomena does it cover that others do not?"
Flickr30K [Young al., 2014] is well-established open-ended image and text retrieval dataset, captioning diverse scenes involving people. Note that we changed the task setup to reduce computation overhead: A model has retrieve among the 10 most similar examples based on the CLIP embedding space. While this might not be fair towards our CL.IP baselines, we chose this setup primarily to study if SD could be useful a second-stage slow retriever, after a fast retriever like narrows down the candidates [Miech et al., 2021]. Both Winoground [Thrush et al., 2022] and ARO [Yuksekgonul al., 2023] are diagnostic benchmarks for compositionality. Winoground is carefully curated the cost of only 400 examples and many SOTA models have not reached significantly beyond random chance. ARO is automatically generated on top of Flickr30K, MS- MS-COCO, and others with only compositional hard text negatives. ImageCoDe [Krojer et al., 2022] is an image retrieval task focusing on highly similar images with complex pragmatic captions crowdsoureed from a guessing game. SVO [Hendricks and Nematzadeh, 2021] disentangles performance along different parts-of-speech by pairing images that differ only subject, object or verb. Lewis al. [2022] introduced a diagnostic controllable benchmark based on simple synthetic CLEVR2 images of 3D shapes, thereby isolating various phenomena like attribute binding or spatial relations. Finally, we include Pets [Parkhi et al.,
We generate CLEVR images and captions ourselves and they are therefore not exactly comparable with the previous task formulation [Lewis al., 2022, Clark and Jaini, 2023].

---

Table 1: Benchmarking Diffusion ITM with vanilla SD and hard-negative fine-tuning on MS- COCO on GDBench. Diffusion Classifier performs around random chance on image retrieval. Hard negative transfer finetuning significantly improves on both.
(a) Accuracy of DiffusionITM and baselines GDBench image retrieval tasks.
SVO
ImageCoDe
Winoground
Flickr Img
Verb
Subj
Obj
Static
Video
Image
CLIP RN50x64
71.9
82.2
85.8
89.3
51.6
23.5
13.8
OpenCLIP VIT-L/14
79.9
85.6
90.7
92.8
69.8
26.0
11.25
Diffusion Classifier
11.1
50.6
55.0
48.7
8.9
9.0
0.25
DiffusionITM (Ours)
61.5
77.3
80.5
86.2
42.5
21.1
10.2
HardNeg-Difftuionl DifftusionIIM (Ours)
66.1
77.8
81.3
84.7
44.9
21.0
12.3
(h) Accuracy of DiffusionITM and baselines on GDBench text retrieval tasks.
ARO
CLEVR
Winoground
Flickr Txt VG Attr VG Rel. coco Oid. Flickr Ord.
Pets
Text
CLIP RN50x64
65.8
62.7
50.8
52.0
58.8
59.5
88.4
26.3
OpenCLIP VIT- 14/L
71.3
59.2
50.3
30.8
39.8
64.3
89.9
30.25
Diffusion Classifier
69.8
62.9
50.0
23.5
33.2
67.9
79.7
37.5
DiffusionITM (Ours)
MaANNeg-DiffusionITY (Ours)
73.5
67.6
52.3
34.4
48.6
73.3
81.3
39.5
2012] as smaller quick-to-run image classification dataset. In contrast to linguistically complex GDBench tasks, Pets covers the complementary skill of fine-grained recognition (37 animals).
Measuring Bias: Bias evaluation is essential large-scale models increasingly suffer from harmful biases that can impact downstream tasks. Diffusion/T' allows automatic, quantitative bias evaluation with bias datasets intended for discriminative -ainn-and-langguge models. We utilize the dataset from Janghorbani and De Melo [2023] and investigate three different types of social biases: religious, nationality, and sexual orientation. Quantitatively, bias present when there substantially stronger association of one target group to pleasant attributes compared to unpleasant attributes over another group, as measured by some notion of distance or similarity. The target groups are represented by sets of images X' Y and the attributes are captured by sets of words AT, B"" Bias is measured by the following normalized association score, d, called the effect size:*
d(X',Y', Y¹ AT, AB) mean xx (x,,, A, B)
mea (/ A, B) stdev (i,,, A.
(8)
stdev 4/(1, A, B)
where (i, mcan o (i, mcan aEA
(i,
BEB
and (:...) our proposed DiffusionITM score, or, in the case of CLIP. cosine similarity.
More concretely, in the case of Religion, the image sets might be X = Christians, Y = Muslims and the attribute word sets would be A = ("joy", "trust", ""good"....) B ("corrupt", "vulgar"., "bad". ...}. Here, a positive effect size means that it is biased towards Christians and a negative effect size means is biased towards Muslims. We provide more details under limitations in Appendix A.
We chose Pets over ImageNet with 1000 classes since we want GDBench to be light to run. Following Radford et al. [2021], we use the text prompt "an image of X"
*We use permutation test to compute significance of effect size scores Caliskan et al. [2017].
*Random accuracy: 10% (Flickr, ImageCoDe), 50% (ARO Rel/Attr. SVO, CLEVR), 25% (Winoground), 20% (ARO Order). 2.7% (Pets).

---

Experiments and Results
Our main two findings are summarized in Tab. 1: First, zero-shot DiffusionITM achieves performance near CLIP on image retrieval (Tab. 1a), overcoming the dose-to-randon performance of Diffusion Classifier [Li et al., 2023]. Second, our best hard negative transfer-finetuning strategy improves performance across the board, on both image and text retrieval.
Hyperparameters: Based on ablations in Li et al. [2023], we adopt most of their setup but aim for more simplicity: Timesteps are sampled uniformly from [0, 1000]. guidance scale kept at 0, but we drop the complicated procedure that iteratively prunes classes after number of noise-timester samples (e, t). Instead we keep samples constant at 250 for the main zero-shot experiments in Tab. and reduce it to much more feasible number of 10 samples for other experiments. 6 This is aligned with our goal that GDBench should be easily adopted by researchers evaluate their image generation method. As shown in Appendix Fig. 9, performance is not at its maximum with few samples but the trends can be studied in the same way when comparing models along different tasks. We adopt the common CLIPP RN50x64 baseline and OpenCLIP ViT-L/14 for fair comparison since SD 2.1's text backbone is from the latter. We fine-tune on the MS-COCO hard negative training set (Yuksekgonul et al., 2023] with Ir = 1e 4, X = 1.0 and batchsize 112. We select checkpoint after epochs based on hard negative validation. Runtime: With 10 noise samples per image-text-pall evaluation on Flickr30K Text Retrieval validation takes 68 minutes
single NVIDIA RTX A6000 GPU (compared to around 4 minutes with OpenCLIP ViT-L/14). We point to Li et al. [2023] for an in-depth analysis of runtime. We emphasize that we envision future stronger SD SD-based discriminators as slow retrievers that are applied to a small set of candidates provided by a fast retriever [Miech et al., 2021], as well the benefit of automatic evaluation.
Table 2: Comparison of finetuning approaches on (top) image and (bottom) text retrieval, with only 10 sampling steps due to runtime feasibility (lower performance than Tab. I but same trends).
(a) Image Retrieval
SVO
ImageCoDe
Winoground
Flickr Img
Verb
Subj
Obj
Static
Video
Image
Vanilla SD
46.1
71.2
74.1
79.4
30.1
15.7
9.0
MS-COCO NoNeg
48.2
71.1
74.7
76.9
29.7
16.1
10.3
MS-COCO RandNegra
47.7
71.5
73.8
77.5
28.3
16.0
10.7
+ MS-COCO HardNegro
47.0
71.3
74.1
76.8
30.6
16.2
9.6
MS-COCO HardNegtme
52.9
73.1
76.1
79.4
34.6
17.2
10.5
Hardra + RandTa + Hard
49.4
71.7
75.4
78.4
31.9
16.6
9.8
(b) Text Retrieval
ARO
CLEVR
Winoground
Flickr Txt vG Attr. VG Rel.
COCO Order Flickr Order
avg
Pets
Text
Vanilla SD
55.3
59.2
49.8
24.8
31.6
65.7
60.9
32.3
MS-COCO NoNeg
62.2
62.3
53.2
33.6
42.9
67.1
70.0
35.0
MS-COCO RandNegna
62.2
61.6
53.1
34.0
42.4
67.6
70.2
32.2
MS-COCO HardNegıx
60.9
62.2
52.9
34.9
44.0
68.5
69.4
33.7
MS-COCO HardNegims
61.1
61.1
52.4
30.6
37.6
67.4
71.0
33.7
Hardya + Randya+ Hardimg
61.7
62.0
53.1
33.9
41.2
67.0
71.0
30.8
DiffusionITM performance: Our DiffusionITM significantly outperforms Diffiusion Classifier on all image retrieval tasks (Tab. 1a) while CLIP still outperforms Diffusion/TM. However on text retrieval, Diffusion/IIM even outperforms CLIP on some tasks (Tab. lb), most notably compositionallty-foouee -ocuuse Winoground Text and CLEVR by a large margin. Here DiffusionITM is clearly behind CLIP on only two ARO subtasks (Flickr and COCO Order).
With the exception of 20 steps for bias evaluation.
8

---

Currdee-Diiffsioon performance: We find that HardNeg HardNeg-Diffusion//H trained on MS-COCO transfers well to all tasks, outperforming DiffusionITM (Tab. 1). In Tab. we disentangle the effect of using higher quality data and various additional hard negatives. We highlight three insights: 1) Despite fine-tuning on only a single dataset, we observe gains across almost all tasks without any negatives (NoNeg) explicable by the general high-quality MS-COCO dataset compared to more noisy pre-training data. 2) Using hard negatives for only one modality HHardNegr:/RandNNee HardNegtma) only improves the respective retrieval performance while showing occasional drops in the other.? ? 3) We therefore combine all three types of hard negatives, allowing us to work with one model, HardNeg-Diffusionf// Diffusion/TM, rather than multiple models specific to each modality retrieval.
vs.
Table 3: Effect sizes for the bias evaluation. Positive effect sizes indicate bias towards target group X. negative effeet sizes indicate bias towards Y. Effect sizes closer to 0 are less biased and statistically significant effect sizes at < 0.01 are denoted by *. We see that all models exhibit biases, with SD 2.1 being the least biased. For brevity, Buddhist scores are omitted here but contribute to the
average (details in Tab. 5).
Target X
Target Y
SD 2.1
SD 1.5
CLIP RN50x64
CLIP ViT-B/32
Christian
Muslim
0.94
1.06*
1.55*
1.71*
Christian
Jewish
1.10*
1
1.54*
1.69*
Religion
Jewish
Muslim
-0.15
0.03
0.23*
0.48*
Hindu
Muslim
0.86
1.21*
1.48*
1.65*
Nationality
American
Arab
0.63*
0.90*
0.72
1.28*
Sexuality
Heterosexual
LGBT
0.84*
1.04
1.38*
1.68*
Average Absolute Effect Size
0.65
0.79
1.09
1.26
Stable Diffusion 1.5 vs. 2.1 Performance: 2.1 improves on the majority of GDBench tasks except for ARO. With GDBench's 's diverse tasks, we can study if later versions of Stable Diffusion improve in all skill-dimensions or just some of them (see Fig. 2). Interestingly SD 2.1 does not show significant gains over SD 1.5 on all of them. Most notable is ARO (compositionality): We see only minor improvements (VG tasks) or slight drops (Order tasks). At the same time, we do see a jump on Winoground Text from 29% to 32.3% and on other less adversarial tasks such Flickr30K or SVO.
Bias: Both CLIP and Stable Diffusion exhibit bias towards the dominant groups namely Christians, Americans, and Heterosexuals (Tab. 3) with Stable Diffusion 2.1 displaying the least bias. Almost all scores are statistically significant for p < 0.01, with the exception of the Jewish-Muslim for both SD versions (Tab. 3) and some of the Buddhist scores (Tab. 5). Version 2.1 has overall lower effect sizes (average absolute effect size of 0.65 in 2.1 vs. 0.79 in 1.5), suggesting that it is less biased than version 1.5 for this metric (Tab. 3). This goes against the trend of increased bias in stronger models [Nadeem et al., 2021]. On the other hand, Luccioni et al. [2023] found that Stable Diffusion 1.4 is less biased than 2.0. Further investigation is needed to draw strong conclusion to whether the 2.x versions are more or less biased than the 1.x versions due to this discrepancy. It is important to note that there was a major weakening of the safety filter between version 2.0 and 2.1, which may have affected the diversity in the training set and as such, model bias.
Analysis: We find higher menae-text-aligmment of images generated by HardNeg- MaindNeg--Difusio//MMM model based on human judgement. Although our method improves discriminative performance, does it also result in more compositional image generation? Crucially our finetuning the generative capabilities despite directly modifying the noise prediction. We therefore compare songeteext-aiggnmeet of Diffusion/TM against HtardNee-Diffusioo/// DiffissinnTMM on DrawBench [Saharia et al., 2022] and find promising results: From 105 complex prompts, we found that HardNeg is closer to the text almost twice often as the zero-shot model. Similarly, HardNeg finetuning also shows slightly better image-text-aligne text-aligmment than NoNeg finetuning. For more Draw Bench details see (Appendix B) and other analyses (Appendix F). One might ask: how complementary are the skills learned in generative vs. a discriminative vision-and-languaal correctly predicted examples. Our hypothesis: Even if Diffusion/TM might have lower performance on task, its correct predictions may still cover new examples that discriminative models fail to capture. On three datasets, we compare DiffusionITM and two discriminative models (CLIP and BLIP) that were trained differently enough to expect varying predictions. However we find no evidence for
MS-COCO preserved
model? We quantify this via the overlap of
Hard negatives for text are obtained by shuffling parts-of- and images through CLIP image similarity.

---

our hypothesis and perhaps even signs of an opposite trend (Fig. 8). We speculate that this points towards the big role the text encoder behind an text-to-image model plays for vision-and aasoo--nd--anguunn -an--agg understanding, After all, Stable Diffusion relies on frozen CLIP text encoder.
6
Post-submission: The intriguing case of Stable Diffusion XL
After the paper submission, Stable Diffusion XL (SDXL) [Podell et al., 2023] was released so it was a reasonable next step to include it in comparison of different SD versions (see Fig. 2). We expected SDXL to outperform previous versions, based on our upwards trend from 1.5 to 2.1 as well as findings in Podell et al. [2023]. Surprsingly, SDXL reaches significantly lower scores, below 2.1 scores on everything except three ARO subtasks, and even below 1.5 on most tasks (App. F Fig. 10 show exact numbers). Moreover, the top predictions of SDXL have the same exact scores (i.e. two images being ranked first) far more often than in SD 2.1. We confirmed that our results are not coming from a bug in our implementation and hope that future work can shed more light on quantifying the higher-level abilities of SDXL. In the meantime we offer a preliminary interpretation here: It is possible that the capabilities of image generation and image editing, specifically providing new prompt on partially noised image, are not always correlated. In light of this, we also offer two broader interpretations new evaluation methodology: Either our proposed evaluation and-language skills in text-to-image models and
for the validity of DiffusionITM
is not generally applicable to all forms of instead measures more nuanced aspects such as image editing or text sensitivity. Alternatively, this anomaly is evidence that our evaluation is precisely working as intended and exposes flaws that might otherwise taken longer to detect via quantitative evaluation or other metrics such as FID or TIFA.
Conclusion and Future Directions
In this paper, we introduce DiffusionITM and GDBench. DiffusionITM allows us to transform any diffusion-basee generative model to perform image-text matching tasks. This in turn leads to an exciting new paradigm of evaluating text-to- text-to-image models vision-and lan--auugge reasoning using diagnostic benchmarks. We also improve vision-and-land and-language understanding of diffusion models with novel hard negative finetuning strategy (HardNeg). GDBench allows one study many different types of vision and language reasoning, ranging from: compositional (ARO, Winoground) and visual fine-grained reasoning (ImageCoDee), to elements of spatial/attribute binding (CLEVR). Through our experiments, we find that our proposed Diffusion/TM approach shows bsioo-anddlanguuge reasoning capability through increased performance on the several evaluations, as well as provides for head-on automatic comparison among generative models. We conclude that Stable Diffusion performs competitively to CLIP, and performs better on compositional tasks. We also conclude that SD 2.1 is less biased than SD 1.5. We hope that this line of work demonstrates that high-quality diffusion methods for generating images performs well on vision-an -ad---nn---angag ann-language reasoning. We see that the simple task of carefully selecting the appropriate negatives. We encourage future research to explore this paradigm on other models families like Masked Generative Transformers [Chang et al., 2023], and with stronger backbone text encoders. While we found that improved discriminative performance translates into better image-text-aligmment in generated images, this should be explored in more detail. We discuss detailed limitations (especially bias-related) in Appendix A
allows us to test many types of reasoning capabilities by
Acknowledgements
We are grateful to the open-source community behind the Huggingface Diffusers library and the anonymous reviewers for their useful suggestions. This project was funded by the Mila-Samsune and Mila-Google grant program. SR acknowledges the support of the NSERC Discovery Grant program and the Facebook CIFAR AI Chair program. CP acknowledges the support of the Canada CIFAR AI program.
10

---

References
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. Vqa: Visual question answering. International Journal of Computer Vision, 123:4-31, 31, 2015.
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1955-1960, 2016.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL htt.:pppwreeeieeeeettrrtrrrrreae
09Fm.
Ryan Burgert, Kanchana Ranasinghe, Xiang Li, and Michael Ryoo. Peckaboo: Text to image diffusion models are zero-shot segmentors. arXiv preprint arXiv:2211.1322 2022.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 35666334):183--86, April 2017. ISSN 0036-8075. doi: 10.1 httpssiience.aa4222....
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 213-229. Springer, 2020.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint (Tt::22301.00004 2023.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. arXiv preprint arXiv:2202.04053, 2022.
Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero-shot classifiers. In ICLR 2023 Workshop Mathematical and Empirical Understanding of Foundation Models. 2023. URL https: / http////eeeeeeeeeeeeeeeeeeeeeeee net/ orrm? weprummid=ccoo ceriidd=o9llpmmm
Lisa Anne Hendricks and Aida Nematzadeh. Probing image-language transformers for verb under- standing. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3635-3644 Online, August 2021. Association for Computational Linguistics. doi: 10.18633/v1/ 2021 1021.indingg--ll.311 URL https: http///aaaaaatoooo.oooooooooolla org 2021 findings-acl. .318.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Joseph Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. ArXiv, abs/2104.08711 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner. Bernhard Nessler, and Sepp Hochre- iter. Gans trained by two time-scale update rule converge to local nash equilibrium. In I. Guyon, U. Von Luxburg,
Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL https: http:///pooee/nee/////eeee//e nwwrins cttppps.cccpppppp wwtpp/ppppp//21///1..... Paper pdf.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic object accuracy for generative text-to-
image synthesis. IEEE transactions on pattern analysis and machine intelligence, 2020.
Seunghoon Hong, Dingdong Yang. Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7986-7994, 2018.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URI. http::ttp////m hatttss://operrevieeee net/ forum?id= VZe.Ff.F.

---

Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303. 1897, 2023.
Sepehr Janghorbani and Gerard De Melo. Multi-modal bias: Introducing framework for stereotypical bias assessment beyond gender and race in uision-laguaae models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. pages 1717-1727, Dubrovnik, Croatia, May 2023, Association for Computational Linguistics. URL https://aclanthhloggy org /22g/2023..2m www.eacclaaae... 126.
Jin-Hwa Kim, Yunji Kim, Jiyoung Lee, Kang Min Yoo, and Sang-Woo Lee. Mutual information divergence: A unified metric for multimodal generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: //te:://openn http///ppeeeeeeeeeeeeeeeeeeeeeee
Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. Image Retrieval from Contextual Descriptions. In Proceedings of the 60th Annual Meeting of the Association for Compulutional Linguistics (Volume I: Long Pupers), pages 3426-3444 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 12118/611122222.al---1 241. URL htt.t:ttt::///ttt http:/://aaaaahholooooe/e org/2022/22222 acl- 241.
Martha Lewis, Qinan Yu, Jack Merullo, and Ellie Pavlick. Does clip bind concepts? probing compositionallty in large image models. arXiv preprint arXiv:2212.105377 2022.
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. arXiv preprint arXiv:2303.16200 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-languaee understanding and generation. In International Conference on Machine Learning, pages 12888 12900. PMLR, 2022a.
Zhiheng Li, Martin Renqiang Min, Kai Li, and Chenliang Xu. Stylet2i: Toward compositional and high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18197-18207, 2022b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias: Analyzing societal representations in diffusion models, 2023.
Nicholas Meade, Elinor Poole- Poole-Dayan, and Siva Reddy. An empirical survey of the effectiveness of debiasing techniques for pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 1878-1898, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. (0.18653/v1/2022 acl-long.132 URL. htw.h:ttt:://ttttm http///aaaaaaoloooeee/ee .
acl-long. 132.
Antoine Miech, Jean- Sann-Batistteeeee Alayrac, Ivan Laptev. Josef Sivic, and Andrew Zisserman. Thinking Fast and Slow: Efficient Text-to- Visual Retrieval with Transformers, March 2021. URL http:
// //aarxivv org/ abs/aaas//./ 16553. MrXiv:2103.1655: [cs].
Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 5356-5371 Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18663//1/22211.aaa aac.:/ooo2...ll--o..g4.--. URL https: / http//aaaaaattoo.......o org/2021 2021. acl acl-long. 416.
Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. VALSE: A task-independent benchmark for vision and language models centered on linguistic phenomena. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages $253-8280 Dublin, Ireland, May 2022. Association for
12

---

Computational Linguistics. doi: 10.18653/112022.adlloo... URL httph:t//cllaanthotte org/2022. acl acl-long.560 567.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. IEEE Conference Computer Vision and Pattern Recognition, 2012.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01992, 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL org/
http://arxiv
/warriiii..g/aieeo wwgg/b//////eoo /a/aabb///111 2203 00020. arXiv:2 S.ii:2103.000 00.000000 [es].
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 0068410095,, June 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li. Jay Whang, Emily L. Denton, Scyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205. 1487, 2022.
Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. It is all about where you start: Text-to-image generation with seed selection. arXiv preprint arXiv:2304.1 14530, 2023.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114. 2021.
Tejas Srinivasan and Yonatan Bisk. Worst of both worlds: Biases compound in pre-trained vision-and-language models. In Proceedings of the 4th Workshop on Gender Bias in Nat- ural Language Processing (GeBNLP), pages 77-85, Seattle, Washington, July 2022. Asso- ciation for Computational Linguistics. doi: 10. 120.655//1002222gbb........ URL https:
l/aclanthology org/2022. www....eel......... gebnlp-1.10.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A Corpus for Reasoning about Natural Language Grounded in Photographs. In Proceedings of the 57th Florence,
Annual Meeting of the Association for Computational Linguistics, pages Italy, July 2019. Association for Computational Linguistics. doi: 10.19 1644. URL https: http://wwwwwccc aclweb. org/ hegg/anthology/PPP P194144499111114
Tristan Thrush, Ryan Jiang, Max Bartolo, A Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing Vision and Language Models for Visio-Linguistic Composi- tionality, April 2022. URL http: arxiv. org/ 22004 03162. arXiv:2204.03162 [cs].
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings the
the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Nenworks for NLP, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v 1/t///////1S// 5446. URL https: htttp:://cllli httpa////acaaahhho/aeceeo org/ /W18/W1860
org/W18- 5446.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014. doi: 10.1162/tacl a_ a_00166. URL https: /atpee://alaaateee /alaaaanannnooyyyyooo orgo/rg//// //////1411100
Place: Cambridge, MA Publisher: MIT Press.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara Berg. Modeling context in referring expressions. ArXiv, hbs/1608.002722 2016.
13

---

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? The Eleventh International Conference on Learning Representations, 2023. URL. httpps///openreview net/ forum??d==RLLUvvxhuua.
Kankan Zhou, Eason Lai, and Jing Jiang. VLStereoSet: A study stereotypical bias in pre-trained vision- language models. In Proceedings of the 2nd Conference of the Asia-Pacifer Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 527-538, Online only, November 2022. Association for Computational Linguistics. URL https: / /aclanthology. org 2022. aacl-main.49 40.
14

---

A Limitations
Our method limited to image-text matching and cannot tackle other task formulations such as VQA [Agrawal et al. 2015 Suhr et al, 2019 Additionally, taking more noise samples for given image-text pair leads to better performance up certain point (see Fig. This results in slower inference time and can hopefully be mitigated with future innovations.
Measuring bias is erucial aspect of model evaluation and should not be considered secondary the "performance" of the model. Unfortunately. the dataset from Janghorbani and De Melo 2026 does not include gender bias but other datasets such as Zhou et al. 2022 could also be incorporated. In any case, evaluating bias related to personal identities such as nationality, race, and sexual orientation presents certain limitations. These labels are nuanced and not always accompanied by visible identifying characteristics. Consequently, image datasets depicting these groups have limited capacity fully represent these demographics and intersectional identities. While we recognize that assigning a bias score based on these limited resources might not be entirely accurate, it is a vital first step in the right direction is also important to note that the bias evaluation has only positive predictive power, meaning that large effect size is an indication of bias, but low one does not necessarily verify that the model fair. Moreover, the bias effect size (Eq S may sometimes
unreliable (Meade et al., 2022]. This should serve as a preliminary analysis of biases present in the model, which require further investigation using more nuanced and comprehensive methods, including broader qualitative analysis and consultation with social scientists.
B DrawBench Evaluation
As described in Sec. 5 we manually compare our best HardNeg Stable Diffusion with vanilla Stable Diffusion on the Draw Bench benchmark [Saharia al., 2022]. It contains 200 curated challenging prompts along 11 categories. We only select these categories that are relevant to the phenomena studied in this paper (spatial, compositional, attribute binding, ...) and drop categories such as Text (i.e. A storefront with 'Hello World' writlen on it.). This leaves us with 6 categories and 104 prompts:
Category
Prompts
Colours
A brown bird and a blue bear.
Conflicting
A horse riding an astronaut.
Counting
One cat and three dogs sitting the grass.
An illustration of a small green elephant standing behind a large red mouse.
DALL-E
Gary Marcus et al.
An elephant is behind a tree. You can see the trunk on one side and the back legs the other.
Positional
A train on top of a surfboard.
Table DrawBench categories with examples
Next we generate images for each prompt with both models and display them randomly to an author left or right image. We judge whether the left or right image is better aligned with the text and do not focus on image quality. Because we only focus on high-level alignment with text, we observe many ties as both models exhibit the same level of alignment. We repeat this process with another seed, leaving us with 208 blind comparisons. Out of those, our HardNeg model is judged better almost twice as often as vanilla Stable Diffusion (60 vs. 33). The rest (115) are ties.
We show an example where HardNeg is more text- exxt-aligned as well as a tic below:
*For a comprehensive discussion on the limitations of fixed labels inferable from a person's 's appearance, creating datasets based on such visual factors for bias analysis, as well as the discussion on the consequences of biased image generation systems, refer to Luccioni al. 2023
15

---

(a) Prompt: A stack of books. A green book is the top, sitting on red book. The red book is in the middle, sitting on blue book. The blue book is on the bottom. Zero-shot (left) gets it completely wrong while our finetuned Stable Diffusion (right) is quite close the text.
(b) Prompt: A wine glass on top of a dog. One of many examples where both models either fail or succeed equal extent to capture the text.
Figure 5: Examples of DrawBench prompts with vanilla Stable Diffusion and HardNeg Stable Diffusion.
Since HardNeg is not strictly better than NoNeg in experiments (see Tab. conduct the same study with these two models: HardNeg wins comparisons and NoNeg only 38. While this is not statistically significant, it indicates that HardNeg finetuning is useful for image-fext-alignment
C
Visualizing image editing with input optimization and standard denoising
Optimizing the input image directly via backpropagation has shown promise in the diffusion literature Poole er al. 2022 Samuel et al., 2023]. also leads to qualitatively insightful images that differ from standard image editing with Stable Diffusion. Concretely, latent 2) with Adam (Ir=0.05, 200 steps) based on the noise prediction loss where noise added at timestep 0.5. In other words: the input image itself is modified such that it leads lower noise prediction error. Note that this was generated with Stable Diffusion 1.5 before we adopted 2. into our experiments.
optimize the input image (i.e. the
(a) Given the caption "Boy in brown shirt with headphones sits woman's 's shoulders in crowd", optimize correct image (left) three similar but incorrect ones, The right of each image shows result after 200 optimization steps.
(b) Given caption "Two men, standing on an ice, looking into something covered with a tarp". we optimize the correct image (left) and three similar but incorrect ones. The right side of each image shows the result after 200 optimization steps.
Figure We visualize the underlying intuition of our presented methods that image-text pairs that do not fit will lead to more edits. For example in a) we can see how the model needs to add boy with headphones except for the correct image.
Next, we compare image optimizaton to the more established image editing approach visualize the models reasoning, i.e. the denoising process does not start from pure noise but from partially noisy image. Below, we show different strength factors between 0.4 and 0.7 the same two examples as above and denoise for the corresponding amount of steps (according to recommended HuggingFace Diffusers parameters value of 0.5 would correspond 25 denoising steps opposed to the usual 50 steps from pure noise). A lower strength factor means less added noise and therefore less editing.
16

---

(a) Given the caption "Boy in brown shirt with headphones on woman's 's shoulders in crowd", we denoise the correct image (left) and similar but incorrect ones with varying strength factors ("how much noise is added before denoising") ") starting with 0.4 at the top row and moving on to 0.5, 0.6 and 0.7 in lower rows.
(b) Given the caption "Two men, standing on an ice, looking into something covered with a blue tarp", we denoise the correct image (left) and three similar but incorrect ones with varying strength factors ("how much noise added before denoising") starting with 0.4 at the top and moving 0.5, 0.6 and 0.7 in lower rows.
Figure 7: Similar to Fig. we can how in headphones added in the incorrect images in or how in the third row of b) blue structure is added. However the edits are overall less reliable and either change too little too much.
17

---

D
Bias Evaluation Details
Table 5: Additional bias evaluation results for Buddhism. Positive effect sizes indicate bias towards target group X, negative effect sizes indicate bias towards Y. Effect sizes closer 0 are less biased and statistically significant effect sizes at < 0.01 are denoted by *.
Target
Target Y
SD 2.1
SD 1.5
CLIP RN50X64
CLIP ViT-B/32
Buddhist
Muslim
0.79*
0.94*
1.62*
1.63*
Buddhist
Christian
-0.19
-0.16
0.24*
-0.78
Religion
Buddhist
Hindu
-0.11
-0.47
0.46*
-0,448
Buddhist
Jewish
0.93*
0.97*
1.68*
1.25*
E CLEVR Detailed Performance
In contrast to many other GDBench tasks, we did not show accuracies on the CLEVR subtasks in the main paper. However depending on the task DiffusionI''' performance varies lot: Ignoring trivial tasks like colour recognition gets the highest score on Pair Binding Colour (84.5% which involves selecting the right caption among two captions such as A gray cylinder and a purple sphere vs A purple cylinder and gray sphere. On the other hand spatial it is close to random chance (i.e. On the right is gray cylinder vs On the left is a gray cylinder), line with findings on the Imagen model Clark and Jaini 2023
CLEVR task
Diffusion Classifier
SaardNeg-Diffusionnll (Ours)
CLIP RN50x64
Pair Binding (Size)
61.1
70.2
35.5
Pair Binding (Colour)
84.5
86.9
53.0
Binding (Shape Colour)
54.7
58.3
50.7
Binding (Colour Shape)
56.8
67.5
52.9
Recognition (Colour)
89.1
93.5
96.3
Recognition (Shape)
85.3
89.4
78.7
Spatial
43.9
47.6
49.6
Average
67.9
73.3
59.5
Table 6: Detailed CLEVR results.
F Further analyses and plots
Diminishing returns of increasing the number of noise-timestep samples? We show in Fig. 9 that accuracy hits plateau on Flickr30K text retrieval around a sample size of 100. We used 250 in main results Tab. I since didn't test this on all datasets.
Can we use the same idea normalizing with unconditionalltex: also for images? For the sake of completeness, also test the "inverse" of proposed method for text retrieval, i.c. "image- unconditional" denoising as normalization to subtract from the normal denoising error. For this we gray images the "average image" but find significant drop in performance.
What happens we apply the hard negative loss without threshold in Eq. ? Not thresholding the negative loss 7 leads to more improvements ARO (i.e. performance jumps to 63. % on COCO Order compared to the best thresholded finetuning performing of 34.9%) but significantly lower performance on the other tasks such drop from 60.9% zero-shot Pets TTb. to 53.2%. However this uncontrolled objective short-lived as it leads to random performance on all tasks shortly after this partially successful checkpoint (which was less than I epoch into training).
18

---

Diffusion ITM
CLIP
Diffusion ITM
BLIP
CLIP
BLIP
(a) Flickr30K Text Retrieval
Diffusion ITM
CLIP
Diffusion
BLIP
CLIP
BLIP
(b) ARO Flickr30K Order
239
QH
203
239
264
579
X24
Diffusion ITM
CLIP
Diffusion ITM
CLIP
BLIP
BLIP
(c) ARO VG Autributinn
Figure 8: As described in Sec. 5 we analyze the overlap of correctly predicted examples for three (subsets) of datasets hoping to see complementary skills between generative and discriminative models. However we do not find any evidence that Diffusion/TM has less overlap with either discriminative model (CLIP or BLIP) compared the two discriminative models among each other.
19

---

0.9
0.8
R@1 and R@5
0.7
0.6
0.5
R@1 1
R@5
100
200
300
400
500
Sample size
Figure 9: Performance on Flickr30 Text Retrieval with varying sample size of noise-timestep pairs (c, t) per image-text score. We see little benefit of using more than 100-200 samples on this dataset.
Model Dillision ITM wath Stabie Diflinton 1.5 Suble Dittusion XL.
70
50
30
10
Pücck Text
Viicka Incago
VT: Ats.
ARD VG Rel.
COCO Grrder
Flicae Order
CLEVE
SVD Verb
SVD Soby
Dij
ImageCate De suale
ImageCaDe CDDe video
Wisegmoodd group
Wiongrogd tres
Winngrand boage
Figure 10: Updated comparison of different Stable Diffusion version GDBench tasks, here also included the recent SDXI. (Podell et al. 2023 unlike Fig. 2 numbers in Sec.
We discuss the surprsingly low SDXL
G More technical details
Task formulation details: The numbers we report for Flickr30K image and text retrieval are not directly comparable with previous tasks since retrieving from thousands of images or texts is not feasible with the current Diffusion/TM method. Instead we frame it as retrieving from the top- 10 candidates selected by a CLIPRN50x64 model (and the correct one not among them, we put in). For all other tasks the setup is the same.
20

---

Hard negative finetuning: We adopt the exact list of negatives used in Yuksekgonul et al. [002 which includes several text hard negatives (subsection 3.2 and several image hard negatives per image-text pair. The images are the nearest neighbor images based on CLIP embeddings.
21


Reviews:
[{"review_id": "cEI3IFpXRq", "reviewer": "Reviewer_JbKR", "summary": "This paper proposes novel techniques for improving the performance of text-to-image diffusion models on zero-shot image-text matching tasks. They first propose subtracting the unconditional denoising error $\\|\\epsilon - \\epsilon_\\theta(x_t, t)\\|_2^2$, which reduces the problem in image retrieval where one image is a priori far more likely under all captions. The also propose using a hard negative finetuning loss to increase the denoising error on hard negatives, which improves its discriminative (and generative) capabilities. Finally, they assemble  GDBench, a benchmark of 7 existing datasets that they use to quantify different aspects of text-image alignment performance. They find that their proposed model significantly improves on prior work that uses text-to-image diffusion models (Diffusion Classifier), and is competitive with CLIP ResNet50x64. They also benchmark model bias and find that Stable Diffusion 2.1 is less biased than Stable Diffusion 1.5 and CLIP RN50x64.", "questions": "**Main questions/issues**: \n- How well does OpenCLIP ViT-H/14 do on these ITM tasks? This is the most relevant discriminative baseline since SD 2 uses its text encoder. \n- The Flickr30K benchmark is biased against the CLIP model used to produce it. The CLIP model is used to produce the top 10 candidates, and *the correct candidate is added if it's not among the top 10*. If the model being evaluated is CLIP, it has to choose between 9-10 hard negatives, whereas other models will receive a boost from a weak ensembling effect with CLIP. I'd suggest another way of creating the 10 candidates without correlation to any model output. \n- Equation 6: is there a major typo here? This appears to still minimize $\\mathcal L_{neg}$.\n- L 246: Hard negative finetuning uses $\\lambda=-1$? Why is it negative when it's only used for $clip(\\mathcal L_{neg}, |\\lambda \\mathcal L_{pos}|)$? Does this mean that $\\mathcal L_{neg}$ is clipped to the range $[-\\mathcal L_{pos}, \\mathcal L_{pos}]$? Something is wrong here.\n\n**Minor questions/comments**: \n- Table 1b: Where does the 37.5 score come from with Diffusion Classifier/DiffusionITM on Winoground text? The Diffusion Classifier paper reports 34.0. \n- Page 6, footnote 2: This is not too important, since the prompt is fixed across methods, but Pets prompt template used in the CLIP paper is \"a photo of a {}, a type of pet.\" This probably increases scores across the board for all methods. \n- L240-241: \"we drop the complicated procedure that iteratively prunes classes after a number of noise-timestep samples\" -- Diffusion Classifier doesn't use its adaptive strategy for its ITM experiments either, so its evaluation strategy matches DIffusionITM here.  \n- Appendix G (runtime) mentions that \"batch size has diminishing returns in terms of runtime after batchsize=4.\" However, based on the Diffusion Classifier github, using FlashAttention + FP16 + `torch.inference_mode()` + batchsize=32 achieves about 80 evals/s on an A6000. I'd suggest double checking your implementation. Computation speed should be an issue overall with this method, but not to this level.", "limitations": "Yes, authors address the main limitations of their paper and method.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper proposes 2 novel modifications that significantly improve performance over Diffusion Classifier (a prior work on using diffusion models for text-image matching tasks). \n  - The hard negative finetuning method in particular is intuitive yet highly useful. It is also evaluated extremely thoroughly in its effect on both discriminative and generative behavior. \n- Authors also gather a set of 8 image-text-matching tasks into a proposed GDBench benchmark for easily measuring and comparing the performance of different ITM methods. \n- Authors do a thorough analysis of the bias of Diffusion ITM, across versions of Stable Diffusion, as well as in comparison to a few CLIP models. \n- Extensive ablations and experiments on variants of HardNeg, comparison between CLIP, BLIP, and Diffusion ITM, and the relationship between the number of timesteps and the performance of the method.", "weaknesses": "- Computational: The proposed method runs in $O(n)$, where $n$ is the number of candidate images or text captions to choose from. This becomes highly impractical for real retrieval applications, compared to approaches like CLIP where fast approximate nearest neighbor solutions work extremely efficiently. \n- Baselines: while the method does outperform previous work on using diffusion models (e.g., Diffusion Classifier), it doesn't compare against the strongest discriminative baselines, namely OpenCLIP Vit-H/14. This is fair since Stable Diffusion 2 actually uses this model as its text encoder. The paper currently only compares against CLIP ResNet50x64, which is quite weak in comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe are thankful for your detailed and extremely thorough review that shows you have engaged with the work and are very confident with the subject area!\nThank you as well for highlighting strengths of the paper such as:\n- \"The hard negative finetuning method in particular is intuitive yet highly useful. It is also evaluated extremely thoroughly in its effect on both discriminative and generative behavior.\"\n- \"do a thorough analysis of the bias of Diffusion ITM, across versions of Stable Diffusion, as well as in comparison to a few CLIP models.\"\n- \"Extensive ablations and experiments on variants of HardNeg, comparison between CLIP, BLIP, and Diffusion ITM, and the relationship between the number of timesteps and the performance\"\nSimilarly other reviewers also noted:\n- “Tackling a challenging problem, i.e., efficient quantitative evaluation of image generation models, with a simple yet effective method [...]”\n- \"[...] This innovation has practical value to apply. The finding of the relative difference between the with and without text conditions (Fig. 3) is pretty interesting.\"\n- \"new benchmark for image generation models over the discriminative tasks provides a useful tool for the research community and enables comparative analysis.\"\n- \"well written and organized with clear logic to read\" (another reviewer: “well written and easy to read”)\n\nWe appreciate your very detailed comments with suggestions that even went into implementation details! We will address them below:\n\n> \"How well does OpenCLIP ViT-H/14 do on these ITM tasks?\"\n\nWe fixed this valid concern by **running the ViT-L/14 baseline on all tasks** and will include it in Table 1 of the paper (and also the Rebuttal PDF here). ViT-H/14 seemed to be temporarily unavailable the last few days for download so we opted for the closest options based on the [OpenCLIP repo](https://github.com/mlfoundations/open_clip). The short answer is: It does better than our original baseline, as also shown in DiffusionClassifier, and we will discuss this in the camera ready paper.\n\n> \"The Flickr30K benchmark is biased against the CLIP model used to produce it\"\n\nThank you for raising this concern! We acknowledge this is not properly explained in the current version of the paper and is therefore misleading. A **better research question in this situation might be**: \"If CLIP was a first-step fast retriever to narrow down the search, could SD as a slow retriever improve upon re-using CLIP for the second step as well?\" (see [Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers](https://arxiv.org/abs/2103.16553) for in-depth discussion) This was an implicit motivation of ours but not clearly enough stated in the paper. We will add a paragraph to the paper with a **disclaimer as well as an explanation how these numbers are still insightful for the stated reason**. While other experiments took priority over the last week, we plan to study another way to retrieve hard negatives via BERT-embedding NN search for camera ready.\n\nThank you for the smaller fixes and comments!\nYou are right that $\\lambda$ should not be $-1$. We will fix this!\n\n>  \"Where does the 37.5 score come from with Diffusion Classifier/DiffusionITM on Winoground text?\"\n\nWe evaluated with 250 samples of uniform timesteps and beyond that we are not sure how to explain the difference. There is a small standard deviation and after a quick search in the DiffusionClassifier paper we did not find how many samples DiffusionClassifier used for Winoground.\n\nRegarding your suggestions regarding speed, we already used fp16 and now we also tried FlashAttention with small speed gains. Thank you! For our final repository or follow-up work we will optimize speed further. We do not have an A6000 unfortunately. We ran the DiffusionClassifier repo on our side the last days (to check something with Stable Diffusion XL) and it seemed comparably slow to ours but hard to tell due to the adaptive strategy!\n\nWe hope we have sufficiently answered all of your comments point by point, and are happy to engage further on more questions! Considering this, we hope you increase your rating of our paper, and champion it for publication at this conference.\n"}, {"review_id": "XgVBRtf1yH", "reviewer": "Reviewer_jr2c", "summary": "This paper studies the discriminative capabilities of diffusion models measured by image-text matching. A new matching score computation enables text-based image retrieval beyond simply text retrieval in existing works. A new benchmark, augmented from existing image-text benchmarks, is proposed for researchers to evaluate current diffusion models.\n", "questions": "Table 1 typo: Difussion.\nAnd see the weaknesses section for questions.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. DiffusionITM enables image retrieval by making simple and minimal changes to existing text retrieval methods. The method generates good results on simpler datasets and performs better than random on challenging ones.\n2. The proposed benchmark covers diverse aspects but is also optimized for lightweight testing. Differences in a few methods have been demonstrated with the benchmark.\n", "weaknesses": "1. Evaluation on Flickr with CLIP retrieved negatives. This process might introduce bias towards or against CLIP based image diffusion models. For example, in Table 1a, CLIPRN50x64 obtains an accuracy of 71.9. According to line 545, does it mean that 28.1% of the cases CLIP does not select the correct positive? In order to fix this, maybe an evaluation against the original model-free Flickr retrieval metric is needed, e.g. measuring if CLIPRN50x64 or CLIP based method has an advantage/disadvantage compared with other methods, measured on the new simplified set vs. the original set. Similarly, the CLIP-based HardNeg training might be exploiting the bias, besides learning useful discrimination skills.\n2. It is preferred to compare different kinds of diffusion model variants in the proposed benchmark and show some insights there. For example, one might see an advantage of T5 based text encoder in grammar/text related metrics, or one could study pixel based diffusion vs. latent diffusion etc. This comparison is challenging in obtaining the models but can provide more value to the community.\n3. The current method makes an assumption that DiffusionITM, based on eq. 2 and 5, is a great way of ITM and will be the case for a while. However, given the field advancing so rapidly, researchers might discover a better zero-shot matching method that could infer quickly and outperform CLIP consistently for example. Then, at that time, why would researchers not adopt the original retrieval tasks, e.g. on Flickr with all images, but adopt GDBench with 10 images?\n4. It is claimed that the proposed tuning strategy keeps the generation capability. Examples are shown in Figure 5 and positive human evaluation is shown, but is it possible to evaluate existing standard metrics and see the difference? FID might not apply as the tuning happens on COCO. One might even see better generation quality with discriminative tuning applied.\n5. How is GDBench different from an ensemble of image-text matching evaluation metrics, except that it is made light-weight, e.g. measuring 10 samples. It seems DiffusionITM is a general enough method that can be evaluated on theoretically any image-text matching tasks. From another perspective, is it possible to show correlation of the GDBench metrics with real compositional generation quality, e.g. measured by human evaluation.\n6. Given the limited samples in some of the evaluation datasets, what are the standard deviation for different runs? From Figure 9, it looks noisy.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments and effort, showing that you engaged with our work! We are happy to note some of your positive feedback such as our method working with “minimal changes” and how “light-weight GDBench covers diverse phenomena”.\nSimilarly other reviewers noted:\n- “Tackling a challenging problem, i.e., efficient quantitative evaluation of image generation, with a simple yet effective method”\n- “ The finding of the relative difference between with and without text conditions is pretty interesting.\"\n- \" provides a useful tool for the research community and enables comparative analysis.\"\n- \"well written and organized with clear logic to read\"\n- “hard negative finetuning method in particular is intuitive yet highly useful”\n- “Extensive ablations and experiments on variants of HardNeg, comparison between CLIP, BLIP, and Diffusion ITM, …”\n\nWe believe the primary criticism can be attributed to our paper not clearly emphasizing a few key contributions. We hope to clarify these contributions and address the reviewer's concerns below.\n> “It is preferred to compare different kinds of diffusion model variants in the proposed benchmark and show some insights”\n\nBased on your and R4's feedback, we are indeed **including more models and have experimented with the recently introduced Stable Diffusion XL** over the rebuttal period.\nSD-XL comes with new hyperparameters & preprocessing and we are therefore still confirming our results and interesting findings. This is why we decided to not present rushed preliminary numbers here in the PDF but will do so in camera ready after further investigation.\nWe are not aware of any **published** open-source model using T5 text encoders but that would be an interesting addition. The only model we are aware of is the recently introduced DeepFloyd IF (without paper).\nAt the time of writing the paper both models had not been released and most competitive open-source models are variations of SD.\n\n> [...] Researchers might discover a better zero-shot matching method that could infer quickly and outperform CLIP consistently for example. [...] Why would researchers not adopt [...] Flickr with all images but adopt GDBench with 10 images?\n\nWhile we do hope GDBench will be used by researchers, we can't predict the future for several years (realistically benchmarks fade but the GDBench might inspire the direction of future ones). For now we predict diffusion-based methods will stick for a while and will not become as fast as CLIP whose speed comes from encoding text & image separately. Any generative-model-turned-discriminative by definition needs to encode both together (=slow). So such a model should be considered as a second-step slow retriever after a fast simpler retriever (CLIP) has narrowed down the selection (details in [Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers](https://arxiv.org/abs/2103.16553)). Recent trends in VL evaluation point towards prioritizing few high-qualitative examples over many easy examples. Therefore **future researchers will get the most value from evaluating few harder examples**, while also saving time with a slower model.\n> Evaluation on Flickr with CLIP retrieved negatives [...] might introduce bias towards or against CLIP based models\n\nWe acknowledge this is not properly explained in the current version of the paper. However, we think a better research question in this situation is: \"If CLIP was a first-step fast retriever to narrow down the search, could SD as a slow retriever improve upon re-using CLIP for the second step as well?\" This was in fact an implicit motivation of us but not clearly enough stated in the paper. We will add a paragraph to the paper with a disclaimer as well as an explanation how these numbers are still insightful for the stated reason. While other experiments took priority over the last week, we plan to study another way to retrieve hard negatives (i.e. BERT-embeddings) for camera ready.\nFor the finetuning we followed prior established methods and believe this has negligible bias.\n>  Is it possible to evaluate existing standard metrics [for generation] and see the difference?\n\nOur main goal of this paper is to **study higher-level semantic skills of these models which are not adequately captured by older standard metrics**. We therefore focus on small-scale human evaluation to validate our claim of on-par or better generation (see DrawBench analysis in Appendix and Global Rebuttal here). FID score would be insightful but, as explained in the global rebuttal, we are focusing on other high-priority experiments in this short time frame.\n> How is GDBench different from an ensemble of image-text matching evaluation metrics?\n\nGDBench is not a fully new dataset but here are some contributions:\n1. We did make an effort to a) generate CLEVR again (not public before), b) make Flickr30K feasible and c) assemble the bias dataset.\n2. Easy setup with our repo\n3. Selecting tasks based on coverage+diversity of skills as well as feasibility criteria and recent trends, so that other researchers will not have to re-think this (like GLUE in NLP).\nBut it is not set in stone and based on suggestions we plan to also include ObjectNet.\nOn top, we are including more models to show how GDBench performance correlates with image-text-alignment ratings.\n\n> What are the standard deviations[...]? Fig. 9 looks noisy.\n\nFig. 9 was generated with less than 200 of examples which still illustrates the point but we will fix the figure caption to mention this.\nWe did not have time during rebuttal to compute std on all data but studied Winoground (as the smallest dataset) with a small sample size of 10 noise-timestep samples per datapoint, i.e. where we expect high std. Result: $31.5$% $\\pm 0.43$.\n\nWe hope we have sufficiently answered all of the reviewer’s comments point by point, and are happy to engage further on more questions! Considering this, we hope the reviewer increases their rating of our paper.\n"}, {"review_id": "8cKZM9Z5Zm", "reviewer": "Reviewer_T2Yx", "summary": "This paper studies how to use a pre-trained text-to-image generative diffusion model to do discriminative tasks like image and text matching. Building upon previous works [Li et al. 2023, Clark and Jaini 2023], this paper introduces two technical contributions: 1) Unconditional normalization largely improves image retrieval; 2) Tuning the diffusion model with hard negatives on better-quality data like MSCOCO. The paper also introduces a benchmark to evaluate different diffusion models by image and text matching performance, and specifically studies the biases of two versions of Stable Diffusion model.", "questions": "1. I am not sure how much MSCOCO HardNeg is clearly better than MSCOCO NoNeg - To me, MSCOCO NoNeg seems to be even better than HardNeg quantitatively, or at least on par with in Table 2. How do the authors land with HardNeg as the default setting?\n\n2. Why is Winground's random accuracy (25%) higher than the algorithms' accuracy in Table 1(a)?\n", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The two technical contributions (unconditional normalization & tuning on MS COCO) are effective and well ablated.\n\n2. The introduced benchmark could be useful for the community to evaluate the prompt-following ability and biases new diffusion models.\n\n3. The paper studied bias in state-of-the-models, which should draw more attention to the community.", "weaknesses": "1. The generative-for-discriminative methods typically take many feedforwards of the diffusion model to evaluate a single sample. Although the paper briefly mentions its slow runtime at the end of the Appendix, I think the computational cost is an important factor of the proposed algorithm and should be rigorously presented and studied in the main text, especially when the paper proposes a benchmark - A clear study on the computational cost would be important to others who plan to evaluate on this benchmark.\n\n2. The GDBench covers several hard image-and-text matching datasets. However, I think results on other standard discriminative datasets would be helpful for the audience to understand the advantage of the proposed algorithm and compare to previous works. For example, how does HardNeg-DiffusionITM work on ImageNet compare to Diffusion Classifier [Li et al. 2023] / DiffusionITM (Ours) in Table 2(b)? I understand ImageNet is a 1000-way classification and would be a long time to run for this kind of methods (therefore studying runtime is important as mentioned above), but Diffusion Classifier [Li et al. 2023]  has already reported their performance on ImageNet and an apple-to-apple comparison would be very helpful. Same applied to the CLEVR dataset. Another question is how tuning on MSCOCO works for image-text matching on MSCOCO?\n\n3. I am not sure what is the use case to introduce MSCOCO tuning into Diffusion Classifier. It indeed improves the performance of image and text matching compare to the plain one, but all these methods are too slow (or not good enough) to be practically applied. On the other hand, tuning the weights of diffusion models prevents it from serving as a tool for evaluating diffusion models. Then in what case do we need MSCOCO tuning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments and effort, showing that you engaged with our work.\nWe are glad you found our “two technical contributions (unconditional normalization & tuning on MS COCO) effective and well ablated\" and that our study contributes to drawing attention to bias.\n\nSimilarly, other reviewers noted that:\n- “Tackling a challenging problem, i.e., efficient quantitative evaluation of image generation, with a simple yet effective method [...]”\n- \"well written and organized with clear logic to read\"\n- “hard negative finetuning method in particular is intuitive yet highly useful”\n- “Extensive ablations and experiments on variants of HardNeg, comparison between CLIP, BLIP, and Diffusion ITM …”\n\n We believe the primary criticism can be attributed to our paper not clearly emphasizing a few key contributions. We hope to clarify these contributions and address the reviewer's concerns below.\n\n> [...] I think the computational cost [...] should be rigorously presented and studied in the main text [...] - important to others who plan to evaluate on this benchmark.\n\nThe computational cost has been mentioned in the Appendix. Given that other reviewers were also looking for it, we will move the discussion of this important subject from the Appendix to the main paper and expand the analysis, as well as cite findings from other papers more prominently:\nBoth concurrent works study time cost/efficiency analysis thoroughly, especially [1], and our paper focused on additional contributions, benefiting from their time cost findings.\nSpecifically, we will a) conduct the same analysis of Fig. 9 for more than just 1 dataset and b) have implemented speed-up suggestions from R1 over the last week (with small improvements of 10-20%).\n\n> [...] I think results on other standard discriminative datasets would be helpful for the audience to understand the advantage of the proposed algorithm and compare to previous works [...] Diffusion Classifier has already reported their performance on ImageNet and an apple-to-apple comparison would be very helpful.\n\nOur **goal and contribution is  primarily to study more complex vision-and-language reasoning** which is why we did not include ImageNet containing simple single-object images. These vision-focused tasks were already studied in detail in related work like [1] and to our knowledge most recent models of vision-and-language also focus less on object recognition.\nFair comparison to previous work is important, which is why we include CLEVR (a complex reasoning task) and put in effort to generate images+text as close to [2] as possible, since [2] did not publish their dataset. Can you clarify what you mean with “\"same applied to the CLEVR dataset\"?\nHowever since a) our focus is VL reasoning and b) our zero-shot text retrieval setting should be identical to DiffusionClassifier (Li et al., 2023) for the ImageNet case we do not see a strong need to add ImageNet. We do see value in ObjectNet as proposed by R4 and will consider it for camera ready!\n\n> Another question is how tuning on MSCOCO works for image-text matching on MSCOCO?\n\nIf we understand the question correctly: We finetuned on MSCOCO train set so we can still test on the validation set. However the transfer setting can indeed not be studied. But our intuition is that Flickr30K covers a very similar task and hence it is not a problem for GDBench.\n\n> I am not sure what is the use case to introduce MSCOCO tuning into Diffusion Classifier.\n\n Regarding “all these methods are too slow (or not good enough) to be practically applied”): a) it is possible diffusion or DiffusionITM become faster soon (i.e. one might get good performance with less samples) , and b) it is **common practice to have a fast retriever to narrow down the best results** (i.e. top-10) and next apply a slower sophisticated retriever for the final selection (see [Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers](https://arxiv.org/abs/2103.16553)). Moreover, our work introduces a **generally new idea for incorporating negative pairs into generative pre-training** - which we believe is independent of applying it as ITM or evaluation of off-the-shelf models.\n\n> [...] how much MSCOCO HardNeg is clearly better than MSCOCO NoNeg\n\nThat is a valid point and we authors discussed this during the submission. HardNeg is better for all image retrieval tasks (except Winoground which is in both cases below random and a tiny dataset). On the other hand, for text retrieval, NoNeg is better except for Pets. We see promise with negatives, especially hard image negatives for image retrieval.\nWe investigated further regarding generative performance (specifically image-text-alignment) and conducted the same study as with DrawBench zeroshot vs. HardNeg in Appendix B, but this time on HardNeg vs. NoNeg. We found that **HardNeg has higher alignment, winning 50 of the comparisons while NoNeg only won 38** (see Rebuttal PDF). Hence, we chose to say that MSCOCO HardNeg is indeed better than MSCOCO NoNeg.\n\n> Why is Winground's random accuracy (25%) higher than the algorithms' accuracy [...]?\n\nThis is very common behaviour: Most models tested in the Winoground paper reach below random! An intuition: Many VL models behave similar to a bag-of-words model, i.e. ignoring word order. So imagine a model that treats $c_0$ and $c_1$ (both containing same words in different order) as the same caption for many examples.\nAt the same it has a default prior for one of the images and prefers it with both captions.\nSuch behavior on many examples would lead to a below random score.\n\nWe hope we have sufficiently answered all of your comments point by point, and are happy to engage further on more questions! Given this, would you consider increasing your rating of our paper?\n\n[1]: Your Diffusion Model is Secretly a Zero-Shot Classifier (Li et al., Arxiv 2023)\n[2]: Text-to-image diffusion models are zero-shot classifiers (Clark & Jaini, ICLR MRL Workshop 2023)"}, {"review_id": "UR59ViYV9b", "reviewer": "Reviewer_cFkA", "summary": "Recently diffusion-based text-to-image generation models have evolved rapidly, but it’s still challenging to evaluate them quantitatively in an efficient way.\n\nThis paper smartly converts evaluations of Stable Diffusion based image generation tasks as simpler image-text matching tasks (e.g. image-text retrieval), and proposes an image-text benchmark (GDBench) including 8 carefully-chosen tasks to measure generative model’s fine-grained properties (e.g. compositionality) and fairness. CLIP and Stable Diffusion models are evaluated on this new benchmark.\n\nOverall, this paper bridges discriminative and generative model evaluations, which could be inspiring to the image generation community and improve the quantitative evaluations of generative models.\n", "questions": "As in “Weaknesses”: how to prove the proposed method can measure the capability of SD models?\n\nTo measure discriminative vision-language models, both classification and retrieval tasks are usually used. Besides “pets”, could we also add some other common 0-shot classification tasks as well? E.g., INET (most common) or ObjectNet (to measure robustness).\n\nminor question/suggestion: this paper’s citation seems to use a different format (e.g. “[name year]”) than others (e.g. “[number]”)? The latter is more convenient to read, especially for prints.\n", "limitations": "This paper has addressed model bias in one of GDBench tasks.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "Tackling a challenging problem, i.e., efficient quantitative evaluation of image generation models, with a simple yet effective method (predicting the noise of the diffusion for image-text alignment).\n\nProposing an image-text benchmark (GDBench), covering diverse dimensions for measuring image generation (e.g. semantics, compositionality, fairness), which is beneficial for the image generation community.\n\nWell written and easy to read.\n", "weaknesses": "The biggest concern on my side is the lack of enough evidence to show such a ITM eval aligns well with SD model quality, which IMO can’t be strongly supported by the evals to compare CLIP and SD models on retrieval numbers. If possible, I would suggest at least to prepare a few SD models with different capability (e.g. trained with different numbers of steps), and show they have different numbers on GDBench; bonus: give some qualitative examples for these models conditioned on the same text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your kind and insightful comments! We are thrilled by your comment that the paper is \"Well written and easy to read\".\n\nWe are glad you recognized that we are **\"tackling a challenging problem, i.e., efficient quantitative evaluation of image generation models\"**, with \"diverse dimensions [...] ( e.g. semantics, compositionality, fairness)\".\n\nSimilarly, other reviewers noted that:\n- \"[...] This innovation has practical value to apply. The finding of the relative difference between the with and without text conditions (Fig. 3) is pretty interesting.\"\n- \"new benchmark for image generation models over the discriminative tasks provides a useful tool for the research community and enables comparative analysis.\"\n- “hard negative finetuning method in particular is intuitive yet highly useful”\n- “Extensive ablations and experiments on variants of HardNeg, comparison between CLIP, BLIP, and Diffusion ITM, and the relationship between the number of timesteps and the performance”\n- “Paper studied bias in state-of-the-models, which should draw more attention to the community”\n\nWe address concerns and suggestions below, highlighting the contributions of our paper and describing additional experiments we have conducted.\n\n> \"[...] prepare a few SD models with different capability (e.g. trained with different numbers of steps), and show they have different numbers on GDBench; bonus: give some qualitative examples for these models conditioned on the same text\"\n\nWe liked your main suggestion and have tested one more model, specifically the recently introduced [Stable Diffusion XL](https://arxiv.org/pdf/2307.01952.pdf). SD-XL comes with new hyperparameters and pre-processing steps and we are therefore in the process of confirming our results and some interesting findings. This is why we decided to not present rushed preliminary numbers here in the PDF.\nWe also plan to include one more model for camera ready, i.e. SD-XL 0.9 or DeepFloyd (with a T5 text encoder), and as you suggested, are currently conducting more analysis similar to the DrawBench study already in the appendix. Note that these models were released after our main submission. Regarding “qualitative examples”, we provided additional analysis of HardNeg vs NoNeg setups in the new PDF (see global response).\n\n> “add some other common 0-shot classification tasks as well? E.g., INET (most common) or ObjectNet (to measure robustness).”\n\nOur goal and **contribution is primarily to study more complex vision-and-language reasoning** which is why we did not include ImageNet containing simple single-object images. These tasks were also studied in detail in related work like (Li 2023)[https://arxiv.org/abs/2303.16203]) and to our knowledge most recent models of vision-and-language also focus less on object recognition.\nThat being said, GDBench doesn’t have to be static and ObjectNet looks like a good fit to test deeper understanding! For camera ready, we might either replace Pets with it or add it.\n\nRegarding citation format, thanks for the suggestion, we will revise the format in the final version.\n\nThank you again for your helpful review! We believe we have addressed all of your concerns point by point. Given this, would you consider increasing your rating of our paper?\n"}, {"review_id": "3vdjC3lar5", "reviewer": "Reviewer_ttjT", "summary": "The paper introduces Diffusion-ITM, a new method that directly adapts diffusion-based models to image-text matching tasks without retraining. Additionally, the authors collected a new benchmark called Generative-Discriminative Evaluation Benchmark (GDBench), which includes seven complex vision-and-language tasks and bias evaluation. The results show that Diffusion-ITM performs competitively on multiple tasks and surpasses CLIP. This paper underscores the significance of jointly considering discriminative and generative models and provides a new benchmark for future work.", "questions": "Please see weakness", "limitations": "Please see weakness", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper presents a new method to adapt diffusion-based models for image-text matching tasks without retraining. This innovation has practical value to apply. The finding of the relative difference between the with and without text conditions (Fig. 3) is pretty interesting.\n* The authors thoroughly evaluate the proposed method on various vision-and-language tasks, offering insights into its performance and potential biases.\n* The introduction of new benchmark for image generation models over the discriminative tasks provides a useful tool for the research community and enables comparative analysis.\n* This paper is well written and organized with clear logic to read.", "weaknesses": "Weaknesses:\n* The Stable Diffusion includes a pre-trained CLIP text encoder and has itself been pre-trained on numerous image-text pairs. Therefore, Stable Diffusion has sufficiently learned the joint distribution of images and texts. The new finding is interesting but not surprising. \n* The performance of the proposed method on more challenging benchmarks, like Winogroud, is relatively poor, suggesting room for potential improvement.\n* The paper lacks theoretical analysis—since diffusion models are generative, applying them to a discriminative task would benefit from more theoretical insight.\n* This paper ignores time cost/efficiency analysis. The Stable Diffusion inference is slow. According to Eq. 5, how to assign the t in your method? How many steps are needed and why?\n* Minor issues, such as missing punctuation after equations, formatting errors, and a redundant space at the start of line 141, need to be addressed.\n\n\nQuestion:\n* Can the Stable Diffusion method be extended to more challenging image-text tasks like Visual Question Answering (VQA)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time and effort to provide their detailed feedback on our submission! We are happy to note all the positive comments from the reviewer including:\n- “... finding of the relative difference between with and without text conditions (Fig. 3) is pretty interesting.\"\n- \"... provides a useful tool for the research community and enables comparative analysis.\"\n- \"well written and organized with clear logic to read\"\nSimilarly, other reviewers noted that:\n- “hard negative finetuning method is intuitive yet highly useful”\n- “Extensive ablations and experiments on variants of HardNeg, comparison between CLIP, BLIP, and Diffusion ITM [...]”\n- “paper studied bias [...] which should draw more attention to the community”\n- “Tackling a challenging problem, i.e., efficient quantitative evaluation of image generation models, with a simple yet effective method [...]”\n\nWe believe the primary criticism can be attributed to our paper not clearly emphasizing a few key contributions. We hope to clarify these contributions and address the reviewer's concerns below.\n\n> \"[...] The new finding is interesting but not surprising.\"\n\nWe respectfully disagree with the reviewer that the result is unsurprising. In fact, **generative models typically have poor discriminative performance**; we point the reviewer to several prior works [4,5,6] that analyze this trade-off between generative and discriminative capabilities.\n\nOur contribution has been to analyze the discriminative capability of a new class of generative models i.e. denoising diffusion models. Little work on this topic exists prior to our submission. We admit that considering their powerful capabilities, it may be anticipated that they might perform well on discriminative tasks. Nevertheless, we fail to see how an “interesting but unsurprising” result is considered a weakness, especially in light of our extensive experimental evaluation.\nGoing into the project we had two plausible hypotheses. 1) The generative objective might lead to deeper understanding of composition of visual scenes beyond the frozen text-encoder which has been shown to mostly work as a BOW model. Or 2): The generative objective might be focused on low-level details at the expense of leveraging the semantics from the text-encoder (i.e. unlearning semantics). **Both hypotheses needed to be empirically tested - hence our paper!**\n\n> \"The performance [...] on more challenging benchmarks like Winoground [...] suggests room for potential improvement.\"\n\nWe would like to emphasize that **progress on Winoground has been generally very slow** ([Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality](https://aclanthology.org/2022.emnlp-main.143)) despite several papers and research groups having tried to tackle it, and similarly with benchmarks like ARO. We believe it is unfair to see this as a weakness of our specific paper and not the nature of well-designed hard tasks.\n\n> \"[...] since diffusion models are generative, applying them to a discriminative task would benefit from more theoretical insight.\"\n\nWe provide theoretical insight in the paragraph on Eq. 4 and 5. These are specific justifications for our method and not about the general concept of making a generative model into a discriminative one. We plan to expand the Related Work subsection on \"Repurposing Text-Conditioned Diffusion Models\", i.e. discussing how several families of generative models are adapted differently to a discriminative setting . We already have an older draft where we cite work such as [1,2,3]. Is this what you had in mind?\n\n> This paper ignores time cost/efficiency analysis. The Stable Diffusion inference is slow. According to Eq. 5, how to assign the t in your method? How many steps are needed and why?\n\nWe provide time cost/efficiency analysis in the Appendix (see Fig. 9). Based on feedback from other reviewers as well, this will be **moved more prominently to the main paper**.\nBoth concurrent works study time cost/efficiency analysis thoroughly, especially [Li et al. 2023], and our paper focused on additional contributions.\nRegarding choosing how many timesteps t to sample, we specify in Section 5 \"Experiments and Results\" under Hyperparameters, that we choose sample_size=250 for the main experiments and sample_size=10 for further ablations (sampled uniformly from [0,1000]).\n \n> Minor issues (punctuation, formatting) [...]\n\nThank you for pointing these out! We will fix these.\n\n> Can the Stable Diffusion method be extended to [...] VQA?\n\nThere is no elegant way but we did consider it (see Limitations section).\nPrior works have tried creating a \"caption\" by concatenating question+answer and then using old-school VQA evaluation (treating it as classification).\nHowever this is **impractical, and we discard it for two reasons**:\nFirst, our main goal was to evaluate SD off-the-shelf, or with minimal changes to its objective. If you treat SD as a backbone you can add all sorts of architectures on top. It is common to use the middle U-Net layer as the image-text representation, see concurrent work (Li et al., 2023).\nSecond, VQA comes with its own problems. While it is “the” standard VL task, we believe that more **recent, often ITM-based, diagnostic benchmarks are more suitable to target phenomena such as compositionality**.\n\nWe hope we have sufficiently answered all of your comments point by point, and are happy to engage further on more questions! Would you consider increasing your ratings given the clarifications?\n\n(Due to character limit constraints we provide the 6 citations from the main text as a comment after the rebuttal deadline)\n"}]

Author rebuttal:
Dear Reviewers and Area Chairs,

First we would like to thank all reviewers for writing detailed and thoughtful responses. It raised interesting discussions among the authors and will make it an overall stronger paper. In particular, we are grateful that Reviewer JbKR gave us a a score of 7 with high confidence and very detailed technical feedback that showed his expertise in this domain.

Reviewers pointed out the following strengths of our paper:
1. **Method contributions**: “hard negative finetuning method in particular is intuitive yet highly useful” (JbKR), “two technical contributions (unconditional normalization & tuning on MS COCO) are effective and well ablated” (T2Yx) and "finding of the relative difference between the with and without text conditions (Fig. 3) is pretty interesting" (ttjT)
2. **Analysis**: “evaluated extremely thoroughly in its effect on both discriminative and generative behavior" (JbKR) and “extensive ablations and experiments on variants of HardNeg, comparison between CLIP, BLIP, and Diffusion ITM, and the relationship between the number of timesteps and the performance” (JbKR)
3. **Evaluation contribution**: “Tackling a challenging problem, i.e., efficient quantitative evaluation of image generation models, with a simple yet effective method” (cFkA), “benchmark covers diverse aspects but is also optimized for lightweight testing” (jr2c), “paper studied bias [...], which should draw more attention to the community” (T2Yx)
4. **Presentation**: “Well written and easy to read” (cFkA) and "well written and organized with clear logic to read" (ttjT)

On top of responding to concerns, we also addressed common feedback from reviewers with further empirical investigations, summarized here:
1. Reviewer JbKR pointed out that an **additional stronger baseline**  (CLIP ViT-H/14) is needed. The results are shown in the Rebuttal PDF (Tab. 1). The baseline is stronger overall and we will address this in the camera ready version.
2. Reviewer T2Yx asked about why we highlighted the **finetuning setup *HardNeg* over the *NoNeg* setup as our strongest model**. While performance on our discriminative benchmark is very close, we did conduct an additional human judgement study where we found that HardNeg has stronger image-text-alignment when it comes to **generative** performance (see details in rebuttal response to T2Yx and sample images in Rebuttal PDF)
3. Both jr2c and cFkA suggested to **evaluate more models** for reasons such as a) insights on model variants (i.e. T5 text encoder) and b) evidence that GDBench correlates with generative performance. Most strong open-source models boild down to variants of Stable Diffusion and we therefore spent most of this week evaluating the recently introduced Stable Diffusion XL. SD-XL comes with new hyperparameters and pre-processing steps and we are therefore still confirming our results and some interesting findings. This is why we decided to not present rushed preliminary numbers here but will do so in camera ready after further investigation. We are not aware of any open-source model (that also comes with a paper) using T5 text encoders but that would be an interesting addition. The only model we are aware of is the recently introduced [DeepFloyd IF](https://github.com/deep-floyd/IF) (without paper). **At the time of writing the paper both models had not been released.**
4. We provide standard deviation for jr2c.

Due to the short time frame we had to prioritize the most insightful experiments and responded to other comments in the rebuttal responses. Here the commom threads were the computational/speed cost of our method (JbKR, T2Yx, ttjT) and including more datasets (cFkA, T2Yx):
1. **Computational cost**: We agree with reviewers that computational cost is an important issue, and had previously discussed it in the Appendix. For the final paper we will move the discussion from the Appendix to the main paper, expand the analysis, and cite findings from other papers more prominently:
Both concurrent works study time cost/efficiency analysis thoroughly, especially Li et al. [2023], and our paper focused on additional contributions, benefiting from their findings. Hence this is already covered in previous literature.
We also emphasized that models such as DiffusionITM are not intended to be a fast retriever such as CLIP but more a “second-step slow retriever” that comes into play once a fast retriever has narrowed down the options to the hardest candidates. This is common practice (see [Miech et al., 2021](https://arxiv.org/abs/2103.16553)).
2. **More datasets**: Two reviewers suggested more vision tasks. Our **goal and contribution is primarily to study more complex vision-and-language reasoning** which is why we did not include ImageNet containing simple single-object images. These vision-focused tasks were already studied in detail in related work like [Li 2023](https://arxiv.org/abs/2303.16203) and to our knowledge most recent models of vision-and-language also focus less on object recognition. Fair comparison (raised by T2Yx) to previous work is important, which is why we include CLEVR (a complex reasoning task) and put in effort to generate images+text as close to [2] as possible ([2] did not publish their dataset). We do see value in ObjectNet as proposed by cFkA and will consider it for camera ready.

We plan on studying the following for a potential camera ready version:
- bias in our modified Flickr30K task (i.e. how to choose hard negatives for Flickr30K)
- include ObjectNet as an additional task

As much as we would like to run all possible experiments, it is unclear whether we will have enough time to fulfill all other suggestions. However we will mention these points in the limitations section!

Overall, we believe we addressed the main criticisms and engaged faithfully and constructively with the feedback. We are looking forward to fruitful exchanges the next weeks to further improve the work!


Meta-review: This paper proposes novel techniques for improving the performance of text-to-image diffusion models on zero-shot image-text matching tasks. The paper received mixed reviews.

On the positive side, the proposed method is novel, intuitive and interesting, achieves reasonably good results. The paper also introduced benchmark GDBench covers diverse skills and provides a useful tool for evaluating image-text alignment. The evaluation is comprehensive, which includes extensive ablations and the model bias study. The paper is well written and easy to follow.

On the negative side, the computational cost of the proposed method is high, making it impractical for real applications compared to efficient methods like CLIP. Performance on very challenging datasets like Winoground is still quite low. More comparisons to other diffusion model variants would provide more insights. Additional experiments validating that the benchmark correlates with generative quality would be useful. More theoretical analysis on repurposing generative models as discriminative models could provide insight.

Overall, this paper provides some interesting observations which may bring useful discussions to the community.
Decision: Accept (poster)