SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Flickr30K [Young et al., 2014] is a well-established open-ended image and text retrieval dataset, captioning diverse scenes involving people. Note that we changed the task setup to reduce computation overhead: A model has to retrieve among the 10 most similar examples based on the CLIP embedding space. While this might not be fair towards our CLIP baselines, we chose this setup primarily to study if SD could be useful a second-stage slow retriever, after a fast retriever like narrows down the candidates [Miech et al., 2021]. Both Winoground [Thrush et al., 2022] and ARO [Yuksekgonul et al., 2023] are diagnostic benchmarks for compositionality. Winoground is carefully curated at the cost of only 400 examples and many SOTA models have not reached significantly beyond random chance. ARO is automatically generated on top of Flickr30K, MS-COCO, and others with only compositional hard text negatives. ImageCoDe [Krojer et al., 2022] is an image retrieval task focusing on highly similar images with complex pragmatic captions crowdsourced from a guessing game. SVO [Hendricks and Nematzadeh, 2021] disentangles performance along different parts-of-speech by pairing images that differ only in subject, object or verb. Lewis et al. [2022] introduced a diagnostic controllable benchmark based on simple synthetic CLEVR $^2$  images of 3D shapes, thereby isolating various phenomena like attribute binding or spatial relations. Finally, we include Pets [Parkhi et al.,
2We generate CLEVR images and captions ourselves and they are therefore not exactly comparable with the same previous task formulation [Lewis et al., 2022, Clark and Jaini, 2023].
Table 1: Benchmarking Diffusion ITM with vanilla SD and hard-negative fine-tuning on MS-COCO on GDBench. Diffusion Classifier performs around random chance on image retrieval. Hard negative transfer finetuning significantly improves on both.
(a) Accuracy of DiffusionITM and baselines on GDBench image retrieval tasks.
Flickr Img	SVO	ImageCoDe	Winoground
Verb	Subj	Obj	Static	Video	Image
CLIP RN50x64	71.9	82.2	85.8	89.3	51.6	23.5	13.8
OpenCLIP ViT-L/14	79.9	85.6	90.7	92.8	69.8	26.0	11.25
Diffusion Classifier	11.1	50.6	55.0	48.7	8.9	9.0	0.25
DiffusionITM (Ours)	61.5	77.3	80.5	86.2	42.5	21.1	10.2
HardNeg-DiffusionITM (Ours)	66.1	77.8	81.3	84.7	44.9	21.0	12.3
(b) Accuracy of DiffusionITM and baselines on GDBench text retrieval tasks.
Flickr Txt	ARO	CLEVR	Winoground
VG Attr.	VG Rel.	COCO Ord.	Flickr Ord.	avg	Pets	Text
CLIP RN50x64	65.8	62.7	50.8	52.0	58.8	59.5	88.4	26.3
OpenCLIP ViT-14/L	71.3	59.2	50.3	30.8	39.8	64.3	89.9	30.25
Diffusion Classifier / DiffusionITM (Ours)	69.8	62.9	50.0	23.5	33.2	67.9	79.7	37.5
HardNeg-DiffusionITM (Ours)	73.5	67.6	52.3	34.4	48.6	73.3	81.3	39.5
2012] as a smaller quick-to-run image classification dataset. In contrast to linguistically complex GDBench tasks, Pets covers the complementary skill of fine-grained recognition (37 animals).<sup>3</sup>
Measuring Bias: Bias evaluation is essential as large-scale models increasingly suffer from harmful biases that can impact downstream tasks. DiffusionITM allows automatic, quantitative bias evaluation with bias datasets intended for discriminative vision-and-language models. We utilize the dataset from Janghorbani and De Melo [2023] and investigate three different types of social biases: religious, nationality, and sexual orientation. Quantitatively, bias is present when there is a substantially stronger association of one target group to pleasant attributes compared to unpleasant attributes over another group, as measured by some notion of distance or similarity. The target groups are represented by sets of images  $X^{I}, Y^{I}$  and the attributes are captured by sets of words  $A^{T}, B^{T}$ . Bias is measured by the following normalized association score,  $d$ , called the effect size:4
$$
d \left(X ^ {I}, Y ^ {I}, A ^ {T}, B ^ {T}\right) = \frac {\operatorname {m e a n} _ {x \in X} \psi (x , A , B)}{\operatorname {s t d e v} _ {i \in X \cup Y} \psi (i , A , B)} - \frac {\operatorname {m e a n} _ {y \in Y} \psi (y , A , B)}{\operatorname {s t d e v} _ {i \in X \cup Y} \psi (i , A , B)} \tag {8}
$$
$$
\text {w h e r e} \quad \psi (i, A, B) = \underset {a \in A} {\text {m e a n}} \sigma (i, a) - \underset {b \in B} {\text {m e a n}} \sigma (i, b)
$$
and  $\sigma (\cdot ,\cdot)$  is our proposed DiffusionITM score, or, in the case of CLIP, cosine similarity.
More concretely, in the case of Religion, the image sets might be  $X =$  Christians,  $Y =$  Muslims and the attribute word sets would be  $A = \{ \text{"joy", "trust", "good", ...}\}$ ,  $B = \{ \text{"corrupt", "vulgar", "bad", ...}\}$ . Here, a positive effect size means that it is biased towards Christians and a negative effect size means it is biased towards Muslims. We provide more details under limitations in Appendix A.
3We chose Pets over ImageNet with 1000 classes since we want GDBench to be light-weight to run. Following Radford et al. [2021], we use the text prompt "an image of  $X$ ".
<sup>4</sup>We use a permutation test to compute the significance of effect size scores Caliskan et al. [2017].
<sup>5</sup>Random accuracy: 10% (Flickr, ImageCoDe), 50% (ARO Rel./Attr., SVO, CLEVR), 25% (Winoground), 20% (ARO Order), 2.7% (Pets).
5 Experiments and Results
Our main two findings are summarized in Tab. 1: First, zero-shot DiffusionITM achieves performance near CLIP on image retrieval (Tab. 1a), overcoming the close-to-random performance of Diffusion Classifier [Li et al., 2023]. Second, our best hard negative transfer-finetuning strategy improves performance across the board, on both image and text retrieval.
Hyperparameters: Based on ablations in Li et al. [2023], we adopt most of their setup but aim for more simplicity: Timesteps  $t$  are sampled uniformly from  $[0,1000]$ , guidance scale is kept at 0, but we drop the complicated procedure that iteratively prunes classes after a number of noise-timestep samples  $(\epsilon ,t)$ . Instead we keep samples constant at 250 for the main zero-shot experiments in Tab. 1 and reduce it to a much more feasible number of 10 samples for other experiments. This is aligned with our goal that GDBench should be easily adopted by researchers to evaluate their image generation method. As shown in Appendix Fig. 9, performance is not at its maximum with few samples but the trends can be studied in the same way when comparing models along different tasks. We adopt the common CLIP RN50x64 baseline and OpenCLIP ViT-L/14 for a fair comparison since SD 2.1's text backbone is from the latter. We fine-tune on the MS-COCO hard negative training set [Yuksekgonul et al., 2023] with  $lr = 1e - 4$ ,  $\lambda = 1.0$  and batchsize 112. We select a checkpoint after 8 epochs based on hard negative validation. Runtime: With 10 noise samples per image-text-pair evaluation on Flickr30K Text Retrieval validation takes 68 minutes on a single NVIDIA RTX A6000 GPU (compared to around 4 minutes with OpenCLIP ViT-L/14). We point to Li et al. [2023] for an in-depth analysis of runtime. We emphasize that we envision future stronger SD-based discriminators as slow retrievers that are applied to a small set of candidates provided by a fast retriever [Miech et al., 2021], as well as the benefit of automatic evaluation.
Table 2: Comparison of finetuning approaches on (top) image and (bottom) text retrieval, with only 10 sampling steps due to runtime feasibility (lower performance than Tab. 1 but same trends).
(a) Image Retrieval	SVO	ImageCoDe	Winoground
Flickr Img	Verb	Subj	Obj	Static	Video	Image
Vanilla SD	46.1	71.2	74.1	79.4	30.1	15.7	9.0
+ MS-COCO NoNeg	48.2	71.1	74.7	76.9	29.7	16.1	10.3
+ MS-COCO RandNegTxt	47.7	71.5	73.8	77.5	28.3	16.0	10.7
+ MS-COCO HardNegTxt	47.0	71.3	74.1	76.8	30.6	16.2	9.6
+ MS-COCO HardNegImg	52.9	73.1	76.1	79.4	34.6	17.2	10.5
+ HardTxt + RandTxt + HardImg	49.4	71.7	75.4	78.4	31.9	16.6	9.8
(b) Text Retrieval	ARO	CLEVR	Winoground
Flickr Txt	VG Attr.	VG Rel.	COCO Order	Flickr Order	avg	Pets	Text
Vanilla SD	55.3	59.2	49.8	24.8	31.6	65.7	60.9	32.3
+ MS-COCO NoNeg	62.2	62.3	53.2	33.6	42.9	67.1	70.0	35.0
+ MS-COCO RandNegTxt	62.2	61.6	53.1	34.0	42.4	67.6	70.2	32.2
+ MS-COCO HardNegTxt	60.9	62.2	52.9	34.9	44.0	68.5	69.4	33.7
+ MS-COCO HardNegImg	61.1	61.1	52.4	30.6	37.6	67.4	71.0	33.7
+ HardTxt + RandTxt+ HardImg	61.7	62.0	53.1	33.9	41.2	67.0	71.0	30.8
DiffusionITM performance: Our DiffusionITM significantly outperforms Diffusion Classifier on all image retrieval tasks (Tab. 1a) while CLIP still outperforms DiffusionITM. However on text retrieval, DiffusionITM even outperforms CLIP on some tasks (Tab. 1b), most notably compositionality-focused Winoground Text and CLEVR by a large margin. Here DiffusionITM is clearly behind CLIP on only two ARO subtasks (Flickr and COCO Order).
With the exception of 20 steps for bias evaluation.
HardNeg-DiffusionITM performance: We find that HardNeg-DiffusionITM trained on MS-COCO transfers well to all tasks, outperforming DiffusionITM (Tab. 1). In Tab. 2 we disentangle the effect of using higher quality data and various additional hard negatives. We highlight three insights: 1) Despite fine-tuning on only a single dataset, we observe gains across almost all tasks without any negatives (NoNeg) explicable by the general high-quality MS-COCO dataset compared to more noisy pre-training data. 2) Using hard negatives for only one modality (HardNegTxr/RandNegTxt vs. HardNegImg) only improves the respective retrieval performance while showing occasional drops in the other.7 3) We therefore combine all three types of hard negatives, allowing us to work with one model, HardNeg-DiffusionITM, rather than multiple models specific to each modality retrieval.
Table 3: Effect sizes for the bias evaluation. Positive effect sizes indicate bias towards target group X, negative effect sizes indicate bias towards Y. Effect sizes closer to 0 are less biased and statistically significant effect sizes at  $p < 0.01$  are denoted by  $*$ . We see that all models exhibit biases, with SD 2.1 being the least biased. For brevity, Buddhist scores are omitted here but contribute to the average (details in Tab. 5).
Target X	Target Y	SD 2.1	SD 1.5	CLIP RN50x64	CLIP ViT-B/32
Religion	Christian	Muslim	0.94*	1.06*	1.55*	1.71*
Christian	Jewish	1.10*	1.11*	1.54*	1.69*
Jewish	Muslim	-0.15	0.03	0.23*	0.48*
Hindu	Muslim	0.86*	1.21*	1.48*	1.65*
Nationality	American	Arab	0.63*	0.90*	0.72*	1.28*
Sexuality	Heterosexual	LGBT	0.84*	1.04*	1.38*	1.68*
Average Absolute Effect Size	0.65	0.79	1.09	1.26
Stable Diffusion 1.5 vs. 2.1 Performance: 2.1 improves on the majority of GDBench tasks except for ARO. With GDBench's diverse tasks, we can study if later versions of Stable Diffusion improve in all skill-dimensions or just some of them (see Fig. 2). Interestingly SD 2.1 does not show significant gains over SD 1.5 on all of them. Most notable is ARO (compositionality): We see only minor improvements (VG tasks) or slight drops (Order tasks). At the same time, we do see a jump on Winoground Text from  $29\%$  to  $32.3\%$  and on other less adversarial tasks such Flickr30K or SVO.
Bias: Both CLIP and Stable Diffusion exhibit bias towards the dominant groups, namely Christians, Americans, and Heterosexuals (Tab. 3) with Stable Diffusion 2.1 displaying the least bias. Almost all scores are statistically significant for  $p < 0.01$ , with the exception of the Jewish-Muslim for both SD versions (Tab. 3) and some of the Buddhist scores (Tab. 5). Version 2.1 has overall lower effect sizes (average absolute effect size of 0.65 in 2.1 vs. 0.79 in 1.5), suggesting that it is less biased than version 1.5 for this metric (Tab. 3). This goes against the trend of increased bias in stronger models [Nadeem et al., 2021]. On the other hand, Luccioni et al. [2023] found that Stable Diffusion 1.4 is less biased than 2.0. Further investigation is needed to draw a strong conclusion as to whether the 2.x versions are more or less biased than the 1.x versions due to this discrepancy. It is important to note that there was a major weakening of the safety filter between version 2.0 and 2.1, which may have affected the diversity in the training set and as such, model bias.
Analysis: We find higher image-text-alignment of images generated by HardNeg-DiffusionITM model based on human judgement. Although our method improves discriminative performance, does it also result in more compositional image generation? Crucially our finetuning on MS-COCO preserved the generative capabilities despite directly modifying the noise prediction. We therefore compare image-text-alignment of DiffusionITM against HardNeg-DiffusionITM on DrawBench [Saharia et al., 2022] and find promising results: From 105 complex prompts, we found that HardNeg is closer to the text almost twice as often as the zero-shot model. Similarly, HardNeg finetuning also shows slightly better image-text-alignment than NoNeg finetuning. For more DrawBench details see (Appendix B) and other analyses (Appendix F). One might ask: how complementary are the skills learned in a generative vs. a discriminative vision-and-language model? We quantify this via the overlap of correctly predicted examples. Our hypothesis: Even if DiffusionITM might have lower performance on a task, its correct predictions may still cover new examples that discriminative models fail to capture. On three datasets, we compare DiffusionITM and two discriminative models (CLIP and BLIP) that were trained differently enough to expect varying predictions. However we find no evidence for
<sup>7</sup>Hard negatives for text are obtained by shuffling parts-of-speech, and images through CLIP image similarity.
our hypothesis and perhaps even signs of an opposite trend (Fig. 8). We speculate that this points towards the big role the text encoder behind an text-to-image model plays for vision-and-language understanding. After all, Stable Diffusion relies on a frozen CLIP text encoder.
6 Post-submission: The intriguing case of Stable Diffusion XL
After the paper submission, Stable Diffusion XL (SDXL) [Podell et al., 2023] was released so it was a reasonable next step to include it in our comparison of different SD versions (see Fig. 2). We expected SDXL to outperform previous versions, based on our upwards trend from 1.5 to 2.1 as well as findings in Podell et al. [2023]. Surprisingly, SDXL reaches significantly lower scores, below 2.1 scores on everything except three ARO subtasks, and even below 1.5 on most tasks (App. F Fig. 10 show exact numbers). Moreover, the top predictions of SDXL have the same exact scores (i.e. two images being ranked first) far more often than in SD 2.1. We confirmed that our results are not coming from a bug in our implementation and hope that future work can shed more light on quantifying the higher-level abilities of SDXL. In the meantime we offer a preliminary interpretation here: It is possible that the capabilities of image generation and image editing, specifically providing a new prompt on a partially noised image, are not always correlated. In light of this, we also offer two broader interpretations for the validity of DiffusionITM as a new evaluation methodology: Either our proposed evaluation is not generally applicable to all forms of vision-and-language skills in text-to-image models and instead measures more nuanced aspects such as image editing or text sensitivity. Alternatively, this anomaly is evidence that our evaluation is precisely working as intended and exposes flaws that might otherwise taken longer to detect via quantitative evaluation or other metrics such as FID or TIFA.
7 Conclusion and Future Directions
In this paper, we introduce DiffusionITM and GDBench. DiffusionITM allows us to transform any diffusion-based generative model to perform image-text matching tasks. This in turn leads to an exciting new paradigm of evaluating text-to-image models on vision-and-language reasoning using diagnostic benchmarks. We also improve vision-and-language understanding of diffusion models with a novel hard negative finetuning strategy (HardNeg). GDBench allows one to study many different types of vision and language reasoning, ranging from: compositional (ARO, Winoground) and visual fine-grained reasoning (ImageCoDe), to elements of spatial/attribute binding (CLEVR). Through our experiments, we find that our proposed DiffusionITM approach shows vision-and-language reasoning capability through increased performance on the several evaluations, as well as provides for head-on automatic comparison among generative models. We conclude that Stable Diffusion performs competitively to CLIP, and performs better on compositional tasks. We also conclude that SD 2.1 is less biased than SD 1.5. We hope that this line of work demonstrates that high-quality diffusion methods for generating images performs well on vision-and-language reasoning. We see that the simple task of image-text-matching allows us to test many types of reasoning capabilities by carefully selecting the appropriate negatives. We encourage future research to explore this paradigm on other models families like Masked Generative Transformers [Chang et al., 2023], and with stronger backbone text encoders. While we found that improved discriminative performance translates into better image-text-alignment in generated images, this should be explored in more detail. We discuss detailed limitations (especially bias-related) in Appendix A
We are grateful to the open-source community behind the Huggingface Diffusers library and the anonymous reviewers for their useful suggestions. This project was funded by the Mila-Samsung and Mila-Google grant program. SR acknowledges the support of the NSERC Discovery Grant program and the Facebook CIFAR AI Chair program. CP acknowledges the support of the Canada CIFAR AI program.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=KRLUvxh8uaX.
Kankan Zhou, Eason Lai, and Jing Jiang. VLStereoSet: A study of stereotypical bias in pre-trained vision-language models. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 527-538, Online only, November 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.aacl-main.40.
Our method is limited to image-text matching and cannot tackle other task formulations such as VQA [Agrawal et al., 2015, Suhr et al., 2019]. Additionally, taking more noise samples for a given image-text pair leads to better performance up to a certain point (see Fig. 9). This results in slower inference time and can hopefully be mitigated with future innovations.
Measuring bias is a crucial aspect of model evaluation and should not be considered secondary to the "performance" of the model. Unfortunately, the dataset from Janghorbani and De Melo [2023] does not include gender bias but other datasets such as Zhou et al. [2022] could also be incorporated. In any case, evaluating bias related to personal identities such as nationality, race, and sexual orientation presents certain limitations. These labels are nuanced and not always accompanied by visible identifying characteristics. Consequently, image datasets depicting these groups have limited capacity to fully represent these demographics and intersectional identities. While we recognize that assigning a bias score based on these limited resources might not be entirely accurate, it is a vital first step in the right direction. It is also important to note that the bias evaluation has only positive predictive power, meaning that a large effect size is an indication of bias, but a low one does not necessarily verify that the model is fair. Moreover, the bias effect size (Eq 8) may sometimes be unreliable [Meade et al., 2022]. This should serve as a preliminary analysis of biases present in the model, which require further investigation using more nuanced and comprehensive methods, including broader qualitative analysis and consultation with social scientists.
B DrawBench Evaluation
As described in Sec. 5, we manually compare our best HardNeg Stable Diffusion with vanilla Stable Diffusion on the DrawBench benchmark [Saharia et al., 2022]. It contains 200 curated challenging prompts along 11 categories. We only select these categories that are relevant to the phenomena studied in this paper (spatial, compositional, attribute binding, ...) and drop categories such as Text (i.e. A storefront with 'Hello World' written on it.). This leaves us with 6 categories and 104 prompts:
Category	Prompts
Colours	A brown bird and a blue bear.
Conflicting	A horse riding an astronaut.
Counting	One cat and three dogs sitting on the grass.
DALL-E	An illustration of a small green elephant standing behind a large red mouse.
Gary Marcus et al.	An elephant is behind a tree. You can see the trunk on one side and the back legs on the other.
Positional	A train on top of a surfboard.
Table 4: DrawBench categories with examples
Next we generate images for each prompt with both models and display them randomly to an author as left or right image. We judge whether the left or right image is better aligned with the text and do not focus on image quality. Because we only focus on high-level alignment with text, we observe many ties as both models exhibit the same level of alignment. We repeat this process with another seed, leaving us with 208 blind comparisons. Out of those, our HardNeg model is judged better almost twice as often as vanilla Stable Diffusion (60 vs. 33). The rest (115) are ties.
We show an example where HardNeg is more text-aligned as well as a tie below:
For a comprehensive discussion on the limitations of fixed labels inferable from a person's appearance, creating datasets based on such visual factors for bias analysis, as well as the discussion on the consequences of biased image generation systems, refer to Luccioni et al. [2023].
(a) Prompt: A stack of 3 books. A green book is on the top, sitting on a red book. The red book is in the middle, sitting on a blue book. The blue book is on the bottom. Zero-shot (left) gets it completely wrong while our finetuned Stable Diffusion (right) is quite close to the text.
(b) Prompt: A wine glass on top of a dog. One of many examples where both models either fail or succeed to an equal extent to capture the text.
Figure 5: Examples of DrawBench prompts with vanilla Stable Diffusion and HardNeg Stable Diffusion.
Since HardNeg is not strictly better than NoNeg in our experiments (see Tab. 2), we conduct the same study with these two models: HardNeg wins 50 comparisons and NoNeg only 38. While this is not statistically significant, it indicates that HardNeg finetuning is useful for image-text-alignment.
C Visualizing image editing with input optimization and standard denoising
Optimizing the input image directly via backpropagation has shown promise in the diffusion literature [Poole et al., 2022; Samuel et al., 2023]. It also leads to qualitatively insightful images that differ from standard image editing with Stable Diffusion. Concretely, we optimize the input image (i.e. the latent  $z$ ) with Adam ( $\mathrm{lr} = 0.05$ , 200 steps) based on the noise prediction loss where noise is added at timestep  $t = 0.5$ . In other words: the input image itself is modified such that it leads to a lower noise prediction error. Note that this was generated with Stable Diffusion 1.5 before we adopted 2.1 into our experiments.