SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Benno Krojer
Mila & McGill University
benno.krojer@mila.quebec
Elinor Poole-Dayan
Vikram Voleti
Mila & University of Montreal
Christopher Pal
Mila & Polytechnique Montréal
Canada CIFAR AI Chair
ServiceNow Research
Siva Reddy
Facebook CIFAR AI Chair
Text-conditioned image generation models have recently shown immense qualitative success using denoising diffusion processes. However, unlike discriminative vision-and-language models, it is a non-trivial task to subject these diffusion-based generative models to automatic fine-grained quantitative evaluation of high-level phenomena such as compositionality. Towards this goal, we perform two innovations. First, we transform diffusion-based models (in our case, Stable Diffusion) for any image-text matching (ITM) task using a novel method called Diffusion-ITM. Second, we introduce the Generative-Discriminative Evaluation Benchmark (GDBench) benchmark with 7 complex vision-and-language tasks, bias evaluation and detailed analysis. We find that Stable Diffusion + DiffusionITM is competitive on many tasks and outperforms CLIP on compositional tasks like CLEVR and Winoground. We further boost its compositional performance with a transfer setup by fine-tuning on MS-COCO while retaining generative capabilities. We also measure the stereotypical bias in diffusion models, and find that Stable Diffusion 2.1 is, for the most part, less biased than Stable Diffusion 1.5. Overall, our results point in an exciting direction bringing discriminative and generative model evaluation closer. We are releasing code and benchmark setup. $^{1}$
Text-to-image generation is rapidly advancing. Generated images are not only highly realistic in various styles, but also reflect the compositional structure of open-ended text prompts [Chang et al., 2023, Sahara et al., 2022, Li et al., 2022b]. In this work, we evaluate language-conditioned generative image models on discriminative tasks to shed light on their fine-grained understanding of vision and language. A generative objective trains a model to understand how various objects and parts compose together, and often brings non-trivial emergent capabilities with it such as latent interpolation of composite concepts [Brock et al., 2019, Rombach et al., 2022]. On the other hand, discriminative vision-and-language models need only focus on the minimal information required to solve their discriminative task, which could often be spurious correlations that don't generalize [Agrawal et al., 2016]. Such erroneous understanding is then exposed downstream on image-text matching tasks purposefully designed to catch it, such as image/text retrieval using Winoground [Thrush et al., 2022], ARO [Yuksekgonul et al., 2023], or ImageCoDe [Krojer et al., 2022]. Following these benchmarks' releases, there has been a growing focus in the vision-and-language community to fix these problems.
$^{1}$ https://github.com/McGill-NLP/diffusion-itm
Figure 1: DiffusionITM allows us to apply a diffusion model to any image-text-matching task. It overcomes the asymmetry between image and text retrieval that previously led to random performance on image retrieval via unconditional normalization: An image is selected based on the lowest noise prediction error when conditioned on the text (upper part of figure) which is normalized by the noise prediction error without text-conditioning (lower part). With this general method the image generation community can benchmark their models on complex vision-and-language tasks such as Winoground [Thrush et al., 2022] or ImageCoDe [Krojer et al., 2022].
Figure 2: Progress of Stable Diffusion from 1.5 to 2.1 on GDBench tasks. GDBench allows fine-grained comparison of models.
We hypothesize that a generative model trained to synthesize compositional data is capable of understanding the complexities required to solve hard image-text-matching tasks. To this end, we transform a text-to-image generative model for zero-shot image-text matching, and introduce Diffusion Image-Text Matcher (DiffusionITM; Fig. 1). In this work, we use Stable Diffusion (SD) [Rombach et al., 2022] as the text-to-image model, but any other diffusion model could be used. DiffusionITM achieves competitive zero-shot performance on both image and text retrieval (Tab. 1).
The naive approach for image retrieval given a text prompt would be to pick the image for which SD gives the least noise prediction error, and vice versa for text retrieval. While this works well for text retrieval [Li et al., 2023], we show it achieves random performance on image retrieval (Tab. 1). Our main insight explaining this discrepancy is that the model's success at denoising depends primarily on the visual properties of the scene, rather than equally on visuals and text (Fig. 3). Therefore, if such a model has to select among several images given a text prompt, it will have the lowest noise prediction error for a visually familiar image regardless of the text prompt. To address this, we compute the error relative to the unconditional (i.e. no text) error (Fig. 1). Our method outperforms existing diffusion-based discriminative methods [Li et al., 2023, Clark and Jaini, 2023] (see Tab. 1).
Since the original generative pretraining objective of noise prediction is far from the ultimate downstream task of image-text-matching, we explore a novel discriminative finetuning scheme with hard negative image-text-pairs on MS-COCO [Lin et al., 2014]. We find that this transfers well to other datasets, and improves discriminative performance (Tab. 1) as well as generated images from DrawBench prompts [Saharia et al., 2022].
Finally, we present the GDBench to foster research progress on image generation. Our DiffusionITM method enables a new automatic, fine-grained, and downstream way to evaluate diverse skills in text-conditioned image generation. Thus, once we cast an image generation model as a discriminator, we can draw from the rich history of diagnostic datasets and detailed analyses in the vision-and-language literature to analyze the model's reasoning ability. Evaluation of image generation has been notoriously difficult, and recent proposals advocate using secondary model probes on generated images to test for fine-grained ability [Hu et al., 2023, Cho et al., 2022]. In contrast, we directly evaluate diffusion models on downstream tasks without requiring any probes. In the same spirit as GLUE [Wang et al., 2018], we systematically select 7 image-text-matching tasks covering diverse reasoning skills from traditional image retrieval to diagnostic tasks (i.e. compositionality). In addition, GDBench also includes a bias evaluation dataset which we use for social bias analysis.
GDBench allows head-on comparison between generative models, as well as with discriminative models like CLIP [Radford et al., 2021]. Our results are as follows:
Stable Diffusion (SD) vs. CLIP: SD is competitive with CLIP on many tasks, and outperforms it on challenging compositional tasks like CLEVR and Winoground Text (Tab. 1). Further tuning of SD on MS-COCO with hard negatives outperforms vanilla SD on both image and text retrieval (Tab. 2). SD has lower stereotypical bias than CLIP (Tab. 3).
Stable Diffusion (SD) 1.5 vs 2.1: We observe varying degrees of progress, but overall SD 2.1 outperforms SD 1.5 (Fig. 2). SD 2.1 is less biased than SD 1.5, contrary to the common trend of increased bias in larger, stronger models [Nadeem et al., 2021].
Image generation: Remarkably, improved discriminative performance via finetuning on MS-COCO also leads to improved high-level understanding of image generation. We find higher image-text-alignment on DrawBench [Saharia et al., 2022] prompts compared to vanilla SD (Appendix B).
Vision-And-Language Understanding: Widely popular tasks for vision-and-language understanding are often framed as discriminative tasks, such as image retrieval [Miech et al., 2021], VQA [Agrawal et al., 2015] or REC [Yu et al., 2016]. We focus on image-text-matching (ITM) tasks (assigning a score to an image-text-pair) due to their general applicability and simplicity. Noisy image-text pairs are used for large-scale ITM training using contrastive learning [Radford et al., 2021], and has been traditionally tackled either via dual encoders like CLIP [Radford et al., 2021], or models with richer cross-modal interaction like BLIP [Li et al., 2022a]. At the same time, ITM allows probing models in a controlled manner with simple metrics [Thrush et al., 2022, Hendricks and Nematzadeh, 2021], as opposed to more complex metrics on generative tasks [Hessel et al., 2021, Sahara et al., 2022]. There has been a growing interest on diagnostic benchmarks for compositionality that use hard negatives [Yuksekgonul et al., 2023, Thrush et al., 2022, Krojer et al., 2022, Hendricks and Nematzadeh, 2021, Lewis et al., 2022, Parcalabescu et al., 2022]. This rich literature can now be transferred to the world of image generation with our GDBench benchmark, enabling fine-grained analysis and benchmarking of text-conditioned image generation.
Repurposing Text-Conditioned Diffusion Models: Text-conditioned diffusion has shown remarkable advancements not only in image quality but also image-text alignment [Rombach et al., 2022, Sahara et al., 2022]. A natural next question is how to leverage these abilities for other tasks [Burgert et al., 2022]. Two concurrent studies use similar methods to our diffusionITM in a more restricted setting: Diffusion Classifier [Li et al., 2023] performs image classification using SD by selecting the class with the lowest noise prediction error; Clark and Jaini [2023] puts more focus on timestep-weighting, and uses Imagen [Saharia et al., 2022] instead of SD. However, they only show competitive results for text retrieval, emphasizing image classification as a special case. Many vision-and-language reasoning tasks are also framed as image retrieval [Krojer et al., 2022, Hendricks and Nematzadeh, 2021, Thrush et al., 2022]. In this work, we tackle the broader scope of vision-and-language understanding, generalize our method to image retrieval, and study hard-negative fine-tuning and transfer.
Evaluation of Image Generation: Image generation is traditionally evaluated along the two axes of image quality and image-text alignment, using metrics based on individual examples. The recently proposed TIFA metric [Hu et al., 2023] relies on an additional VQA model to answer a set of LLM-generated questions about the generated image. More traditional metrics include FID [Heusel et al.,
2017] for image quality; CLIPScore [Hessel et al., 2021, Kim et al., 2022] for image-text alignment based on CLIP-embedding; object-centric metrics [Hinz et al., 2020, Cho et al., 2022] leveraging object detectors such as DETR [Carion et al., 2020]; and caption-based metrics [Hong et al., 2018] like BLEU on captions of the generated image. In contrast, GDBench is not a metric on individual examples, but rather a holistic evaluation framework. GDBench does not require another large model (e.g. VQA) for evaluation, and can be run on many diverse datasets.
Bias in Image Generation Models: It is well known that LLMs learn and amplify harmful biases present within text training corpora [Caliskan et al., 2017]. Due to the lack of automatic evaluation techniques for generative models, bias investigation has mostly focused on discriminative vision-and-language (VL) models [Srinivasan and Bisk, 2022, Janghorbani and De Melo, 2023]. Only few works have tackled bias in recent text-to-image models [Luccioni et al., 2023, Cho et al., 2022] and to our knowledge only Luccioni et al. [2023] focus on bias alone: They quantify social biases by generating images over several social groups (ethnicity and gender) and measuring their variation over selected attributes (gendered adjectives and professions). They found that SD 1.4 and 2.0 are biased towards groups "associated with whiteness and masculinity" across target attributes, and that SD 2.0 was more biased than 1.4. While their methods are thorough and work for black-box systems, the evaluation is quite time consuming and manual.
3 Our Approach to Image-Text Matching with Diffusion Models
3.1 Diffusion Image-Text Matching: Overcoming the modality asymmetry for image retrieval
We present our method Diffusion Image-Text Matching (ITM). Our goal is to assign a score to an image(x)-text(w) pair  $(\mathbf{x},\mathbf{w})$  which is broadly useful for downstream applications. We provide  $(\mathbf{x},\mathbf{w})$  to the diffusion model and task it to "edit" the image according to the text. Our main intuition is if the image is not described by the text, a lot of edits are needed to fit the text, in which case it gets a low score, and vice-versa. See Appendix C for visualization.
Text-conditioned Diffusion: The objective of diffusion models is to denoise an image  $\mathbf{x}$  by predicting the noise  $\epsilon$  added to its clean version, conditioned on text  $\mathbf{w}$  and noise level  $t$ :
$$
D i f f u s i o n \quad \mathbb {E} _ {\mathbf {x}, \epsilon , t} \left[ \| \epsilon - \epsilon_ {\theta} (\mathbf {x}, t, \mathbf {w}) \| _ {2} ^ {2} \right] \tag {1}
$$
Intuitively, the predicted noise is farther from the true noise when the image-text do not fit together. To transform this for ITM tasks, the sample with the lowest L2-distance of predicted and true noise could be used to select among a set of image-text-pairs. Li et al. [2023] and Clark and Jaini [2023] (concurrent works) have focused primarily on text retrieval (with classification as a special case), where the model selects from a number of texts (or class names for classification):
$$
\text {T e x t r e t r i e v a l :} \quad \underset {\mathbf {w}} {\arg \min } \mathbb {E} _ {\epsilon , t} \left[ \| \epsilon - \epsilon_ {\theta} (\mathbf {x}, t, \mathbf {w}) \| _ {2} ^ {2} \right] \tag {2}
$$
However, naively applying this in practice would imply sampling a different  $\epsilon$  for each pair  $(\mathbf{x},\mathbf{w})$  to reduce the variance in L2-distance. Li et al. [2023] a) sample many (hundreds!) noise-timestep pairs  $(\epsilon ,t)$  uniformly, and b) crucially keep the sampled  $(\epsilon ,t)$  constant across different  $\mathbf{w}$  when calculating Equation 2. Finally, the guidance scale is kept at 0, thereby discarding unconditional noise prediction. This could be repurposed to perform image retrieval by iterating over images instead of text:
$$
\text {N a i v e i m a g e r e t r i e v a l :} \quad \underset {\mathbf {x}} {\arg \min } \mathbb {E} _ {\epsilon , t} \left[ \| \epsilon - \epsilon_ {\theta} (\mathbf {x}, t, \mathbf {w}) \| _ {2} ^ {2} \right] \tag {3}
$$
We observe that while this results in SD as a strong text retriever, it achieves random chance on all image retrieval tasks! Interestingly, Li et al. [2023] observed that a Bayesian posterior from a discrete-label class-conditional generative model can be approximated using the analysis:
$$
p _ {\theta} \left(\mathbf {c} _ {i} \mid \mathbf {x}\right) = \frac {p \left(\mathbf {c} _ {i}\right) p _ {\theta} \left(\mathbf {x} \mid \mathbf {c} _ {i}\right)}{\sum_ {j} p \left(\mathbf {c} _ {j}\right) p _ {\theta} \left(\mathbf {x} \mid \mathbf {c} _ {j}\right)} \approx \frac {\exp \left\{- \mathbb {E} _ {t , \epsilon} \left[ \| \epsilon - \epsilon_ {\theta} \left(\mathbf {x} _ {t} , \mathbf {c} _ {i}\right) \| ^ {2} \right] + c o n s t . \right\}}{\sum_ {j} \exp \left\{- \mathbb {E} _ {t , \epsilon} \left[ \| \epsilon - \epsilon_ {\theta} \left(\mathbf {x} _ {t} , \mathbf {c} _ {j}\right) \| ^ {2} \right] + c o n s t . \right\}}. \tag {4}
$$
Li et al. [2023] use pairwise samples between pairs of labels  $\mathbf{c}_i$  and  $\mathbf{c}_j$ , to produce a low complexity approximate computation reminiscent of paired differences tests in statistics. Their analysis is based on a marginal across all possible discrete label captions  $\mathbf{c}$  with respect to an image  $\mathbf{x}$ . In
Figure 3: Denoising losses for two similar Flickr30K images Image1 and Image2, and Caption1 of Image1. The absolute denoising error of Image2-Caption1 is smaller than that of Image1-Caption1, hence Diffusion Classifier would have erroneously picked Image2 for Caption1. Whereas, the difference between the errors for Image1-Caption1 and Image1-Unconditional is greater than between Image2-Caption1 and Image2-Unconditional, so our approach would correctly pick Image1.
contrast, in our approach, we replace the marginal in the denominator with simply a sample from the unconditional diffusion model for  $p(\mathbf{x})$ , and operate in log space. The intuition behind our approach is that taking an integral over all possible caption strings ( $\mathbf{w}$ ) is equivalent to simply the unconditional, since the "caption" dimension is (implicitly) marginalized out. Fig. 3 illustrates another view of this intuition that for a correct text-image pair (Image1-Caption1), the denoising error is visibly smaller than for all mismatched text-image pairs for the same image (Image1). Moreover, for an incorrect but similar image (Image2), the denoising error for Caption1 is close to random, but could be less than that of Image1-Caption1 in absolute value. The incorrect Image2 would be selected regardless of the text since it is visually easier to denoise. Hence, the denoising error depends almost entirely on the image, independent of text conditioning ("modality asymmetry").
This analysis naturally leads to the intuitive solution of normalizing the error by the unconditional (no text) error. After all, we only care about the relative difference of how much easier or harder it becomes to denoise the image with a given text relative to when no text is given (see Fig. 1):
$$
\text {I m a g e r e t r i e v a l :} \quad \underset {\mathbf {x}} {\arg \min } \mathbb {E} _ {\epsilon , t} \left[ \left(\| \epsilon - \epsilon_ {\theta} (\mathbf {x}, t, \mathbf {w}) \| _ {2} ^ {2} - \| \epsilon - \epsilon_ {\theta} (\mathbf {x}, t) \| _ {2} ^ {2}\right) \right] \tag {5}
$$
3.2 HardNeg-DiffusionITM: Tuning with compositional hard negatives and transfer
Our goal is to transform diffusion-based models for discriminative image-text-matching (ITM). However, the denoising diffusion objective only considers positive image-text pairs, and the large pre-training corpus LAION [Schuhmann et al., 2021] contains many noisy/simple examples, not conducive to complex linguistic reasoning. In contrast, models specifically dedicated to vision-and-language reasoning such as CLIP [Radford et al., 2021] or BLIP [Li et al., 2022a] were pre-trained with negatives and, in the case of BLIP, had access to high-quality curated image-text data. Previous baselines [Li et al., 2023] ditched the generative objective and turned it fully discriminative by finetuning a ResNet on top of the frozen mid-layer features of the U-Net for each dataset separately. We instead adopt parameter-efficient finetuning with LORA layers [Hu et al., 2022] that are added to the cross-attention from U-Net to the text, so as not to deviate too far from pretraining representations.
We address the lack of high-quality image-text-data by fine-tuning the diffusion model on MS-COCO (109K examples) with the standard diffusion objective (see Equation 1). As MS-COCO contains diverse high-quality image-text pairs, we finetune only once, and evaluate using GDBench tasks. This could be thought of as a second limited pre-training.
We address the lack of negative examples by adopting the hard negatives from Yuksekgonul et al. [2023] on MS-COCO: swapped text elements of the same part-of-speech (e.g. "Men keep watch on a herd of goats"  $\rightarrow$  "Goats keep watch on a herd of men"), and CLIP-based image hard negatives. The naive approach would be to minimize the noise prediction error on positive pairs  $(\mathbf{x},\mathbf{w}_{pos})$ , and maximize for negative pairs  $(\mathbf{x},\mathbf{w}_{neg})$ . However, if this inverse loss were applied unhinged with potentially infinite gains, it would lead to the model predicting non-sense for everything. Therefore we threshold the hard-negative error at a relative scaling factor  $\lambda$  of the value of the positive error:
$$
\mathcal {L} = \mathcal {L} _ {p o s} + \operatorname {c l i p} \left(\mathcal {L} _ {n e g}, | \lambda \mathcal {L} _ {p o s} |\right) \text {w h e r e} \tag {6}
$$
$$
\mathcal {L} _ {p o s} = \mathbb {E} _ {\mathbf {x}, \epsilon , t} \left[ \| \epsilon - \epsilon_ {\theta} (\mathbf {x}, t, \mathbf {w} _ {p o s}) \| ^ {2} \right], \quad \mathcal {L} _ {n e g} = - \mathbb {E} _ {\mathbf {x}, \epsilon , t} \left[ \| \epsilon - \epsilon_ {\theta} (\mathbf {x}, t, \mathbf {w} _ {n e g}) \| ^ {2} \right] \tag {7}
$$
We choose relative thresholding since we want to ensure that the model never deviates too much from its original objective of noise prediction from positive prompts. Hence  $\mathcal{L}_{neg}$  is clipped between
Flickr30K: A lady in a blue dress riding on a scooter (left) and CLIP-based hard negative (right)
ARO: The man is wearing the shirt (positive) and The shirt is wearing the man (hard negative)
Pets: A photo of a english cocker spaniel
CLEVR: A large sphere and a small cylinder (positive) and A small sphere and a large cylinder (negative)
SVO: Person standing at window (left) and Person sitting at window (right)
ImageCoDe: There is an equal amount of yellow and white between both hands (left) and hard negative (right)
Figure 4: Examples from the datasets in GDBench.
Winoground: Some plants surrounding a lightbulb (left) and A lightbulb surrounding some plants (right)
Bias: Christian (left), Jewish (right). An unbiased model should match "pleasant" and "unpleasant" words equally to both groups.
$[- \mathcal{L}_{pos}, \mathcal{L}_{pos}]$ . Unlike CLIP, we cannot include a large number of hard negatives in our batches since diffusionITM encodes image and text together. The resulting model, HardNeg-DiffusionITM, is still evaluated in a zero-shot fashion on the target evaluation tasks, i.e., how well does DiffusionITM trained on MS-COCO transfer to target tasks.
4 Data: The GDBench Benchmark
There is a fundamental need to measure downstream performance of diffusion-based generative models on a wide range of vision-and-language reasoning tasks, to facilitate quick improvement and progress tracking. This has worked in the past, i.e. with the NLP GLUE benchmark [Wang et al., 2018]. With this motivation, we introduce GDBench, a benchmark of eight diverse image-text-matching (ITM) tasks to explore many types of vision-and-language reasoning. These include 7 ability-centric and 1 bias dataset for the image generation community (examples shown in Fig. 4). Additionally, most GDBench tasks have further fine-grained scores on sub-phenomena without requiring manual inspection, as is the case with evaluating generative models [Saharia et al., 2022]. ITM as a reasoning benchmark offers simplicity and a surprising amount of diversity. After all, ITM has become a standard paradigm for diagnostic vision-and-language datasets (see task list below) and therefore allows interpretable evaluation on many downstream skills. GDBench is similar to the spirit of downstream evaluation of generative models such as in TIFA evaluation [Hu et al., 2023] which makes use of a secondary model like VQA on the generated images and shows that downstream performance correlates with image quality. Whereas, we stay closer to the generative realm and evaluate generative models on several discriminative tasks without the need for secondary models. Below we introduce each dataset, asking "What phenomena does it cover that others do not?"