Text-to-Image Diffusion Models are Zero Shot Classifiers
Authors: Kevin Clark
Priyank Jaini
Venue: NeurIPS 2023 spotlight (2023)
Keywords: diffusion models
zero-shot
text-to-image
generative models
foundation models
stable diffusion

Abstract: The excellent generative capabilities of text-to-image diffusion models suggest they learn informative representations of image-text data.
However, what knowledge their representations capture is not fully understood, and they have not been thoroughly explored on downstream tasks.
We investigate diffusion models by proposing a method for evaluating them as zero-shot classifiers.
The key idea is using a diffusion model's ability to denoise a noised image given a text description of a label as a proxy for that label's likelihood.
We apply our method to Stable Diffusion and Imagen, using it to probe fine-grained aspects of the models' knowledge and comparing them with CLIP's zero-shot abilities. 
They perform competitively with CLIP on a wide range of zero-shot image classification datasets. 
Additionally, they achieve state-of-the-art results on shape/texture bias tests and can successfully perform attribute binding while CLIP cannot.
Although generative pre-training is prevalent in NLP, visual foundation models often use other methods such as contrastive learning. 
Based on our findings, we argue that generative pre-training should be explored as a compelling alternative for vision and vision-language problems.

Text-to-lmage Diffusion Models are Zero-Shot Classifiers
Kevin Clark* Google DeepMind Toronto kevclark@google.com com
Priyank Jaini" Google Deep.Mind Toronto pjaini@google.com com
Abstract
The excellent generative capabilities of text-to-image diffusion models suggest they learn informative representations of image-text data. However, what knowl- edge their representations capture is not fully understood, and they have not been thoroughly explored on downstream tasks. We investigate diffusion models by proposing method for evaluating them zero-shot classifiers. The key idea is using diffusion model's s ability to denoise a noised image given a text description of a label as proxy for that label's likelihood. We apply our method to Stable Dif- fusion and Imagen, using to probe fine grained aspects the models' knowledge and comparing them with CLIP's zero-shot abilities. They perform competitively with CLIP on a wide range of zero-shot image classification datasets. Additionally, they achieve state-of-the-are results shape/texture bias tests and successfully perform attribute binding while CLIP cannot. Although generative pre-training is prevalent NLP, visual foundation models often use other methods such as contrastive learning. Based on our findings, we argue that generative pre-training should be explored compelling alternative for languae tasks.
Introduction
Large models pre-trained internet-scale data adapt effectively variety of downstream tasks. Increasingly, they are being used as zero-shot learners with no task-specific training, such as with CLIP (Radford et al., 2021) for images and GPT-3 (Brown al., 2020) for text. In natural language processing, many successful pre-trained models are generative (i.e., language models). However, generative pre-training is less commonly used for visual tasks. Until recently, the usual practice for vision problems was to pre-train models on labeled datasets such as Imagenet (Deng et al., 2009), or JFT (Sun al., 2017). Later research in visual and vision-language problems has led image-text models pre-trained primarily using either contrastive losses (Radford et al., 2021; Jia et al., 2021; Yuan et al., 2021) or autoencoding tasks (Vincent et al., 2010; He al., 2022).
On the other hand, generative ext-to-image models based on denoising diffusion probabilistic models (Ho al., 2020) such as Imagen (Saharia al., 2022a), Dalle-2 (Ramesh al., 2022), and Stable Diffusion (Rombach et al., 2022) can generate realistic high-resolution images and generalize to diverse text prompts. Their strong performance suggests that they learn effective representations of image-text data. However, their ability to transfer to downstream discriminative tasks and how they compare other pre-trained models has not been explored thoroughly.
In this paper, we investigate these questions by transferring Imagen and Stable Diffusion (SD) to discriminative tasks. While previous studies have used representations from diffusion models for downstream tasks (Brempong et al., 2022; Burgert et al., 2022; Zhao et al., 2023), we instead propose
"equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Build scores using Aq. prompts, c, Uni1 n= (1.2,n,,)
Classify using matrix and weighting function W.
condilioinn METEl
Cat
W
ofa Cat
Home
X
arg min
Original image
Nokay Image
photo
Frog
W
Scores
Vector
Scores Matrix
Timesteps Weighting
Image
Figure 1: Zero-Shot Classification using Diffusion Models. We first compute denoising scores for each label prompt across multiple time-steps to generate scores matrix. We then classify an image by aggregating the scores for each class using a weighting function over the time-steps. The image assigned the class with the minimum aggregate score. In Section 3.1, we discuss how efficiency can be improved only computing a subset of the full scores matrix.
a way of using text-to-image diffusion models directly as zero-shot image classifiers. Our method essentially runs the models generative classifiers (Ng & Jordan, 2001), using a re-weighted version of the variational lower bound to score images since diffusion models do not produce exact likelihoods. More specifically, the method repeatedly noises and denoises the input image while conditioning the model on a different text prompt for each possible class. The class whose text prompt results in the best denoising ability is predicted. This procedure expensive because it requires denoising many times per class (with different noise levels). To make it usable in practice, we present improvements that increase the method's 's sample efficiency by up to 1000x, such as pruning obviously-incone classes early. While still requiring too much compute to be an easily-deployable classifier, our method allows us to quantitatively study fine-grained aspects of a diffusion model's learned knowledge through evaluation classification tasks (as opposed to qualitatively examining model generations).
We compare Imagen and SD against CLIP² (Radford al., 2021), a widely used model for zero-shot image-text tasks trained with contrastive learning. A high-level goal of the experiments is to see the strengths and weaknesses of generative and contrastive pre-training for computer vision. First, we demonstrate that diffusion models have strong zero-shot classification accuracies (competitive with CLIP) on several diverse vision datasets. Next, we show both Imagen and SD performs remarkably well on the conflicting with their labels. For example, Imagen achieves >50% error reduction over CLIP and even outperforms the much larger ViT-22B (Dehghani et al., 2023) model. This finding is particularly interesting because, unlike supervised classifiers, humans are known be much more reliant on shape than texture when identifying images. Lastly, we study attribute binding using the synthetic data from Lewis et al. (2022), and find that, unlike CLIP, diffusion models can successfully bind together attributes in some settings.
dataset (Geirhos et al., 2019), where images have been stylized with textures
The main contributions of this paper are:
We show text-to-image diffusion models can be used as effective zero-shot classifiers. While using too much compute to be very practical downstream tasks, the method provides way of quantitatively studying what the models learn.
We develop techniques that hugely lower the compute cost of these zero-shot classifiers, making them usable (although still slow) on datasets with many classes.
We demonstrate the strong generalization capabilities of Imagen and Stable Diffusion, resulting in good zero-shot performance on vision datasets (comparable CLIP).
We show that diffusion models are robust to misleading textural cues, achieving state-of-the- art results on Cue-Conflict.
We use our framework to study attribute binding in diffusion models and find that they can perform some binding tasks while CLIP cannot.
We use VIT-L/14, the largest public CLIP model

---

Together, our results suggest that text-to- -oxtat diffusion models learn powerful representations that can effectively be transferred to tasks beyond image generation.
2 Preliminaries
Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020; Song & Ermon, 2020) are latent variable generative models defined by forward and reverse Markov chain. Given an unknown data distribution, ((*o), over observations, #o E RR, the forward process corrupts the data into a sequence noisy latent variables, #1.T == {01, 22, ... ,TT,, J. by gradually adding Gaussian noise with fixed schedule defined as:
(11r//0) == II (x)
where [[ == Normal( vI 1, The reverse Markov process gradually de- noises the latent variables to the data distribution with learned Gaussian transitions starting from Normal(æ<:: 0, i.e.
T
Po (o == p(ar))
II Pe (1x0))
Pe (e(xx-1/=;) := -or t). Eg (z), t)). The aim of training is for the forward process distribution {xt ) to match that of the reverse process {æ, { i.c., the generative model pe (x closely matches the data distribution g(z0). Specifically, these models can be trained by optimizing the variational lower bound of the marginal likelihood (Ho et al., 2020; Kingma et al., 2021):
log Po (xx) S VVLB(z)) == Cprir + CRecon + Diffusion
Lirior and LRecon are the prior and reconstruction loss that can be estimated using standard techniques the literature (Kingma Welling, 2014). The rri-hegghted) diffusion loss can written as:
Diffussin Exgge. un |aa to (æx 1)]
with zo q(zo), (0r I). and (1) Here, w, is weight assigned to the timestep, and ie (20, t) is the model's prediction the observation vo from the noised observation Φt- Diffusion models can be conditioned on additional inputs like class labels, text prompts, segmentation masks or low-resolution images, in which case Ig also takes a conditioning signal y as input.
Zero-Shot Classification using Diffusion Models
In this section, show how convert the generation process of text-to-imape diffusion model into a zero-shot classifier to facilitate quantitative evaluation on downstream tasks. Figure shows an overview of method.
Diffusion Generative Classifier: We begin with dataset, {a= y (y²),.... (a",,")) C Rdixds x
[yx] of n images² where each image belongs to one of classes [yx] = {y1, V2, ... .Kk. Given an image æ, our goal is predict the most probable class assignment
1 = arg max p(y yx(æ)) = arg max p(aly yx)) ply = yk) = arg max logp(æly = yи).
yk
where assume uniform prior p(y, yk) that can be dropped from the arg max. A generative classifier (Ng & Jordan, 2001) uses a conditional generative model with parameters 0 to estimate the likelihood as pe (xlly yι).
Using a text-to-image diffusion model as a generative classifier requires two modifications. First, the models are conditioned on text prompts rather than class labels. Thus we convert each label, yk. to text using mapping & with dataset-specific template (e.g. Y& - photo of yι). Second,
è For simplicity, we use æ place mg to refer to an image.
We can't learned prior in the shot setting.
3

---

diffusion models do not produce exact log- oggliikelhhooss (i.e. we cannot compute log ( ly = y&) directly). Our key idea for a solution use the VLB (more specifically Diifussion as Imagen and SD are not trained with the other losses) as proxy. Thus we have:
arg max log Poxlyy = yk.) = arg min Diifiusion (2c, ys)
Vk
yk
arg min E., wr|lx To , ,(yx).t)13 y
Note that for SD, z and žg are latent representations, with æ obtained by encoding the image using a VAE. With Imagen on the other hand, æ consists of the raw image pixels.
Estimating the Expectation: We approximate the expectation in Equation (2) using Monte-Carle estimation, At each step, we sample a t ~ [00 11) and then a H according to the forward diffusion process (Equation (1)): Zt ((x))(a)) Next, we denoise this noisy image using the model (i.e. we use it to predict from Zt),, obtaining a Teg (a,, Φ(y)), ()) We call the squared error of the prediction, || c 12. score for (æ, Yt)). We score each class N times, obtaining a K N scores matrix² for the image. Finally, we weight the scores according to the corresponding wi and take the mean, resulting in an estimate of LDiffusion for each class.
Choice of Weighting Function: Imagen and SD are trained with the "simple" loss, where w; SNR(t), the signal-to-noise ratio (Kingma al., 2021) for timestep t. However, we found other weighting functions can improve results. First, we experimented with learning times into 20 buckets and training a 20-features logistic regression model to learn weights for the buckets that maximize classification accuracy. However, using that weighting is not truly zero-shot since it requires label information to learn. We thus, also handcrafted weighting function that can be used across datasets. We designed w, by finding a simple function that looked close to our learned weighting function on CIFAR- 100 (we did not look other datasets to preserve zero-shot protocol). Interestingly, we found that the simple function w, := exp( (7)) works well for both Imagen an SI> across tasks and used it for our experiments. As is monotonic, LDiffusion with this weighting can still be viewed likelihood- data augmentations (Kingma & Gao, 2023). We provide details on learning w, and an empirical comparison of different weighting functions in Appendix B.
by binning the
objective that maximizes-the variational lower bound under simple
3.1 Improving Efficiency
Computing ÿ with naive Monte-Carle estimation can be expensive because Diffusiin has fairly high variance. Here, we propose techniques that reduce the classes. The key idea is to leverage the fact that we only need to compute the arg min and do not require good estimates of the actual expectations.
compute cost of estimating the arg min over
Shared Noise:
Differences between individual Monte-Cario samples from LDiffusion can of course be due to different or forward diffusion samples from q( xx-1),, whereas we are only interested in the effect of the text conditioning ((yx). We find far fewer samples are necessary when we use the same t and æ, across different classes, as shown in Figure 1. After sampling a t ~ U[O,, 1J) and I ~ q( 2o)) we score all classes against this noised image instead of single one. As result, the differences between these estimates are only due to the different text conditioning signals.
Candidate Class Pruning: Rather than using the same amount of compute to estimate the expec- lation for each class, we can further improve efficiency by discarding implausible classes early and dynamically allocating more compute to plausible ones. In particular, we maintain a set of candidate classes for the image being classified. After collecting new set of scores for each candidate class, we discard classes that are unlikely to become the lowest-scoring (i.e. predicted) class with more samples. Since we are collecting paired samples (with the same and i we use a paired student's s t-test to identify classes that can pruned. This pruning can be viewed as a succesive elimination algorithm for best-arm identification in a multi-armed bandit setting (Paulson, 1964; Evve-DDr et al., 2002). Of course, scores do not exactly follow the standard assumptions of a student's t-test, so we
LLter we discuss how we avoid computing the full matrix for efficiency.

---

use small p-value (2e in our experiments) and ensure each class is scored minimum number of times (20 in our experiments) minimize the chance of pruning the correct class. The full procedure is shown in Algorithm 1.
Comparison: Figure compares the number of samples needed to accurately classify CIFAR- 100 images for different efficiency strategies. Using shared noise and pruning greatly improves effi- ciency, requiring up to 1000x less compute than naive scoring. Nevertheless, classifying with dif- fusion model still typically takes 10s of scores per class average, making the diffusion classifier expensive to use for datasets with many classes
Algorithm Diffusion model classification with pruning.
given: Example classify æ, diffusion model w/ params 0, weighting function w. hyperparameters min_scores, max_scores, cutoff_pval
//Map from classes diffusion model scores. scores {y; : for E [yx]]
Empirical Analysis and Results
YI
= 0
while |ccores| > 1 and < max scores: n +
In this section, we detail our analysis for the dif- fusion zero-shot classifiers on variety of tasks. These include classification on various vision datasets to study generalization capabilities on diverse domains, evaluating model robustness conflicting cues between texture and shape, and studying attribute binding ability through targeted evaluation synthetic data.
//Noise the image
~~ [(0 1])
~ <=([x(()))
//Score against the remaining classes.
for y; E scores:
add w: wr we (z,, (by:), 1)
to scoresly,]
//Prune away implausible classes.
① = arg min,, scoresly;).menn
We mainly compare Imagen and SD with CLIP (Radford et al., 2021). We chose CLIP because it well-studied and widely-used model, as our primary aim is to study diffusion models rather than push state-of-the-arr zero-shot accuracies. Our experiments reveal strengths and weaknesses of image-text representations learned via genera- tive training vs. CL.IP's contrastive training.
if 2 min_scores:
for y: E scores:
if paired paired_ttest ttest ttest_ppal(
scores [?]. scores[y:]) < cutoff_ remove from scores.
return y
Model details: Imagen is a cascaded diffusion model (Ho et al., 2022) consisting of 64 64 low-resolution model and two super-resolution models. We only use the 64 64 model for our experiments because we found the high-resolution models performed poorly classifiers. Combin-
70
164R-100 Accunae
60
50
shared noise pruning
shared noise
unshared
ing the 64 64 model's 'ss scores with scores from the riglutinssolutiions models did not improve sults either (see Appendix E for details). The issue is that high-resolution models condition strongly on their low-resolution inputs and are therefore less sensitive the text prompt. Unlike with Fig- ure 3, high-resolution denoising with different text prompts produces images imperceptibly different to the human eye because they all agree with the same low resolution image.
10a
10'
10'
10*
Diffusion
Comparison of efficiency improve- ments for Imagen on CIFAR-100 100. Shared noise improves sample efficiency roughly 100x and pruning by an additional 8-10x.
Figure 2:
We use version 1.4 of Stable diffusion for our experiments. It uses a pre-trained text encoder
from CLIP to encode the text and pre-trained variational autoencoder to map images to latent space.
CLIP consists of vision and text transformers trained with contrastive learning. We use the largest CLIP model VIT-L/14@224ppp We provide more details on all the models in Appendix A.
5

---

pillow
analog clock putterfish
leopard high train
lorkeet vs brain coral
golden refriever vs. african elephant
gray whale vs pirate ship
cat vs bottle (cue conflict)
keyboard elephant conflict)
chair clock conflitt
Figure 3: Example predictions from Imagen when denoising the same image with different text prompts. Each set of images shows the original, noised, and denoised images for the two classes. The top two rows use ImageNet images and the bottom row uses Cue-Com Con-Conf.
Experiment details: For each experiment, we obtain scores using the heuristic timestep weighting function and the efficient scoring method in Algorithm 1. Due the method's still-substantial compute cost, we use reduced-size datasets (4096 examples) for our experiments. We preprocess each dataset by performing central crop and then resizing the images 64 64 resolution for Imagen, 512 x 512 for SD, and 224 x 224 for CLIP. We use min sioo__scres__ = 20, max_sce max_scores 2000, and cutoff __l___lll 2 x e These are simply reasonable choices that keep the method efficient to run; changing the values does effect the model behavior in expectation but trades off compute for reduction in variance. We use single prompt for each image. While we could have used an ensemble of prompts (i.e, made the expectation in Equation (2) also over different prompt templates), we chose not to for the sake of simplicity, as our goal is better understand models rather than achieve state-of-the-art zero-shot performance. Therefore, our reported results are slightly lower than in the CLIP paper, which uses prompt ensembling. We found in our experiments that diffusion models were quite robust to the choice of prompt. For example, we tried four different prompts from the CLIP templates for CIFAR- 100 and found accuracies to all be within 1.5% of each other.
Imagen, SD and CLIP have different model sizes, input resolutions. and are trained on different datasets for different amounts of time, so the comparison is not direct. While ideally we would train models of the same size the same data, this would be very expensive and challenging in practice; we instead use these strong existing pre-trained models. Our comparisons are geared towards highlighting the strengths and weaknesses of text-image diffusson models.
Comparing models:
4.1 Image Classification
We first evaluate the performance at zero-shot classification. For this purpose, we consider 13 datasets from Radford et al. (2021) as reported in Table 1, We use the prompt templates and class labels used by Radford al. (2021), which renames some classes that confuse models (e-g. "crane + "crane bird"" in Imagenet) (OpenAI, 2021b). We use the first prompt from the list, except for Imagenet, where we use "A bad photo labe! " since this is a good prompt for Imagen, SD and CLIP (OpenAI, 2021a).
Setup:
Since we use the low-resolution Imagen model, we obtain results using CLIP under two settings for more thorough comparison. In the first setting, we resize all the datasets to 64 x 64 which serves as the base low-resolution dataset. Imagen uses this dataset directly. For CLIP, we subsequently upsample the images, resizing them to 224 x 224 resolution. In the second setting, we directly resize all datasets to 224 x 224 resolution to obtain the best results possible using CLIIP where it can take advantage of its higher input resolution.
Results: Results are shown in Table 1. The first eight datasets (up through EuroSAT) on the top block of the table are all originally of resolution 64 x 64 or less. On these datasets, Imagen generally outperforms CLIP and Stable Diffusion on classification accuracy under the same evaluation setting i.e., the models are conditioned on the same text prompts, etc. Imagen significantly outperforms CLIP on SVHN and SD on digit recognition datasets like MNIST and SVHN, which requires recognizing text in an image. Saharia et al. (2022b) observe that Imagen is particularly good at generating text,

---

Model
CIFARIO
CIFARI00
STL10
MNIST
DTD
Camelyon
SVHN
EuroSAT
Imagen
96.6
84.3
99.6
79.2
37.3
60.3
62.7
60.3
Stable Diffusion
72.1
45.3
92.8
19.1
44.6
51.3
13.4
12.4
CLIP/VIT-L/14
94.7
68.6
99.6
74.3
36.0
58.0
21.5
58.0
Model
Stanford Cars
Imagenet
Caltech 101
Oxford Pets
Food101
Imagen
81.0
62.7
68,9
66.5
68.4
Stable Diffusion
77.8
61.9
73.0
72.5
71.6
CLIP/ViT-L/14
62.8/75.8
63.4/75.1
70.2/84.1
76.0/89.9
83,9/93.3
Table 1: Percent accuracies for zero-shot image classification. For CLIP where two numbers are reported, the accuracy correspond to two settings: downsizing the images to 64x64 and then resizing the images up to 224x224 (so CLIP does not have an advantage in input resolution over the 64x64 Imagen model) and resizing the images directly to 224x224 (so CLIP has the advantage of higher resolution). Variances accuracy are < 1% across different random seeds. The top set of results are on low-resolution datasets (which is why SD performs poorly).
while SD generally performs poorly (see Figure 6 in the appendix). This demonstrates that Imagen's s areas strength in generation carry over to downstream tasks and suggests that classification on OCR datasets could be used quantitative metric study model's cext-generation abilities. SD generally performs poorly on the low-resolution datasets, perhaps because it is only trained on high-resolution images." To better understand how much low-resolution is to blame, we evaluated SD on ImageNet after down-sampling the images to 32 x 32 and 64 x 64 resolution. SD's accuracy drops from 61.9% to 15.5% and 34.6% respectively. The next five datasets higher-resolution images. For some of these, taking advantage CLIP's higher input resolution substantially improves results. SD performs comparably to Imagen on all these datasets (although of course has an advantage in terms of input resolution).
Due to our reduced-size evaluation sets, variances in accuracy on zero-shot classification tasks across different random splits are roughly +0.4% for CLIP, +0.7% for Imagen, and #0.6% for Stable Diffusion. The diffusion models have higher variance due to the inherent randomness in noising images (while CLIP is deterministic). Overall, we are not interested small accuracy differences anyway, as the comparison between models is non-direct in various ways; instead we are trying go get a broad understanding of the models' abilities.
To our knowledge, these results are the first instance of generative model achieving classification accuracy competitive with strong transformer-based discriminative methods. Lastly, we note that our method relies on the model being highly sensitive text prompt, which we observe qualitatively in Figure 3.
Imagen
Stable Diffusion
CLIP
VIT-22B
ResNet50 (supervised)
84.4
72.5
51.6
68.7
79 (top-5)
Table 2: Percent shape accuracy for zero-shot classification on the Cue-Conflict Imagenet dataset.
4.2 Robustness to Shape-Texture Conflicting Cues
We next study diffusion models' robustness to presence of texture cues in images by evaluating their performance the Cue-Conflict dataset from Geirhos al. (2019). The dataset consists of Imagenet images altered to have a shape-texture conflict. While (for example) changing an image of a cat to have the texture of an elephant skin doesn't 't confuse humans, could cause a model to classify the image as an elephant. Geirhos et al. (2019) showed that CNNs trained on Imagenet were strongly biased towards recognising textures rather than shapes, which is in stark contrast human behavioural evidence.
'while low-resolution images were incorporated in CLIP's training, doing so with SD would run the risk of the model producing blurry images during generation
7

---

Control tasks test if the model can identify basic image features by scoring an attribute in the image against one not present. For example:
Shape: sphere. vs. A cylinder. Color: A gray object. A red object.,
Binding tasks test if the model binds a given attribute to the correct object. For example: Color|Shape: A yellow sphere, vs. A gray sphere.
Color Poootioo::o:: On the right is gray object vs. On the right is yellow object.
Pair binding tasks are easier binding tasks where information about both objects is provided. For example:
Shape,Size: A small sphere and large cube. vs. A large sphere and small cube. Color,Size: A small yellow object and a large gray object. vs. A large yellow object and a small gray object.
Figure 4: Examples of the synthetic-data attribute binding tasks. We explored more sophisticated prompts than in the figure (e.g.. "A blender rendering of two objects, one of which is a yellow sphere."), but they didn't substantially change results.
We test Imagen's and Stable Diffusion's 's robustness detecting shapes in presence of texture cues by using the same setting for classification as in Section 4.1. We report shape accuracy which is the percentage of images for which the model predicts the image's 's shape correctly in Table 2. We compare Imagen and SD with CLIP, the recently proposed ViT-22B model (Dehghani et al., 2023) which was trained on JFT (Sun et al., 2017) extended to 4B images (Zhai et al., 2022) and fine-tuned on Imagenet, and (not zero-shot) supervised ResNet50 model trained on the training set. Imagen and SD comphrehensively outperform all previous methods on this dataset. Imagen further achieves an accuracy of 84.4% outperforming SD, CLIP, and ViT-22B by more than 12%, 30% and 15% respectively, and the top-5 accuracy performance of the supervised ResNet50 model by 5%.
We believe that the denoising process of the diffusion model critical in removing the texture bias commonly observed in supervised models, making it robust to presence of textural cues. These findings are in line with Nie et al. (2022), who achieve state-of-the-ar adversarial robustness through denoising adversarial examples with diffusion model. We further qualitatively confirm this in Figure 3 in the appendix which depicts example images from this dataset and Imagen's 's result after denoising those images conditioned on text prompts with both the correct and incorrect shape class,
4.3 Evaluating Attribute Binding on Synthetic Data
We have shown that Imagen and SD perform comparably to CLIP at zero-shot classification, and much better than CLIP disregarding misleading textural cues. Do diffusion models have additional capabilities that are difficult to obtain through contrastive pre-training? We hypothesize that one such area may be in compositional generalization, and specifically compare the models attribute binding. Text-to-image generative models have shown emergent compositional generalization at large enough scale, being able to combine diverse concepts handle prompts such as "a chair shaped like an avacado" (Ramesh et al., 2021). Attribute binding is key piece of compositional reasoning, as it enables the understanding and integration of multiple concepts into a coherent whole. For example in the statement "a yellow sphere and a gray cube" we understand the sphere is yellow and the cube is gray, not the other way around. While previous work has examined attribute binding in text-to-image models by examining model generations (Nichol et al., 2021; Yu et al., 2022; Feng et al., 2023), our diffusion model classifier offers way of more precisely studying the question quantitatively. We hope in the future, this type of study enabled by our method will be useful for comparing other abilities of generative image models at fine- fine-grained level.
We use synthetic images similar Lewis al. (2022), where images are generated based on the CLEVR (Johnson et al., 2017) visual question answering dataset. CLEVR images contain various object (cubes, cylinders, and spheres) with various attributes (different sizes, colors, and materials). A modified version of the CLEVR rendering script is used to generates images containing two objects of different shapes. From these images, we construct binary classification tasks of 1000 examples each; see Figure 4 for more details and examples. We follow the same setup as in the classification evaluation, using the 64 x 64 Imagen model and 512 x 512 Stable Diffusion model with heuristic timestep weighting and largest public CLIP model (with full-resolution inputs).
Dataset Construction:
8

---

Tasks
Imagen
Stable Diffusson
CLIP
Shape (control task)
85
91
91 94
Color (control task)
96
85
Shape,Color Shape|Color Color[Shap
100 / 66 73
85 65 59
54 / 52 53
Shape,Size / Shape|Siie Sizze / Size| Size|Shape
99 48 51
63/48/55
52 51 50
Shape,Positions Chape/Positionn Position (Shape
74 51 5
49/50/50
50 /
Color,Size Color Size / Size|Color
86 54
59 /9/52/0 /48
48 / 51 48
Color,Position / Color Position Position|Color
72/49/ 49
53/ /
49 / 50 49
Sizitoosition/ Size Position / Position Siiee
/9/50/54
54/49/4
51 50 48
Table 3: Percent accuracy for models on zero-shot synthetic-data tasks investigating attribute binding. Bold results are significant (p < 0.01) according to a two-sided binomial test. CLIP is unable to bind attributes, while Imagen and SD sometimes can.
Scores for Imagen, SD, and CLIP these tasks is shown in Table 3. On the control tasks, all the models are able to identify shapes and colors that occur in the image with high accuracy. Imagen slightly worse at shape identification; we find most of these are due to mixing up "cylinder" and "cube" when the objects are small. Mistakes in color recognition generally occur when the distractor color is similar to the true color or to the color of the other object the image (e.g. the distractor color is blue and there is large cyan object in the image).
Results:
While CLIP can recognize image attributes, it performs better than random chance for the attribute binding tasks. This result shows it is unable to connect attributes to objects and is consistent with the prior study from Subramanian et al. (2022). In contrast, Imagen can perform (at least some extent) the pair binding tasks, and does better than chance on the Shape|Cole Coloor and Color|Shape tasks. SD cannot perform the positional tasks, but can perform shape/color binding.
Part of Imagen's 's advantage might be in its text encoder, the pre-trained T5 (Raffel et al., 2020) model. Saharia al. (2022b) find that instead using CLIP's text encoder for Imagen decreased its performance on generations involving specific colors or spatial positions. Similarly, Ramesh et al. (2022) find that DALLE-22 which uses CLIP text encoder, is worse at attribute binding than GLIDE, which uses representations from a jointly-raiindd transformer processing the text. An advantage of the diffusion models over CLIP is their use of cross attention to allow interaction between textual and visual features. A visual model without completely independent text and image encoders such as LXMERT (Tan & Bansal, 2019) or CMA-Clip (Liu al., 2021) might perform better, but of course these models come with the added compute cost of having to jointly process all image-text pairs with the model instead of embedding the text and images separately.
One mistake we observed frequently in Color Shape with Imagen is it preferring the color of the larger object in the image; e.g. scoring "A gray sphere" over "A yellow sphere" in Figure 4. We hypothesize that is helpful for denoising at high noise levels when the text conditioning provides the color for large region of the image, even when the color is associated with the wrong shape. In the pair task, the full color information for both objects is always provided, which avoids this issue, and perhaps explains why accuracies at pair tasks are much higher.
5 Conclusion and Future Work
We have proposed method that enables diffusion models to be used as zero-shot classifiers and developed ways of improving its efficiency to make it usable. Our experiments demonstrate strong results on image classification. Furthermore, we show Imagen and Stable Diffusion are remarkably robust to misleading textures, achieving state-of-the of-te----th-arr art results cue-conflict dataset. While existing analysis of diffusion models usually studies generated images qualitatively, our framework provides way of quantitatively evaluating text-to-image generative models through evaluating them on controlled classification tasks. We showcase this through our study on attribute binding, where we find that diffusion models are sometimes able to bind attributes while CLIP does not appear to have this ability. Similar experiments could used in the future to study other properties of pre-trained diffusion models, such as toxicity or bias.

---

Our paper is complementary to coneurrent work from Li et al. (2023), who use Stable Diffusion as zero-shot classifier and explore some different tasks like relational reasoning. While their approach is similar to ours, they perform different analysis, and their results are slightly worse than ours due to them using a simple hand-tuned class pruning method and no timestep weighting.
We hope our findings will inspire future work in using text-to-image diffusion models as foundation models for tasks other than generation. One direction is fine-tuning diffusion models on downstream tasks; given the strong zero-shot performance of Imagen and Stable Diffusion, a natural next step is evaluating them after further supervised training. As models become larger, another key question for further study is how do the scaling laws (Hestness et al., 2017; Kaplan et al., 2020) of contrastive vs generative pre-training compare. Additionally, we are interested in applying our analysis other generative models to study to what extent our results are a consequence of generative pre-training generally compared to diffusion pre-training specifically.
Ultimately, our method does not produce a practical classifier, as it requires substantial compute when scoring many classes. Instead, we see the main value of this work is in revealing more about the abilities of large pre-trained diffusion models and providing a method for enabling future fine-grained studies of diffusion model abilities. In total, our results suggest that generative pre-training may be useful alternative to contrastive pre-training for text-image self-supervised learning.
Acknowledgements
We thank Kevin Swersky, Mohammad Norouzi, and David Fleet for helpful discussions and feedback, Martha Lewis and Ellie Pavlick for answering our questions about their CLIP attribute binding experiments, and Robert Geirhos for answering our questions about Stylized Imagenet and ViT-22B. We also thank the anonymous reviewers for their comments and suggestions.
References
Emmanuel Asiedu Brempong. Simon Korbliih, Ting Chen, Niki Parmar, Matthias Minderer, and Mohammad Norouzi. Denoising Pretraining for Semantic Segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4175-4186, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Ryan Burgert, Kanchana Ranasinghe, Xiang Li, and Michael Ryoo. Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors. arXiv preprint arXiv:2211.1322 2022.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12 1983.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, al. Scaling vision transformers to 22 billion parameters. arXiv preprint 23020:2300.05442,, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 255. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021.
Eyal Even-Dar, Shie Mannor, and Y. Mansour. PAC Bounds for Multi-armed Bandit and Markov Decision Processes. In Annual Conference Computational Learning Theory, 2002.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training- Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. In International Conference on Learning Representations, 2023.
10

---

Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. International Conference on Learning Representations, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded Diffusion Models for High Fidelity Image Generation. J. Mach. Learn. Res., 23(47): 1-33, 2022.
Chao Jia, Yinfci Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2901-2910, 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 000301,0.8661.... 2020.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
Diederik P Kingma and Ruiqi Gao. Understanding the Diffusion Objective as Weighted Integral of ELBOs. arXiv preprint 3Xiv:2303.00844 2023.
Diederik Kingma and Max Welling, Auto-encoding Variational Bayes, International Conference on Learning Representations, 2014.
Martha Lewis, Qinan Yu, Jack Merullo, and Ellie Pavlick. Does CLIP Bind Concepts? Probing Compositionality in Large Image Models. arXiv preprint 22X 2022.
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your Diffusion Model is Secretly a Zero-Shot Classifier. arXiv preprint arXiv:2303. /6203, 2023.
Huidong Liu, Shaoyuan Xu, Jinmiao Fu, Yang Liu, Ning Xie, Chien-Chin Wang, Bryan Wang, and Yi Sun. CLIP- Cross moddalty attention clip for image-text classification. arXiv preprint 12.0v:21 2021.
Andrew Ng and Michael Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. Advances in neural information processing systems, 14, 2001.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In International Conference Machine Learning, 2021.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion Models for Adversarial Purification. International Conference on Machine Learning, 2022.
11

---

OpenAl. Prompt Engineering for Imagenet. Github, 2021a.
OpenAl. Prompts for Datasets. Github, 2021b.
Edward Paulson. A sequential procedure for selecting the population with the largest mean from normal populations. The Annals of Mathematical Statistics, pp. 174-180, 1964.
John Platt al. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 55551, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821-8831, PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2244.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. Advances in Neural Information Processing Systems, 2022b.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint 1 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pp. 2256-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 13:12238-12448, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 133456, 2020.
Sanjay Subramanian, Will Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension. In ACL, 2022.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843 852, 2017.
Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. arXiv preprint 090.::::9900044404 2019.
12

---

Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Antoin- Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206.00785 2022.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprini arXiv:2111.11432. 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12104 12112, 2022.
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing Text-to-lmage Diffusion Models for Visual Perception. arXiv preprint arXiv:2303.02113 2023,
13

---

A Model Details.
Imagen is text-conditioned diffusion model that comprises of frozen T5 (Raffel et al., 2020) language encoder that encodes an input prompt into sequence of embeddings, a 64 x 64 image diffusion model, and two cascaded super-resolution diffusion models that generate 256 256 and 1024 x 1024 images. Unlike Stable Diffusion, it operates directly on pixels instead of in a latent space. For our experiments. we use the 64 x 64 model which has 2B parameters and is trained using batch size of 2048 for 2.5M training steps on a combination of internal datasets, with around 460M image-text pairs, and the publicly available Laion dataset (Schuhmann et al., 2021), with 400M image-text pairs.
Imagen details:
Stable Diffusion details. We use Stable diffusion v1.4 which is latent text-to-image diffusion model. It uses the pre-trained text encoder from CLIP to encode text and a pre-trained variational autoencoder map images to a latent space. The model has 890M parameters and takes 5 512x512. 12- resolution images as input. It was trained on various subsets of Laion-5B, including a portion filtered only contain aesthetic images, for 1.2M steps using batch size of 2048.
CLIP details: CLIP encodes image features using a ViT-like transformer (Dosovitskiy et al., 2021) and uses causal language model to get the text features. After encoding the image and text features to a latent space with identical dimensions, evaluates a similarity score between these features. CLIP is pre-trained using contrastive learning. Here, we compare to the largest CLIP model (with a ViT-L/14 @ 224px as the image encoder). The model is smaller than Imagen (400M parameters), but is trained for longer (12.8B images processed 5.B). While Imagen trained primarily generative model, CLIP was primarily engineered to be transferred effectively to downstream tasks.
B Weighting Functions Details.
While for most experiments for wt, we also explored learning an effective weighting function (although this is not truly zero-shot). To do this, aggregate scores for each image and class yą. into 20 buckets, with each bucket covering a small slice of timestep values:
Learned Weighting Function:
heuristic weighting function
(x y*) E, wtt.tt/00....0..ooo. 0.05((000)) ze b(yx).t)
where we estimate the expectation with Monte Carlo sampling (typically around 100 samples). We then learn a 20-feature linear model with parameters [Vo..... U19] over these buckets:
expl Σ i== (2,
(D = ys(æ) = Ey, exp (E (x,,(,,);))
trained with standard maximum likelihood over the data. At test-time we use the weighting
w. [r/0.055
We generally found that (1) learned weighting functions are pretty similar across datasets, and (2) the weighting functions are transferable: the us learned on one dataset get good accuracy when evaluated on other ones. On average, learned weights produced around 1 1% higher accuracy on zero-shot classification tasks, but we omitted the results from the main paper because using learned weights is not truly zero-shot.
Comparison of Weighting Functions. We compare the learned weighting functions with several heuristic functions on the Caltech101 dataset. We chose Caltech101 because is high-resolution (SD performs poorly on low-resolution datasets), contains a diversity of image classes, was not used when we developed the heuristic weighting function, and only has 100 classes, it is much faster to evaluate models on than ImageNet. We compare the following functions:
VDM: w. SNR'(I). (1), the derivative of the signal to noise ratio with respect to l. This
weighting scheme from Variational Diffusion Models (Kingma et al., 2021) directly trains the model on lower bound of the likelihood.
Simple: w. = SNR((). This "simple" loss from Ho et al. (2020) results in model that produces better images according to human judgements and FID scores, even though results in worse likelihoods.
14

---

Weighting
Imagen
Stable Diffusion
VDM
62.0
71.9
Simple
56.1
71.4
Heuristic
68.9
73.0
Learned
70.2
73.1
Table 4: Percent accuracy for models on Caltech101 with different weighting schemes
Heuristic: w, = exp (-6t). Our hand-engineered weighting function; we found this by searching for simple weighting function that works well on CIFAR-100, although found empirically it generalizes very well to other datasets.
Learned: Learning an effective weighting function on a held-out set of examples as de- scribed above.
Results are shown in Table The heuristic weighting function outperforms Simple and VDM for both models. Interestingly, SD appears to be more robust to the choice of weighting function than Imagen. Mechanistically, the reason that "Simple" and "VDM" weighting put more weight on earlier timesteps than "Heuristic" and Imagen tends to be an inaccurate classifier very small noise levels. We intuitively believe this is a consequence pixel vs latent diffusion. The learned weighting only does slightly better than heuristic weighting despite not being truly zero-shot. We found similar results to hold on other datasets.
C Details on Attribute Binding Tasks and Prompts
We use the relational dataset from Lewis et (2022) for the attribute binding experi- ments. Each image consists of two objects of different shapes and colors; for tasks in- volving size we filter out examples where both objects are the same size. Each image contains two objects with different attributes shape {cube, sphere, cylinder), color e (blue, cyan, blue, brown, gray, green, purple, red, yellow). size {small, large), and position (left, right).
Given task (e.g. Shape Size), we construct a task spec-fpe description for an object as follows:
"On the /position/ a "if Position tasks else "A "+
"size" " if Size task else ""+
"color if Color task else ""+
"shape." if Shape task else "object."
For recognition and binding tasks, we randomly select one of the two objects in the image to be the positive example and then use its deseription the positive prompt. For pair tasks, we join the descriptions for both objects together with "and" (removing the period from the first description and lowercasing the second one) for the positive prompt.
To construct negative example for recognition tasks, we replace the positive attribute with a random attribute not in the image. For binding tasks, replace one of positive description's attributes with the other object's 's attribute (e.g., for Shape (Color, we replace shape).
For pair tasks, there is choice in how the two objects are ordered (e.g. "On the left is cube and on the right is a sphere" vs "On the right is a sphere and on the left is a cube". We follow the preference of stating the leftmost position//hhape/cllrssiii
first in that order. For example, this means we will always start with "On the left..." rather than "On the right...". Similarly, the negative example for Soler,Size in Figure 4 is "A large yellow object and small gray object" rather than "A small gray object and large yellow object" because we prefer to first put the leftmost color over the leftmost size.
We experimented with a variety of other prompts, but found none to work substantially better than these simple ones.
15

---

1.0
0.8
0.6
0.4
Average weightte soue (ECE-0. 12) Number model calls (ECE-0.04
Confidence
Figure 5: Model reliability diagram comparing confidence measures of Imagen CIFAR-100. The number of model calls used in Algorithm produces eitter-calibrate confidences than using the actual scores for different classes.
D Calibration
It is desirable for classifiers, especially when used in the zero-shot setting with possibly out-of-domain examples, to be well calibrated. In other words, if a classifier predicts label ÿ; with probability the label should be Ú; roughly 100 p% of the time. However, the diffusion model classifier does not directly produce probabilities for classes. While p(y: yı (x;)) should roughly be proportional to the expectation in Equation (2) when exponentiated (i.e. weighted scores to get probabilities), in practice our estimates of the expectations are very noisy and do provide well-calibrated scores.
apply softmax the average
We propose simple alternative that takes advantage of early pruning: we use the total number of diffusion model calls used for the image calibration measure. The intuition is that harder example will require more scores to determine the arg min class with good statistical significance.
More details on the two calibration methods are below:
Temperature-scaled raw scores. We Sys (x) to denote the weighted average squared error for class y& on image i.e., the Monte-Carlo estimate for the re-weighted VLB in equation 2. We turn these scores into an estimated probability by applying softmax with temperature:
pe yk (x) = exp(-sy,. (x)/T)
Eby) exp (-5y, (x)/T)
Note that this approach requires good score estimates for each class, it is not compatible with the class pruming method presented in Section 3.1.
Platt-scaled number of scores. Our other confidence method relies on the total number of scores needed to eliminate other classes as candidates. Let ja()) denote the predicted class for example æ and n((x) be the total number of calls to T used to obtain the prediction when running Algorithm 1. Then we estimate
(o (((=)) (z: sigmoid(-n(z))) (8)
We learn (and 3 for Platt scaling) small held-out set of examples.
We show reliability diagrams (DeGroot Fienberg. 1983) and report Expected Calibration Error (Guo et al., 2017) (ECE) for the methods in Figure 5. Using small held-out set of examples, we apply temperature scaling (Guo et al., 2017) for the score-based confidences and Platt scaling (Platt et al., 1999) for the number-of-score fairly well-calibrated, showing it possible to obtain reasonable confidences from diffusion model classifiers despite them not providing a probability distribution over classes.
confidences, (see Appendix D for details). Number of scores is
E Imagen's Super-resolution Models
Imagen is cascaded diffusion model (Ho et al., 2022) consisting of 64 64 low-resolution model and two super-resolution models, one that upsample the image from 64 64 to 256 x 256 and one
16

---

Prompt: A photo the number
Stable Diffusion
9988
010
10
8
888
8
Imagen
Figure 6: Example generated images from Stable Diffusion and Imagen for generating photos with text.
that upsamples from 256 x 256 to 1024 1024. However, found only the 64 x 64 model to work well as a zero-shot classifier. The super-resolution models condition on a low-resolution input image, which means they denoise effectively regardless of the input prompt and thus aren't 't sensitive the class label. For example, unlike with Figure 3, high-resolution denoising with different text prompts produces images imperceptibly different the human eye because they all agree with the same low resolution image. Imagen's super-resolution models are trained with varying amount of Gaussian noise added to the low-resolution input image (separate from the noise added to the high-resolution image being denoised). We were able to alleviate the above issue somewhat by using a large amount of such noise, but ultimately did not achieve very strong results with the high-resolution models. For example, the 64 x 64 to 256 x 256 model achieves an accuracy of 16. % on ImageNet.
We further experimented with combining the low-resolution model's scores with the 64 64 to 256 x 256 model's. To do this, we used the learned weighting scheme detailed in Appendix B, but with learning 40 weights: 20 for the low resolution model and 20 for the super-resolution model. However, we found the learned weighting scheme put almost no weight on the super-resolution model's scores, and did not perform significantly better than the low-resolution model did its own.
17


Reviews:
[{"review_id": "tRwSFw3S3W", "reviewer": "Reviewer_LkNq", "summary": "The paper inverts pre-trained text-to-image diffusion models by using bayes rule, and evaluates them over a variety of benchmarks. For the evaluation they use two diffusion models: Stable Diffusion and Imagen. They compare these models against CLIP-L/14. They show a variety of benchmarks where the diffusion model does better at classification than a standard SOTA discriminative model such as CLIP-L/14. ", "questions": "i) In Table 4 weighting seems to help significantly help Imagen however not so much for SD? why is that? what if u learned weighting for SD seperately and tried to generalize it to different datasets?\n\nii) Are any of the models trained on the same/similarish datasets (ViT22B vs Imagen)? \nIt would be good to clarify this in the paper.\n\niii) The paper raises resolution mismatch of the generative model (specifically Imagen), a big concern. Do the authors expect future generative models that are explicitly trained for classification to resolve this issue? If so how? as SD does train on higher resolution but doesn't get better performance than CLIP\n\n\n\n", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "i) The paper does a very dense evaluation of their proposed method \n\nii) They give good analysis on why/where generative classifiers would be useful over their discriminative counterpart.\n\niii) The paper proposes weighted timesteps and sampling methods that improve the accuracy and speed of the classifier.", "weaknesses": "i) the analysis become a bit weak , as none of the models considered are trained on the same dataset. \n\nii) Ablations are in the supplementary and results are ablated only on one dataset, would be good to have atleast 2-3 diverse datasets (small/high resolution)\n\nI have few qs that i have listed below.\n\nminor - \nLine 271 shouldn't it be zero-shot instead of few-shot? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful comments and suggestions. We address your questions and concerns below:\n\n\n- **Different training datasets**: As we mentioned in the general response above, in this work, we focused on studying the capabilities of existing powerful models because training such models from scratch would require huge compute resources. However, many of the differences between models are striking enough that we are confident they point to fundamental differences in what is learned from their training objectives rather than differences in their training data. For example, diffusion models greatly outperform CLIP on robustness to shape-texture conflicts and attribute binding, and pixel-based diffusion substantially outperforms latent-space based diffusion models on OCR data like MNIST and SVHN (see Lines 231-236).\n\n\n- **Ablations**: As we state in line 536 of the supplementary materials, we did find similar results to hold across many datasets. We used our results on Caltech 101 as a representative example to support our findings, but we are happy to include numbers on other datasets in the revised version of the paper. \n\n\n- **Line 271**: Good catch – yes, that should say zero-shot. \n\n\n- **Q1 Learned weighting**: We did try learning the weighting for SD separately on Caltech101 (last row of Table 4 in Appendix B). We did not try applying the learned weights on other datasets because it barely improved over the heuristic weighting while being more complicated. We also found it interesting that SD is more robust to the choice of weighting scheme. Mechanistically, the reason is essentially that (1) “Simple” and “VDM” weighting put more weight on earlier timesteps than “Heuristic” and (2) Imagen tends to be an inaccurate classifier at very small noise levels. We intuitively believe this is a consequence of pixel vs latent diffusion. \n\n\n- **Q2 Models trained on similar datasets**: Imagen and Stable Diffusion are perhaps most similar in that both are trained on LAION (although Imagen uses some additional data). ViT22B is trained on JFT, which is less similar because it is a (semi-automatically labeled) classification dataset. We discuss training data for ViT on line 259 and for Imagen, SD, and CLIP in section A of the supplementary materials. \n\n\n- **Q3 Resolution mismatch**: We expect non-cascaded diffusion models such as latent diffusion models or simple diffusion (https://arxiv.org/abs/2301.11093) to avoid the resolution mismatches. We also think explicitly fine-tuning generative models for classification would improve results and would be an interesting future direction of research. While CLIP outperforms SD on most classification tasks, SD is better at attribute binding and the cue-conflict dataset, suggesting that the different pre-training methods may have different areas of strength rather than one being strictly better than the other. Intuitively, it makes sense that CLIP is better at standard image classification, as it was designed with transfer to classification tasks in mind while SD and Imagen were designed as image generators. "}, {"review_id": "mFxHmci8EW", "reviewer": "Reviewer_yeML", "summary": "This study explores the potential of text-to-image diffusion models as zero-shot classifiers. The models show competitive performance with CLIP on zero-shot image classification datasets and excel in shape/texture bias tests and attribute binding. The findings suggest that generative pre-training should be considered as a compelling alternative for vision-language tasks.", "questions": "See weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors thoroughly investigate the zero-shot classification capabilities of text-to-image diffusion models through extensive experiments, covering both standard and challenging benchmarks.\n\n2. Additionally, the authors compare the classification performance of Stable Diffusion (SD) and Imagen, revealing intriguing results that suggest diffusion models trained on original images exhibit superior generalization compared to those trained on the latent space of VAE.\n\n3. The authors offer a range of practical and effective techniques to accelerate the classification process, providing valuable insights for improved efficiency.", "weaknesses": "1. While the authors present various strategies to speed up the classification process, it remains significantly slower compared to traditional classification models.\n\n2. It would be beneficial to include a comparison with other multimodal models like BLIP.\n\n3. There appears to be a disconnect between lines 192 and 193, as the introduction of Imagen is only mentioned in that paragraph.\n\n4. Considering that CLIP is trained with 400M text-image pairs and SD is trained on Laion-5B, a direct comparison may not be entirely fair. It would be interesting to see the performance gap between SD and CLIP when both models are trained on the same amount of data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful comments and suggestions. We address your questions and concerns below:\n\n- **Runtime**: As we say in the paper, the method does not produce a very practical classifier. However, we believe it still has a lot of value for illuminating what kinds of visual knowledge diffusion models learn, comparing diffusion model abilities on fine-grained tasks (such as attribute binding), and more generally comparing generative vs. contrastive pre-training.\n\n- **Comparison with other multimodal models**: We focused on CLIP as it is the most widely used zero-shot classifier and most subsequent methods (including BLIP) use a similar contrastive training method. As far as we know, BLIP is not often used as a zero-shot image classifier (although of course it is used for other tasks like captioning). \n\n- **Lines 192,193**: Sorry for the confusion: that sentence was meant to indicate which version of Stable Diffusion we are using, not suggest we are only using Stable Diffusion. We will rephrase it to avoid confusion.\n\n- **Different training datasets**: We completely agree, and hope in the future the community will release more comparable models. Please see the general response above for more discussion and our explanation on model comparisons. We also note that the dataset comparison between Stable Diffusion and CLIP isn’t entirely clear because (1) CLIP was trained for 32 epochs, while SD was trained for less than 1, and (2) SD was largely trained on an aesthetic subset of LAION-2B, which is substantially smaller than the full LAION-5B dataset.\n"}, {"review_id": "w7lVwaMmm2", "reviewer": "Reviewer_13fF", "summary": "The paper presents a new method that utilizes generative models as image classifiers and initiates explorations of notable, open-source models using this approach. Beginning with recognized image datasets, the researchers examine the models and evaluate the scores they have achieved. Several experiments are conducted to profile the behavior of these models, such as determining their optimal operating resolutions and dataset types. The paper assesses these models' capabilities by evaluating dataset competencies, such as pointing out the prominence of the Imagen model with the MNIST dataset due to its strong text generation skills.\n\nFurther characterizations are made on the models' ability to handle shape-texture conflicting cues using the Cue-Conflict dataset. The paper suggests that generative models, utilized through their proposed method for recognition tasks, exhibit more robustness compared to traditional ConvNets.\n\nLastly, the paper explores how these models perform in attribute binding tasks. The researchers report that while the CLIP model's performance is near random, both the Stable Diffusion (SD) and Imagen models show promising potential in these tasks.", "questions": "1. L271: few-shot classification?\n2. Why not use more commonly used zero-shot (or few-shot) learning datasets such as CUB, AwA or miniImageNet?\n3. Why not compare against strong (even if not sota) ZSL baselines with various benchmarks?\n4. How do you tune the hyper-parameters? Setting any HP wrt. test performance is a violation of ZSL protocol. Can you describe a reproducible HP tuning procedure?\n5. Regarding the compositional generalization experiments: why not use existing compositional zsl benchmarks and compare against existing methods?\n", "limitations": "Most limitations are listed above. No additional (especially societal) limitation to be reported here.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper shine through in several ways. Firstly, the paper explains the research and results in a way that's easy to understand. This clear writing helps to make complex ideas accessible to a wider audience.\n\nThe paper's thorough exploration of generative models also sets this work apart. They dive deep into understanding how these models behave. This includes looking at how well the models can handle images where shapes and textures don't match, as well as how they can connect attributes together. These insights are not just interesting, but they also add to our understanding of how these models work.\n\nAnother key strength of this paper is the smart improvements the paper makes. For instance, they use a technique called timestep weighting to improve how the models classify images. This technique is a smart way to reduce the impact of noise, making the models more reliable as they handle larger timesteps if they are sampled. Such improvements amplify the significance of the paper.", "weaknesses": "# Pre-rebuttal\nWhile the paper continually emphasizes the intention to introduce a method for using generative models as classifiers, there's a significant overlap with the methodology presented in \"Your Diffusion Model is Secretly a Zero-Shot Classifier\", which also uses the Stable Diffusion generative model as a classifier. Despite the similar scoring mechanisms, they attribute their improved results to the application of timestep weighting and a more efficient class pruning method. This does not, however, fully offset the lack of originality due to the similarity of their method with the cited work. Furthermore, by choosing not to pursue enhancements such as prompt engineering, the authors have seemingly missed an opportunity to establish a stronger benchmark for future studies. This decision, coupled with the questions regarding the paper's originality, could limit the overall significance and potential impact of their work.\n\nCertain parameters used in the paper, such as the cutoff_pval, are not adequately examined for their effects, making them appear as arbitrary choices or 'magic numbers'. Similarly, the weighting function used remains largely unexplored and unexplained, leaving a gap in understanding its impact on the results.\n\nThe paper provides comparisons for their efficiency improvements, including shared noise and class pruning methods. However, the paper falls short in determining the peak accuracy reach for the vanilla method, leaving an element of uncertainty and a lack of thorough comparison between the methods employed.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful comments and suggestions. We address your questions and concerns below:\n\n- **Overlap with another paper**: As we state in the conclusion, “Your Diffusion Model is Secretly a Zero-Shot Classifier” is concurrent work. It was released on arxiv (but not in a peer-reviewed venue) around a month before NeurIPS submission deadline. In fact, our method and results were submitted and presented in workshops from the end of January 2023 and were thus conducted before the other work (to preserve the sanctity of double blind reviewing, we can not provide exact details on this). It is our understanding that the existence of a recent non-peer-reviewed pre-print that is similar to the work under submission should not affect novelty/originality judgements.\n\n- **Not pursuing enhancements such as prompt engineering**: We actually extensively explored different prompts, but found none to significantly outperform simple prompts like “a photo of a __.” (e.g. see lines 207-209). \n\n- **Weighting function unexplored**: We did explore the weighting function in detail, and included those results and discussion in the supplementary section due to space constraints (see Appendix B and its reference in the main text in Line 127). \n\n- **Hyperparameter selection**: The choice of *min_scores, max_scores*, and *cutoff_pval* are simply reasonable choices that keep the method efficient to run, not crucial “magic numbers.” Different choices of these parameter values will make the method use a different amount of compute (with less compute leading to noisier estimates of the best-scoring classes). However, different choices of these values do not change the actual behavior of the model in expectation because the method is always estimating the same argmin (eq 2). Furthermore, we selected these values and weighting function using CIFAR-100 (see section 3, Lines 116-127) but not on any of the many other datasets we experimented on. We have been careful to preserve true ZS classification protocol in our experiments. \n\n- **Peak accuracy for vanilla method**: The “efficient” and “vanilla” methods are different ways of estimating exactly the same argmin (eq 2 in the paper), so they have the same peak accuracy given enough compute. However, the methods do have different efficiencies, which is shown in Figure 2.\n\n- **Q1 L271**: Good catch! That should be “zero-shot” classification. \n\n- **Q2 Other datasets**: CUB, AwA, and miniImageNet are mostly used for few-shot classification, which we don’t explore in our paper. Instead, we use datasets from the CLIP paper, which have become pretty standard for evaluating zero-shot classification (e.g. used to evaluate ALIGN, BASIC, and LiT)\n\n- **Q3 Strong baselines**: We think CLIP is a pretty strong baseline for zero-shot classification. While there exist stronger models, many are either not open source (e.g. CoCa) or are pre-trained on classification-like data (e.g. LiT). Additionally, most subsequent zeros-shot methods are based on CLIP. We are not sure which other baseline the reviewer had in mind. \n\n- **Q4 Hyperparameters**: See our “hyperparameter selection” bullet point above. Briefly, we selected the hyperparameter values and weighting function using CIFAR-100 (see section 3, Lines 116-127), but not on any of the many other datasets we experiment on. \n\n- **Q5 Why not use an existing benchmark**: We are using an existing benchmark from Lewis et al. (https://arxiv.org/abs/2212.10537). \n\n"}, {"review_id": "voTvyydSiP", "reviewer": "Reviewer_edqQ", "summary": "This paper uses the text-to-image diffusion models as zero-shot classifiers. It proposes to compute a subset of the full scores matrix to be more efficient. It proves Imagen and Stable Diffusion have good zero-shot performance and are robust to misleading textural cues.", "questions": "1. Is it possible to extent this work to various downstream tasks such as detection or segmentation?\n2. In large datasets like ImageNet, there are many kinds of sub-classes such as many types of foxes. In this case, is it reasonable to use the diffusion model since they are very similar in noisy images?", "limitations": "Please refer to \"Questions\".", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is novel to use pre-trained diffusion models as zero-shot classifiers.\n2. Building a scores matrix with each label prompt is reasonable for me.\n3. The efficiency is very important in this setting. The author well discussed this point in section 3.1.\n4. It is interesting to see the generalization ability of large pretrained model leads to robustness to shape-texture conflicting cues.\n5. Empirical results are sufficient to support the conclusion.", "weaknesses": "1. It lacks some theoretical discussion about why diffusion models can perform well in classification.\n2. Classification is a simple task. I am not sure whether it is too easy for a large model. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful comments and suggestions. We address your questions and concerns below:\n\n- **Theoretical discussion**: Generally, the idea of using a generative model as a classifier is a fairly old and well-studied idea (e.g. “On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes” NeurIPS 2001). Our main contribution is in showing how to efficiently use a diffusion model this way and presenting empirical results comparing them with contrastive zero-shot models. \n\n- **Classification is a simple task**: We think image classification tasks certainly can be very challenging; for example CLIP gets around 50% on cue-conflict imagenet and does no better than chance in our attribute binding experiments. \n\n- **Q1 Extension to other downstream tasks**: There have been several recent works that have used pre-trained text-image diffusion models (either as zero-shot or through their representations) for segmentation and detection e.g. https://arxiv.org/abs/2303.04803, https://arxiv.org/abs/2211.13224, https://arxiv.org/abs/2303.02153 etc. However, the method proposed in the paper would not work directly for many downstream tasks because it scores images based on the text prompt rather than predicting some structured output.  \n\n- **Q2 Fine-grained image classification**: We agree with the intuition that distinguishing sub-classes would be hard when there is lots of noise. However, our method uses many noise levels and we expect the scores at low noise-levels will be more discriminative for distinguishing similar classes. Indeed, our heuristic weighting function puts more weight on lower noise levels. "}]

Author rebuttal:
### **General response to all reviewers**

We thank all the reviewers for their helpful comments and suggestions. 

Generally, reviewers (*LkNq* and *yeML*) had questions around the fairness of comparison between models trained on different datasets (such as CLIP and Stable Diffusion). In this work, we focused on studying the capabilities of existing powerful models because training such models from scratch would require huge compute resources. However, many of the differences between models are striking enough that we are confident they point to fundamental differences in what is learned from their training objectives rather than differences in their training data. For example, diffusion models greatly outperform CLIP on robustness to shape-texture conflicts and attribute binding, and pixel-based diffusion substantially outperforms latent-space based diffusion models on OCR data like MNIST and SVHN (see Lines 231-236).

Secondly, reviewers (*LkNq* and *13fF*) had questions regarding ablation studies for the weighting function. We studied the effects of different weighting functions in detail and have included this discussion in Appendix B.

Reviewers (*yeML* and *13fF*) also had questions regarding comparisons to other baselines. In our work, we focused on CLIP as it is the most widely used zero-shot classifier and most subsequent methods incorporate similar contrastive training methods. While stronger models than CLIP do exist, many are either not open source (e.g. CoCa) or are pre-trained on classification-like data (e.g. LiT). For all our experiments, we have compared to existing benchmarks like datasets used in the CLIP paper for image classification, benchmark proposed by Geirhos et.al 2021 for robustness to shape-texture bias and the benchmark proposed by Lewis et.al 2023 for attribute binding.  

We address the further concerns and questions of each reviewer in detail below. 


Meta-review: The paper compares generative diffusion models (SD and Imagen) and discriminative image-text models such as CLIP on the task of zero-shot classification. After the rebuttal all reviewers recommend acceptance. As large-scale diffusion models are relatively new, their properties are not yet fully understood. This paper provides thorough analysis and insights and will be very valuable, especially for many downstream applications of these models.
Decision: Accept (spotlight)