SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Results: Results are shown in Table 1. The first eight datasets (up through EuroSAT) on the top block of the table are all originally of resolution  $64 \times 64$  or less. On these datasets, Imagen generally outperforms CLIP and Stable Diffusion on classification accuracy under the same evaluation setting i.e., the models are conditioned on the same text prompts, etc. Imagen significantly outperforms CLIP on SVHN and SD on digit recognition datasets like MNIST and SVHN, which requires recognizing text in an image. Sahara et al. (2022b) observe that Imagen is particularly good at generating text,
Model	CIFAR10	CIFAR100	STL10	MNIST	DTD	Camelyon	SVHN	EuroSAT
Imagen	96.6	84.3	99.6	79.2	37.3	60.3	62.7	60.3
Stable Diffusion	72.1	45.3	92.8	19.1	44.6	51.3	13.4	12.4
CLIP/ViT-L/14	94.7	68.6	99.6	74.3	36.0	58.0	21.5	58.0
Model	Stanford Cars	Imagenet	Caltech 101	Oxford Pets	Food101
Imagen	81.0	62.7	68.9	66.5	68.4
Stable Diffusion	77.8	61.9	73.0	72.5	71.6
CLIP/ViT-L/14	62.8/75.8	63.4/75.1	70.2/84.1	76.0/89.9	83.9/93.3
while SD generally performs poorly (see Figure 6 in the appendix). This demonstrates thatImagen's areas of strength in generation carry over to downstream tasks and suggests that classification on OCR datasets could be used as a quantitative metric to study a model's text-generation abilities. SD generally performs poorly on the low-resolution datasets, perhaps because it is only trained on high-resolution images. To better understand how much low-resolution is to blame, we evaluated SD on ImageNet after down-sampling the images to  $32 \times 32$  and  $64 \times 64$  resolution. SD's accuracy drops from  $61.9\%$  to  $15.5\%$  and  $34.6\%$  respectively. The next five datasets use higher-resolution images. For some of these, taking advantage of CLIP's higher input resolution substantially improves results. SD performs comparably toImagen on all these datasets (although of course it has an advantage in terms of input resolution).
Due to our reduced-size evaluation sets, variances in accuracy on zero-shot classification tasks across different random splits are roughly  $\pm 0.4\%$  for CLIP,  $\pm 0.7\%$  forImagen, and  $\pm 0.6\%$  for Stable Diffusion. The diffusion models have higher variance due to the inherent randomness in noisy images (while CLIP is deterministic). Overall, we are not interested in small accuracy differences anyway, as the comparison between models is non-direct in various ways; instead we are trying go get a broad understanding of the models' abilities.
To our knowledge, these results are the first instance of a generative model achieving classification accuracy competitive with strong transformer-based discriminative methods. Lastly, we note that our method relies on the model being highly sensitive to text prompt, which we observe qualitatively in Figure 3.
Table 1: Percent accuracies for zero-shot image classification. For CLIP where two numbers are reported, the accuracy correspond to two settings: downsizing the images to  $64 \times 64$  and then resizing the images up to  $224 \times 224$  (so CLIP does not have an advantage in input resolution over the  $64 \times 64$  Imagen model) and resizing the images directly to  $224 \times 224$  (so CLIP has the advantage of higher resolution). Variances in accuracy are  $< 1\%$  across different random seeds. The top set of results are on low-resolution datasets (which is why SD performs poorly).
Imagen	Stable Diffusion	CLIP	ViT-22B	ResNet50 (supervised)
84.4	72.5	51.6	68.7	79 (top-5)
Table 2: Percent shape accuracy for zero-shot classification on the Cue-Conflict Imagenet dataset.
4.2 Robustness to Shape-Texture Conflicting Cues
We next study diffusion models' robustness to presence of texture cues in images by evaluating their performance on the Cue-Conflict dataset from Geirhos et al. (2019). The dataset consists of Imagenet images altered to have a shape-texture conflict. While (for example) changing an image of a cat to have the texture of an elephant skin doesn't confuse humans, it could cause a model to classify the image as an elephant. Geirhos et al. (2019) showed that CNNs trained on Imagenet were strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence.
while low-resolution images were incorporated in CLIP's training, doing so with SD would run the risk of the model producing blurry images during generation
Figure 4: Examples of the synthetic-data attribute binding tasks. We explored more sophisticated prompts than in the figure (e.g., "A blender rendering of two objects, one of which is a yellow sphere.", but they didn't substantially change results.
Control tasks test if the model can identify basic image features by scoring an attribute in the image against one not present. For example:
Shape: A sphere. vs. A cylinder. Color: A gray object. vs. A red object.
Binding tasks test if the model binds a given attribute to the correct object. For example: Color|Shape: A yellow sphere. vs. A gray sphere.
Color|Position: On the right is a gray object vs. On the right is a yellow object.
Pair binding tasks are easier binding tasks where information about both objects is provided. For example:
Shape, Size: A small sphere and a large cube. vs. A large sphere and a small cube.
Color, Size: A small yellow object and a large gray object. vs. A large yellow object and a small gray object.
We test Imagen's and Stable Diffusion's robustness to detecting shapes in presence of texture cues by using the same setting for classification as in Section 4.1. We report shape accuracy which is the percentage of images for which the model predicts the image's shape correctly in Table 2. We compare Imagen and SD with CLIP, the recently proposed ViT-22B model (Dehghani et al., 2023) which was trained on JFT (Sun et al., 2017) extended to 4B images (Zhai et al., 2022) and fine-tuned onImagenet, and a (not zero-shot) supervised ResNet50 model trained on the training set. Imagen and SD comprehensively outperform all previous methods on this dataset. Imagen further achieves an accuracy of  $84.4\%$  outperforming SD, CLIP, and ViT-22B by more than  $12\%$ ,  $30\%$  and  $15\%$  respectively, and the top-5 accuracy performance of the supervised ResNet50 model by  $5\%$ .
We believe that the denoising process of the diffusion model is critical in removing the texture bias commonly observed in supervised models, making it robust to presence of textural cues. These findings are in line with Nie et al. (2022), who achieve state-of-the-art adversarial robustness through denoising adversarial examples with a diffusion model. We further qualitatively confirm this in Figure 3 in the appendix which depicts example images from this dataset and Imagen's result after denoising those images conditioned on text prompts with both the correct and incorrect shape class.
4.3 Evaluating Attribute Binding on Synthetic Data
We have shown that Imagen and SD perform comparably to CLIP at zero-shot classification, and much better than CLIP at disregarding misleading textural cues. Do diffusion models have additional capabilities that are difficult to obtain through contrastive pre-training? We hypothesize that one such area may be in compositional generalization, and specifically compare the models at attribute binding. Text-to-image generative models have shown emergent compositional generalization at large enough scale, being able to combine diverse concepts to handle prompts such as "a chair shaped like an avacado" (Ramesh et al., 2021). Attribute binding is a key piece of compositional reasoning, as it enables the understanding and integration of multiple concepts into a coherent whole. For example in the statement "a yellow sphere and a gray cube" we understand the sphere is yellow and the cube is gray, not the other way around. While previous work has examined attribute binding in text-to-image models by examining model generations (Nichol et al., 2021; Yu et al., 2022; Feng et al., 2023), our diffusion model classifier offers a way of more precisely studying the question quantitatively. We hope in the future, this type of study enabled by our method will be useful for comparing other abilities of generative image models at a fine-grained level.
Dataset Construction: We use synthetic images similar to Lewis et al. (2022), where images are generated based on the CLEVR (Johnson et al., 2017) visual question answering dataset. CLEVR images contain various object (cubes, cylinders, and spheres) with various attributes (different sizes, colors, and materials). A modified version of the CLEVR rendering script is used to generate images containing two objects of different shapes. From these images, we construct binary classification tasks of 1000 examples each; see Figure 4 for more details and examples. We follow the same setup as in the classification evaluation, using the  $64 \times 64$  Imagen model and  $512 \times 512$  Stable Diffusion model with heuristic timestep weighting and largest public CLIP model (with full-resolution inputs).
Tasks	Imagen	Stable Diffusion	CLIP
Shape (control task)	85	91	91
Color (control task)	96	85	94
Shape,Color / Shape|Color / Color|Shape	100 / 66 / 73	85 / 65 / 59	54 / 52 / 53
Shape,Size / Shape|Size / Size|Shape	99 / 48 / 51	63 / 48 / 52	52 / 51 / 50
Shape,Position / Shape|Position / Position|Shape	74 / 51 / 52	49 / 50 / 50	50 / 48 / 51
Color,Size / Color|Size / Size|Color	86 / 54 / 54	59 / 52 / 48	48 / 51 / 48
Color,Position / Color|Position / Position|Color	72 / 49 / 49	53 / 51 / 49	49 / 50 / 49
Size,Position / Size|Position / Position|Size	69 / 50 / 54	54 / 49 / 49	51 / 50 / 48
Table 3: Percent accuracy for models on zero-shot synthetic-data tasks investigating attribute binding. Bold results are significant ( $p < 0.01$ ) according to a two-sided binomial test. CLIP is unable to bind attributes, while Imagen and SD sometimes can.
Results: Scores for Imagen, SD, and CLIP at these tasks is shown in Table 3. On the control tasks, all the models are able to identify shapes and colors that occur in the image with high accuracy. Imagen is slightly worse at shape identification; we find most of these are due to it mixing up "cylinder" and "cube" when the objects are small. Mistakes in color recognition generally occur when the distractor color is similar to the true color or to the color of the other object in the image (e.g. the distractor color is blue and there is a large cyan object in the image).
While CLIP can recognize image attributes, it performs no better than random chance for the attribute binding tasks. This result shows it is unable to connect attributes to objects and is consistent with the prior study from Subramanian et al. (2022). In contrast, Imagen can perform (at least to some extent) the pair binding tasks, and does better than chance on the Shape|Color and Color|Shape tasks. SD cannot perform the positional tasks, but can perform shape/color binding.
Part of Imagen's advantage might be in its text encoder, the pre-trained T5 (Raffel et al., 2020) model. Sahara et al. (2022b) find that instead using CLIP's text encoder for Imagen decreased its performance on generations involving specific colors or spatial positions. Similarly, Ramesh et al. (2022) find that DALLE-2, which uses a CLIP text encoder, is worse at attribute binding than GLIDE, which uses representations from a jointly-trained transformer processing the text. An advantage of the diffusion models over CLIP is their use of cross attention to allow interaction between textual and visual features. A visual model without completely independent text and image encoders such as LXMERT (Tan & Bansal, 2019) or CMA-Clip (Liu et al., 2021) might perform better, but of course these models come with the added compute cost of having to jointly process all image-text pairs with the model instead of embedding the text and images separately.
One mistake we observed frequently in Color|Shape with Imagen is it preferring the color of the larger object in the image; e.g. scoring "A gray sphere" over "A yellow sphere" in Figure 4. We hypothesize that it is helpful for denoising at high noise levels when the text conditioning provides the color for a large region of the image, even when the color is associated with the wrong shape. In the pair task, the full color information for both objects is always provided, which avoids this issue, and perhaps explains why accuracies at pair tasks are much higher.
We have proposed a method that enables diffusion models to be used as zero-shot classifiers and developed ways of improving its efficiency to make it usable. Our experiments demonstrate strong results on image classification. Furthermore, we showImagen and Stable Diffusion are remarkably robust to misleading textures, achieving state-of-the-art results on cue-conflict dataset. While existing analysis of diffusion models usually studies generated images qualitatively, our framework provides a way of quantitatively evaluating text-to-image generative models through evaluating them on controlled classification tasks. We showcase this through our study on attribute binding, where we find that diffusion models are sometimes able to bind attributes while CLIP does not appear to have this ability. Similar experiments could be used in the future to study other properties of pre-trained diffusion models, such as toxicity or bias.
Our paper is complementary to concurrent work from Li et al. (2023), who use Stable Diffusion as a zero-shot classifier and explore some different tasks like relational reasoning. While their approach is similar to ours, they perform different analysis, and their results are slightly worse than ours due to them using a simple hand-tuned class pruning method and no timestep weighting.
We hope our findings will inspire future work in using text-to-image diffusion models as foundation models for tasks other than generation. One direction is fine-tuning diffusion models on downstream tasks; given the strong zero-shot performance ofImagen and Stable Diffusion, a natural next step is evaluating them after further supervised training. As models become larger, another key question for further study is how do the scaling laws (Hestness et al., 2017; Kaplan et al., 2020) of contrastive vs generative pre-training compare. Additionally, we are interested in applying our analysis to other generative models to study to what extent our results are a consequence of generative pre-training generally compared to diffusion pre-training specifically.
Ultimately, our method does not produce a practical classifier, as it requires substantial compute when scoring many classes. Instead, we see the main value of this work is in revealing more about the abilities of large pre-trained diffusion models and providing a method for enabling future fine-grained studies of diffusion model abilities. In total, our results suggest that generative pre-training may be a useful alternative to contrastive pre-training for text-image self-supervised learning.
We thank Kevin Swersky, Mohammad Norouzi, and David Fleet for helpful discussions and feedback, Martha Lewis and Ellie Pavlick for answering our questions about their CLIP attribute binding experiments, and Robert Geirhos for answering our questions about Stylized Imagenet and ViT-22B. We also thank the anonymous reviewers for their comments and suggestions.
Emmanuel Asiedu Brempong, Simon Kornblith, Ting Chen, Niki Parmar, Matthias Minderer, and Mohammad Norouzi. Denoising Pretraining for Semantic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4175-4186, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Ryan Burgert, Kanchana Ranasinghe, Xiang Li, and Michael S Ryoo. Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors. arXiv preprint arXiv:2211.13224, 2022.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12-22, 1983.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. IEEE, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021.
Eyal Even-Dar, Shie Mannor, and Y. Mansour. PAC Bounds for Multi-armed Bandit and Markov Decision Processes. In Annual Conference Computational Learning Theory, 2002.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, P. Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. In International Conference on Learning Representations, 2023.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and León Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2022.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12104-12113, 2022.
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing Text-to-Image Diffusion Models for Visual Perception. arXiv preprint arXiv:2303.02153, 2023.
A Model Details.
Imagen details: Imagen is a text-conditioned diffusion model that comprises of a frozen T5 (Raffel et al., 2020) language encoder that encodes an input prompt into a sequence of embeddings, a  $64 \times 64$  image diffusion model, and two cascaded super-resolution diffusion models that generate  $256 \times 256$  and  $1024 \times 1024$  images. Unlike Stable Diffusion, it operates directly on pixels instead of in a latent space. For our experiments, we use the  $64 \times 64$  model which has 2B parameters and is trained using a batch size of 2048 for 2.5M training steps on a combination of internal datasets, with around 460M image-text pairs, and the publicly available Laion dataset (Schuhmann et al., 2021), with 400M image-text pairs.
Stable Diffusion details. We use Stable diffusion v1.4 which is a latent text-to-image diffusion model. It uses the pre-trained text encoder from CLIP to encode text and a pre-trained variational autoencoder to map images to a latent space. The model has 890M parameters and takes 512x512-resolution images as input. It was trained on various subsets of Laion-5B, including a portion filtered to only contain aesthetic images, for 1.2M steps using batch size of 2048.
CLIP details: CLIP encodes image features using a ViT-like transformer (Dosovitskiy et al., 2021) and uses a causal language model to get the text features. After encoding the image and text features to a latent space with identical dimensions, it evaluates a similarity score between these features. CLIP is pre-trained using contrastive learning. Here, we compare to the largest CLIP model (with a ViT-L/14@224px as the image encoder). The model is smaller than Imagen (400M parameters), but is trained for longer (12.8B images processed vs 5.B). While Imagen was trained primarily as a generative model, CLIP was primarily engineered to be transferred effectively to downstream tasks.
B Weighting Functions Details.
Learned Weighting Function: While for most experiments we use a heuristic weighting function for  $\boldsymbol{w}_t$ , we also explored learning an effective weighting function (although this is not truly zero-shot). To do this, we aggregate scores for each image  $\boldsymbol{x}$  and class  $y_k$  into 20 buckets, with each bucket covering a small slice of timestep values:
$$
\boldsymbol {b} _ {i} (\boldsymbol {x}, \mathrm {y} _ {k}) = \mathbb {E} _ {\epsilon , t \sim \mathcal {U} [ 0. 0 5 i, 0. 0 5 (i + 1) ]} \| \boldsymbol {x} - \tilde {\boldsymbol {x}} _ {\boldsymbol {\theta}} \big (\boldsymbol {x} _ {t}, \phi (\mathrm {y} _ {k}), t \big) \| _ {2} ^ {2}
$$
where we estimate the expectation with Monte Carlo sampling (typically around 100 samples). We then learn a 20-feature linear model with parameters  $[v_0, \dots, v_{19}]$  over these buckets:
$$
p _ {\boldsymbol {v}} (y = \mathrm {y} _ {k} | \boldsymbol {x}) = \frac {\exp \left(\sum_ {i = 0} ^ {1 9} - \boldsymbol {v} _ {i} \boldsymbol {b} _ {i} (\boldsymbol {x} , \mathrm {y} _ {k})\right)}{\sum_ {\mathrm {y} _ {j} \in [ \mathrm {y} _ {K} ]} \exp \left(\sum_ {i = 0} ^ {1 9} - \boldsymbol {v} _ {i} \boldsymbol {b} _ {i} (\boldsymbol {x} , \mathrm {y} _ {j})\right)}
$$
trained with standard maximum likelihood over the data. At test-time we use the weighting
$$
\boldsymbol {w} _ {t} = \boldsymbol {v} _ {\left[ t / 0. 0 5 \right]}
$$
We generally found that (1) learned weighting functions are pretty similar across datasets, and (2) the weighting functions are transferable: the vs learned on one dataset get good accuracy when evaluated on other ones. On average, learned weights produced around  $1\%$  higher accuracy on zero-shot classification tasks, but we omitted the results from the main paper because using learned weights is not truly zero-shot.
Comparison of Weighting Functions. We compare the learned weighting functions with several heuristic functions on the Caltech101 dataset. We chose Caltech101 because it is high-resolution (SD performs poorly on low-resolution datasets), contains a diversity of image classes, was not used when we developed the heuristic weighting function, and only has 100 classes, so it is much faster to evaluate models on than ImageNet. We compare the following functions:
- VDM:  $\boldsymbol{w}_t = \mathrm{SNR}'(t)$ , the derivative of the signal to noise ratio with respect to  $t$ . This weighting scheme from Variational Diffusion Models (Kingma et al., 2021) directly trains the model on a lower bound of the likelihood.
- Simple:  $\mathbf{w}_t = \mathrm{SNR}(t)$ . This "simple" loss from Ho et al. (2020) results in a model that produces better images according to human judgements and FID scores, even though it results in worse likelihoods.
Weighting	Imagen	Stable Diffusion
VDM	62.0	71.9
Simple	56.1	71.4
Heuristic	68.9	73.0
Learned	70.2	73.1
Table 4: Percent accuracy for models on Caltech101 with different weighting schemes
- Heuristic:  $w_{t} = \exp (-6t)$ . Our hand-engineered weighting function; we found this by searching for a simple weighting function that works well on CIFAR-100, although we found empirically it generalizes very well to other datasets.
- Learned: Learning an effective weighting function on a held-out set of examples as described above.