SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Results are shown in Table 4. The heuristic weighting function outperforms Simple and VDM for both models. Interestingly, SD appears to be more robust to the choice of weighting function than Imagen. Mechanistically, the reason is that "Simple" and "VDM" weighting put more weight on earlier timesteps than "Heuristic" and Imagen tends to be an inaccurate classifier at very small noise levels. We intuitively believe this is a consequence of pixel vs latent diffusion. The learned weighting only does slightly better than heuristic weighting despite not being truly zero-shot. We found similar results to hold on other datasets.
C Details on Attribute Binding Tasks and Prompts
We use the relational dataset from Lewis et al. (2022) for the attribute binding experiments. Each image consists of two objects of different shapes and colors; for tasks involving size we filter out examples where both objects are the same size. Each image contains two objects with different attributes shape  $\in$  {cube, sphere, cylinder}, color  $\in$  {blue, cyan, blue, brown, gray, green, purple, red, yellow}, size  $\in$  {small, large}, and position  $\in$  {left, right}.
Given a task (e.g. Shape|Size), we construct a task-specific description for an object as follows:
"On the {position} is a" if Position tasks else "A" +
"size" if Size task else "+"
"color" if Color task else "+"
"shape." if Shape task else "object."
For recognition and binding tasks, we randomly select one of the two objects in the image to be the positive example and then use its description as the positive prompt. For pair tasks, we join the descriptions for both objects together with "and" (removing the period from the first description and lowercasing the second one) for the positive prompt.
To construct a negative example for recognition tasks, we replace the positive attribute with a random attribute not in the image. For binding tasks, we replace one of positive description's attributes with the other object's attribute (e.g., for Shape|Color, we replace shape).
For pair tasks, there is a choice in how the two objects are ordered (e.g. "On the left is a cube and on the right is a sphere" vs "On the right is a sphere and on the left is a cube". We follow the preference of stating the leftmost position/shape/color/size first in that order. For example, this means we will always start with "On the left..." rather than "On the right...". Similarly, the negative example for Color, Size in Figure 4 is "A large yellow object and small gray object" rather than "A small gray object and a large yellow object" because we prefer to first put the leftmost color over the leftmost size.
We experimented with a variety of other prompts, but found none to work substantially better than these simple ones.
Figure 5: Model reliability diagram comparing confidence measures ofImagen on CIFAR-100. The number of model calls used in Algorithm 1 produces better-calibrated confidences than using the actual scores for different classes.
D Calibration
It is desirable for classifiers, especially when used in the zero-shot setting with possibly out-of-domain examples, to be well calibrated. In other words, if a classifier predicts a label  $\tilde{y}_i$  with probability  $p$ , the true label should be  $\tilde{y}_i$  roughly  $100 \cdot p\%$  of the time. However, the diffusion model classifier does not directly produce probabilities for classes. While  $p(y_i = y_k | x_i)$  should roughly be proportional to the expectation in Equation (2) when exponentiated (i.e. we can apply a softmax to the average weighted scores to get probabilities), in practice our estimates of the expectations are very noisy and do not provide well-calibrated scores.
We propose a simple alternative that takes advantage of early pruning: we use the total number of diffusion model calls used for the image as a calibration measure. The intuition is that a harder example will require more scores to determine the arg min class with good statistical significance.
More details on the two calibration methods are below:
Temperature-scaled raw scores. We use  $s_{\mathbf{y}_k}(\pmb{x})$  to denote the weighted average squared error for class  $\mathbf{y}_k$  on image  $\pmb{x}$ , i.e., the Monte-Carlo estimate for the re-weighted VLB in equation 2. We turn these scores into an estimated probability by applying a softmax with temperature:
$$
p _ {\theta} (y = y _ {k} | \boldsymbol {x}) = \frac {\exp \left(- s _ {y _ {k}} (\boldsymbol {x}) / \tau\right)}{\sum_ {y _ {j} \in [ y _ {K} ]} \exp \left(- s _ {y _ {j}} (\boldsymbol {x}) / \tau\right)}
$$
Note that this approach requires good score estimates for each class, so it is not compatible with the class pruning method presented in Section 3.1.
Platt-scaled number of scores. Our other confidence method relies on the total number of scores needed to eliminate all other classes as candidates. Let  $\tilde{y}(\boldsymbol{x})$  denote the predicted class for example  $\boldsymbol{x}$  and  $n(\boldsymbol{x})$  be the total number of calls to  $\tilde{\boldsymbol{x}}_{\theta}$  used to obtain the prediction when running Algorithm 1. Then we estimate
$$
p _ {\theta} (y = \tilde {y} (\boldsymbol {x}) | \boldsymbol {x}) = \operatorname {s i g m o i d} (- n (\boldsymbol {x}) / \tau + \beta)
$$
We learn  $\tau$  (and  $\beta$  for Platt scaling) on a small held-out set of examples.
We show reliability diagrams (DeGroot & Fienberg, 1983) and report Expected Calibration Error (Guo et al., 2017) (ECE) for the methods in Figure 5. Using a small held-out set of examples, we apply temperature scaling (Guo et al., 2017) for the score-based confidences and Platt scaling (Platt et al., 1999) for the number-of-scores confidences, (see Appendix D for details). Number of scores is fairly well-calibrated, showing it is possible to obtain reasonable confidences from diffusion model classifiers despite them not providing a probability distribution over classes.
E Imagen's Super-resolution Models
Imagen is a cascaded diffusion model (Ho et al., 2022) consisting of a  $64 \times 64$  low-resolution model and two super-resolution models, one that upsamples the image from  $64 \times 64$  to  $256 \times 256$  and one
Prompt: A photo of the number 8.
Figure 6: Example generated images from Stable Diffusion and Imagen for generating photos with text.
that upsamples from  $256 \times 256$  to  $1024 \times 1024$ . However, we found only the  $64 \times 64$  model to work well as a zero-shot classifier. The super-resolution models condition on a low-resolution input image, which means they denoise effectively regardless of the input prompt and thus aren't as sensitive to the class label. For example, unlike with Figure 3, high-resolution denoising with different text prompts produces images imperceptibly different to the human eye because they all agree with the same low resolution image.Imagen's super-resolution models are trained with varying amount of Gaussian noise added to the low-resolution input image (separate from the noise added to the high-resolution image being denoised). We were able to alleviate the above issue somewhat by using a large amount of such noise, but ultimately did not achieve very strong results with the high-resolution models. For example, the  $64 \times 64$  to  $256 \times 256$  model achieves an accuracy of  $16.1\%$  on ImageNet.
We further experimented with combining the low-resolution model's scores with the  $64 \times 64$  to  $256 \times 256$  model's. To do this, we used the learned weighting scheme detailed in Appendix B, but with learning 40 weights: 20 for the low resolution model and 20 for the super-resolution model. However, we found the learned weighting scheme put almost no weight on the super-resolution model's scores, and did not perform significantly better than the low-resolution model did on its own.