SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On ImageNet, we use the pre-trained robust Resnet18 models form https://github.com/microsoft/robust-models-transfer. To load these models, we use the robustness library https://github.com/MadryLab/robustness.
B.8 Estimating the Score on ImageNet
We estimate the score on ImageNet using the diffusion model for ImageNet-64x64. To estimate the score, we simply down-scale an image to 64x64.
B.9 MNIST with a Distractor
The MNIST data set with a distractor is inspired by [11]. The data set consists of gray-scale images of size 56x28. Every image contains a single MNIST digit and the distractor. We choose the fixed letter "A" as the distractor. On every image, we randomly place the distractor on top or below the MNIST digit. In order to estimate the relative noise robustness, we separately add different levels of noise to the signal or distractor. Figure 16 depicts images and models gradients on this data set.
B.10 The LPIPS metric
The LPIPS metric measures the perceptual similarity between two different images. The metric itself corresponds to a loss, meaning that lower values correspond to more similar images [31]. The figures in the main paper depict 1-LPIPS, that is higher values correspond to more similar images.
C Additional Plots
The figures below depict the model gradients of different types of models, ranging from weakly robust to excessively robust. The figures depict the relationship between model gradients and the score qualitatively. This complements the quantitative results in the main paper.
Figure 8: Perturbations of different size, as measured by the  $L_{2}$ -norm of the perturbation. The Figure depicts the original images with random perturbations of the given  $L_{2}$ -norm.
(a) Perturbation norm  $L_{2} = 1.4$
(b) Perturbation norm  $L_{2} = 2.6$
(c) Perturbation norm  $L_{2} = 4.8$
Figure 9: On- and off-manifold robustness of Resnet18 models trained with different objectives on CIFAR-10 (larger values correspond to less robustness). The Figure depicts the on- and off-manifold robustness for perturbations of different size, as measured by the  $\bar{L}_2$ -norm of the perturbation. Compare Figure 3 in the main paper.
(d) Perturbation norm  $L_{2} = 8.9$
Increasing Robustness
Figure 10: The input gradients of different models trained with gradient norm regularization on CIFAR-10. The top rows depict the image, the score, and the input gradients of unrobust models. The middle rows depict the perceptually aligned input gradients of robust models. The bottom rows depict the input gradients of excessively robust models. Best viewed in digital format.
Increasing Robustness
Figure 11: The input gradients of different models trained with projected gradient descent on CIFAR-10. The top rows depict the image, the score, and the input gradients of unrobust models. The lower middle rows depict the perceptually aligned input gradients of robust models. The bottom rows depict the input gradients of excessively robust models. Best viewed in digital format.
Increasing Robustness
Figure 12: The input gradients of different models trained with a smoothness penalty on CIFAR-10. The top rows depict the image, the score, and the input gradients of unrobust models. The middle rows depict the perceptually aligned input gradients of robust models. The bottom rows depict the input gradients of excessively robust models. Best viewed in digital format.
Increasing Robustness
Figure 13: The input gradients of different models trained with randomized smoothing on CIFAR-10. The top rows depict the image, the score, and the input gradients of unrobust models. The middle rows depict the perceptually aligned input gradients of robust models. The bottom rows depict the input gradients of excessively robust models. Best viewed in digital format.
Increasing Robustness
Figure 14: The input gradients of different models trained with projected gradient descent on ImageNet-64x64. The top rows depict the image, the score, and the input gradients of unrobust models. The middle rows depict the perceptually aligned input gradients of robust models. The bottom rows depict the input gradients of excessively robust models. Best viewed in digital format.
Figure 15: The input gradients of different models trained with projected gradient descent on ImageNet. The models are from [29]. The top rows depict the image, the score, and the input gradients of unrobust models. The bottom rows depict the perceptually aligned input gradients of robust models. Best viewed in digital format.
(a) Images from the data set.
(b) Noise on the signal.
(c) Noise on the distractor.
(d) Input gradients of a Resnet18.
(e) Input gradients of an adversarially robust Resnet18.
Figure 16: The MNIST dataset with a distractor used to create Figure 5 in the main paper.
Figure 17: Measuring the perceptual alignment between the input gradients of Resnet18 models and the score as estimated with a conditional diffusion model. Compare Supplement Section B.4 and Figure 3 in the main paper. The Figure in the main paper depicts results with an unconditional diffusion model.