SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: Top Row: Robust models are off-manifold robust. The figure depicts the inverse on- and off-manifold robustness of Resnet18 models trained with different objectives on CIFAR-10 (larger values correspond to less robustness). As we increase the importance of the robustness term in the training objective, the models become increasingly robust to off-manifold perturbations. At the same time, their robustness to on-manifold perturbations stays approximately constant. This means that the models become off-manifold robust. As we further increase the degree of robustness, both on- and off-manifold robustness increase. Bottom Row: The input gradients of robust models are perceptually similar to the score of the probability distribution, as measured by the LPIPS metric. We can also identify the models that have the most perceptually-aligned gradients (the global maxima of the yellow curves). Figures depict mean and minimum/maximum across 10 different random seeds. Additional results including the smoothness penalty can be found in Supplement Figure 9.
However, there are nuances to be observed here: while the data lies on the data manifold, the gradients of the optimal model lie on the signal manifold so this argument may not be exact. Nonetheless, for cases where the signal manifold and data manifold are identical, this argument holds and can explain a preference for off-manifold robustness over on-manifold robustness.
**Argument 2: Robust linear models are off-manifold robust.** It is a well-known result in machine learning (from, for example the representer theorem) that the linear analogue of gradient norm regularization, i.e., weight decay causes model weights to lie in the linear span of the data. In other words, given a linear model  $f(\mathbf{x}) = \mathbf{w}^{\top}\mathbf{x}$ , its input-gradient are the weights  $\nabla_{\mathbf{x}}f(\mathbf{x}) = \mathbf{w}$ , and when trained with the objective  $\mathcal{L} = \mathbb{E}_{\mathbf{x}}(f(\mathbf{x}) - y(\mathbf{x}))^{2} + \lambda \| \mathbf{w}\|^{2}$ , it follows that the weights have the following property:  $\mathbf{w} = \sum_{i=1}^{N}\alpha_{i}\mathbf{x}_{i}$ , i.e., the weights lie in the span of the data. In particular, if the data lies on a linear subspace, then so do the weights. Robust linear models are also infinitely off-manifold robust: for any perturbation  $\mathbf{u}_{\mathrm{off}}$  orthogonal to the data subspace,  $\mathbf{w}^{\top}(\mathbf{x} + \mathbf{u}_{\mathrm{off}}) = \mathbf{w}^{\top}\mathbf{x}$ , thus they are completely robust to off-manifold perturbations.
In addition, if we assume that there are input co-ordinates  $\mathbf{x}_i$  that are uncorrelated with the output label, then  $\mathbf{w}_i\mathbf{x}_i$  is also uncorrelated with the label. Thus the only way to minimize the mean-squared error is to set  $\mathbf{w}_i = 0$  (i.e., a solution which sets  $\mathbf{w}_i = 0$  has strictly better mean-squared error than one that doesn't), in which case the weights lie in the signal subspace, which consists of the subspace of all features correlated with the label. This shows that even notions of signal-distractor decomposition transfer to the case of linear models.
In this section, we conduct extensive empirical analysis to confirm our theoretical analyses and additional hypotheses. We first demonstrate that robust models exhibit relative off-manifold robustness (Section 4.1). We then show that off-manifold robustness correlates with the perceptual alignment of gradients (Section 4.2). Finally, we show that robust models exhibit a signal-distractor decomposition, that is they are relatively robust to noise on a distractor rather than the signal (Section 4.3). Below we detail our experimental setup. Any additional details can be found in the Supplementary material.
Figure 4: The input gradients of robust models trained with projected gradient descent on ImageNet and Imagenet-64x64 are perceptually similar to the score of the probability distribution, as measured by the LPIPS metric. On Imagenet-64x64, we also trained excessively robust models.
Data sets and Robust Models. We use CIFAR-10 [25], ImageNet and ImageNet-64 [26], and an MNIST dataset [27] with a distractor, inspired by [11]. We train robust Resnet18 models [28] with (i) gradient norm regularization, (ii) randomized smoothing, (iii) a smoothness penalty, and (iv)  $l_{2}$ -adversarial robust training with projected gradient descent. The respective loss functions are given in the Supplementary material. On ImageNet, we use pre-trained robust models from [29].
Measuring On- and Off-Manifold Robustness. We measure on- and off-manifold robustness by perturbing data points with on- and off-manifold noise (Section 3.1). For this, we estimate the tangent space of the data manifold with an auto-encoder, similar to [22, 30, 21]. We then draw a random noise vector and project it onto the tangent space. Perturbation of the input in the tangent direction is used to measure on-manifold robustness. Perturbation of the input in the orthogonal direction is used to measure off-manifold robustness. To measure the change in the output of a classifier with  $C$  classes, we compute the  $L_{2}$ -norm  $||f(x) - f(x + u)||_{2}$ .
Measuring Perceptual Alignment. To estimate the perceptual alignment of model gradients, we would ideally compare them with the gradients of the Bayes optimal classifier. Since we do not have access to the Bayes optimal model's gradients  $\nabla_{\mathbf{x}}p(y\mid \mathbf{x})$ , we use the score  $\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid y)$  as a proxy, as both lie on the same data manifold. Given the gradient of the robust model and the score, we use the Learned Perceptual Image Patch Similarity (LPIPS) metric [31] to measure the perceptual similarity between the two. The LPIPS metric computes the similarity between the activations of an AlexNet [32] and has been shown to match human perception well [31]. In order to estimate the score  $\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid y)$ , we make use of the diffusion-based generative models from Karras et al. [10]. Concretely, if  $D(\mathbf{x},\sigma)$  is a denoiser function for the noisy probability distribution  $p(\mathbf{x},\sigma)$  (compare Section 3.2), then the score is given by  $\nabla_{\mathbf{x}}\log p(\mathbf{x},\sigma) = (D(\mathbf{x},\sigma) - \mathbf{x}) / \sigma^2$  [10, Equation 3]. We use noise levels  $\sigma = 0.5$  and  $\sigma = 1.2$  on CIFAR-10 and ImageNet-64, respectively.
4.1 Evaluating On- vs. Off-manifold Robustness of Models
We measure the on- and off-manifold robustness of different Resnet18 models on CIFAR-10, using the procedure described above. We measure how much the model output changes in response to an input perturbation of a fixed size (results for different perturbation sizes can be found in Supplement Figure 9). The models were trained with three different robustness objectives and to various levels of robustness. The results are depicted in the top row of Figure 3. Larger values in the plots correspond to a larger perturbation in the output, that is less robustness. For little to no regularization (the left end of the plots), the models are less robust to random changes off- than to random changes on- the data manifold (the red curves lie above the green curves). Increasing the amount of robustness makes the models increasingly off-manifold robust (the red curves decrease monotonically). At the same time, the robustness objectives do not affect the on-manifold robustness of the model (the green curves stay roughly constant). This means that the robust models become relatively off-manifold robust (Definition 1). At some point, the robustness objectives also start to affect the on-manifold behavior of the model, so that the models become increasingly on-manifold robust (the green curves start to fall). As can be seen by a comparison with the accuracy curves in the bottom row of Figure 3, increasing on-manifold robustness mirrors a steep fall in the accuracy of the trained models (the
Figure 6: The figure depicts the input gradients of models belonging to different regimes of robustness. The numbers above the images indicate the perceptual similarity with the score, as measured by the LPIPS metric. Top row: ImageNet-64x64. Bottom row: CIFAR-10. Weakly robust models are accurate but not off-manifold robust. Bayes-aligned robust models are accurate and off-manifold robust. These are exactly the models that have perceptually aligned gradients. Excessively robust models are excessively on-manifold robust which makes them inaccurate. Best viewed in digital format.
green and blue curves fall in tandem). Remarkably, these results are consistent across the different types of regularization.
4.2 Evaluating the Perceptual Alignment of Robust Model Gradients
We now show that the input gradients of an intermediate regime of accurate and relatively off-manifold robust models ("Bayes-aligned robust models") are perceptually aligned, whereas the input gradients of weakly robust and excessively robust models are not. As discussed above, we measure the perceptual similarity of input gradients with the score of the probability distribution. The bottom row of Figure 3 depicts our results on CIFAR-10. For all three robustness objectives, the perceptual similarity of input gradients with the score, as measured by the LPIPS metric, gradually increases with robustness (the orange curves gradually increase). The perceptual similarity then peaks for an intermediate amount of robustness, after which it begins to decrease. Figure 4 depicts our results on ImageNet and ImageNet-64x64. Again, the perceptual similarity of input gradients with the score gradually increases with robustness. On ImageNet-64x64, we also trained excessively robust models that exhibit a decline both in accuracy and perceptual alignment.
To gain intuition for these results, Figure 6, as well as Supplementary Figures 10-15, provide a visualization of model gradients. In particular, Figure 6 confirms that the model gradients belonging to the left and right ends of the curves in Figure 3 are indeed not perceptually aligned, whereas the model gradients around the peak (depicted in the middle columns of Figure 6) are indeed perceptually similar to the score.
While we use the perceptual similarity of input gradients with the score as a useful proxy for the perceptual alignment of input gradients, we note that this approach has a theoretical foundation in the energy-based perspective on discriminative classifiers [33]. In particular, Srinivas and Fleuret [20], Zhu et al. [34] have suggested that the input gradients of softmax-based discriminative classifiers could be related to the score of the probability distribution. To the best of our knowledge, our work is the first to quantitatively compare the input gradients of robust models with independent estimates of the score.
Figure 5: Robust Models are relatively robust to noise on a distractor.
4.3 Evaluating Signal vs. Distractor Robustness for Robust Models
We now show that robust models are relatively robust to noise on a distractor. Since a distractor is by definition not part of the signal manifold (Section 3.2), this serves as evidence that the input
gradients of robust models are aligned with the signal manifold. In order to control the signal and the distractor, we create a variant of the MNIST dataset where we artificially add the letter "A" as a distractor. This construction is inspired by Shah et al. [11], for details see Supplement Figure 16. Because we know the signal and the distractor by design, we can add noise to only the signal or only the distractor and then measure the robustness of different models towards either type of noise. We call the ratio between these two robustness values the relative noise robustness of the model. Figure 5 depicts the relative noise robustness both for a standard- and an adversarially robust Resnet18. From Figure 5, we see that the standard model is already more robust to noise on the distractor. The robust model, however, is relatively much more robust to noise on the distractor. Since distractor noise is by definition off-manifold robust w.r.t. the signal manifold, this result serves as evidence for Hypothesis 2. Moreover, the input gradients of the robust model (depicted in the supplement) are perceptually aligned. Hence, this experiment also serves as evidence for Hypothesis 1.
Three Regimes of Robustness. Our experimental results show that different robust training methods show similar trends in terms of how they achieve robustness. For small levels of robustness regularization, we observe that the classifiers sensitivity to off-manifold perturbations slowly decreases (weak robustness), eventually falling below the on-manifold sensitivity, satisfying our key property of (relative) off-manifold robustness, as well as alignment with the Bayes classifier (Bayes-aligned robustness). Excessive regularization causes models to become insensitive to on-manifold perturbations, which often corresponds to a sharp drop in accuracy (excessive robustness).
The observation that robust training consists of different regimes (weak-, Bayes-aligned-, and excessive robustness) calls us to rethink standard robustness objectives and benchmarks [35], which do not distinguish on- and off-manifold robustness. An important guiding principle here can be not to exceed the robustness of the Bayes optimal classifier.
The Limits of Gradient-based Model Explanations. While Shah et al. [11] find experimentally that gradients of robust models highlight discriminative input regions, we ground this discussion in the signal-distractor decomposition. Indeed, as the gradients of the Bayes optimal classifier are meaningful in that they lie tangent to the signal manifold, we can also expect gradients of robust models to mimic this property. Here the zero gradient values indicate the distractor distribution. However, if a meaningful signal-distractor decomposition does not exist for a dataset, i.e., the data and the signal distribution are identical, then we cannot expect gradients even the gradients of the Bayes optimal classifier to be "interpretable" in the sense that they highlight all the input features as the entire data is the signal.
Limitations. First, while our experimental evidence strongly indicates support for the hypothesis that off-manifold robustness emerges from robust training, we do not provide a rigorous theoretical explanation for this. We suspect that it may be possible to make progress on formalizing Argument 1 in Section 3.3 upon making suitable simplifying assumptions about the model class. Second, we only approximately capture the alignment to the Bayes optimal classifier using the score-gradients from a SOTA diffusion model as a proxy, as we do not have access to a better proxy for a Bayes optimal classifier.
The authors would like to thank Maximilian Müller, Yannic Neuhaus, Usha Bhalla and Ulrike von Luxburg for helpful comments. Sebastian Bordt is supported by the German Research Foundation through the Cluster of Excellence "Machine Learning - New Perspectives for Science" (EXC 2064/1 number 390727645) and the International Max Planck Research School for Intelligent Systems (IMPRS-IS). Suraj Srinivas and Hima Lakkaraju are supported in part by the NSF awards IIS-2008461, IIS-2040989, IIS-2238714, and research awards from Google, JP Morgan, Amazon, Harvard Data Science Initiative, and the Digital, Data, and Design  $(\mathrm{D}^{3})$  Institute at Harvard. The views expressed here are those of the authors and do not reflect the official policy or position of the funding agencies.
Theorem A.1 (Equivalence between off-manifold robustness and on-manifold alignment). A function  $f: \mathbb{R}^d \to \mathbb{R}$  exhibits on-manifold gradient alignment if and only if it is off-manifold robust wrt normal noise  $\mathbf{u} \sim \mathcal{N}(0, \sigma^2)$  for  $\sigma \rightarrow 0$  (with  $\rho_1 = \rho_2$ ).
Proof. We proceed by observing that we can decompose the input-gradient into on-manifold and off-manifold components by projecting onto the tangent space and its orthogonal component respectively, i.e.,  $\nabla_{\mathbf{x}}f(\mathbf{x}) = \mathbb{P}_x\nabla_{\mathbf{x}}f(\mathbf{x}) + \mathbb{P}_x^\perp \nabla_{\mathbf{x}}f(\mathbf{x})$
We also observe that we can write the gradient norm in terms of an expected dot product, i.e.,  $\frac{1}{\sigma^2}\mathbb{E}_{\mathbf{u}\sim \mathcal{N}(0,\sigma^2)}(\nabla_{\mathbf{x}}f(\mathbf{x})^\top \mathbf{u})^2 = \frac{1}{\sigma^2}\nabla_{\mathbf{x}}f(\mathbf{x})^\top \mathbb{E}(\mathbf{uu}^\top)\nabla_{\mathbf{x}}f(\mathbf{x}) = \| \nabla_{\mathbf{x}}f(\mathbf{x})\|^2.$
Using these facts we can compute the norm of the off-manifold component as follows,
$$
\begin{array}{l} \underbrace {\frac {\| \nabla_ {\mathbf {x}} f (\mathbf {x}) - \mathbb {P} _ {x} \nabla_ {\mathbf {x}} f (\mathbf {x}) \| ^ {2}}{\| \nabla_ {\mathbf {x}} f (\mathbf {x}) \| ^ {2}}} _ {\text {O n - m a n i f o l d g r a d i e n t a l g r i n m e n t}} = \frac {\| \mathbb {P} _ {x} ^ {\perp} \nabla_ {\mathbf {x}} f (\mathbf {x}) \| ^ {2}}{\| \nabla_ {\mathbf {x}} f (\mathbf {x}) \| ^ {2}} \\ = \frac {\frac {1}{\sigma^ {2}} \mathbb {E} _ {\mathbf {u} _ {\mathrm {o f f}} \sim \mathcal {N} (0 , \sigma^ {2} \Sigma)} (\nabla_ {\mathbf {x}} f (\mathbf {x}) ^ {\top} \mathbf {u} _ {\mathrm {o f f}}) ^ {2}}{\frac {1}{\sigma^ {2}} \mathbb {E} _ {\mathbf {u} \sim \mathcal {N} (0 , \sigma^ {2})} (\nabla_ {\mathbf {x}} f (\mathbf {x}) ^ {\top} \mathbf {u}) ^ {2}}; \Sigma = \operatorname {C o v} (\mathbf {u} _ {\mathrm {o f f}}) = \mathbb {P} _ {x} ^ {\perp} (\mathbb {P} _ {x} ^ {\perp}) ^ {\top} \\ = \lim  _ {\sigma \rightarrow 0} \underbrace {\frac {\mathbb {E} _ {\mathbf {u} _ {\text {o f f}} \sim \mathcal {N} (0 , \sigma^ {2} \Sigma)} \left(f (\mathbf {x} + \mathbf {u} _ {\text {o f f}}) - f (\mathbf {x})\right) ^ {2}}{\mathbb {E} _ {\mathbf {u} \sim \mathcal {N} (0 , \sigma^ {2})} \left(f (\mathbf {x} + \mathbf {u}) - f (\mathbf {x})\right) ^ {2}}} _ {\text {O f f - m a n i f o l d r o b u s t n e s s}} \\ \end{array}
$$
The second line is obtained by using the fact above regarding re-writing the gradient norm in terms of the expected dot product, and the final line is obtained by using a first order Taylor expansion, which is exact in the limit of small sigma. From the equality of first and last terms, we have that the on-manifold gradient alignment  $\Leftrightarrow$  the off-manifold robustness.
Theorem A.2. The input-gradients of Bayes optimal classifiers lie on the signal manifold  $\Leftrightarrow$  Bayes optimal classifiers are relative off-manifold robust.
Proof. From definition 3, it is clear that given a classification problem, there exists a single distractor distribution  $d(\mathbf{x})$ . Now, we take gradients of log probabilities of the Bayes optimal classifiers, which results in:
$$
\nabla_ {\mathbf {x}} \log p (y = i \mid \mathbf {x}) = \nabla_ {\mathbf {x}} \log p (\mathbf {x} \mid y = i) - \sum_ {j} p (y = j \mid \mathbf {x}) \nabla_ {\mathbf {x}} \log p (\mathbf {x} \mid y = j)
$$
We notice first that the vectors  $\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid y)$  all lie tangent to the data manifold by definition, as this data generating process  $p(\mathbf{x}\mid y)$  itself defines the data manifold. As  $\nabla_{\mathbf{x}}\log p(y\mid \mathbf{x})$  is a linear combination of the class-conditional generative model gradients, it follows that the input-gradient of the Bayes optimal model also lie tangent to the data manifold. Now, like any vector on the tangent space at  $\mathbf{x}$ , it can be decomposed into signal and distractor components. Computing the distractor, we find that
$$
\nabla_ {\mathbf {x}} \log p (y \mid \mathbf {x}) \odot (1 - \mathbf {m} ^ {*} (\mathbf {x})) = d (\mathbf {x}) - \sum_ {j} p (y = j \mid \mathbf {x}) d (\mathbf {x}) = 0
$$
This happens because the distractor is independent of the label, thus the distractor component is zero, and the input-gradient of the Bayes optimal model lies entirely on the signal manifold. From Theorem A.1, it follows that when a model gradients lie on a manifold, it is also off-manifold robust wrt that manifold.
B.1 Robust Training Objectives
We consider the following robust training objectives, where  $l(x,y)$  denotes the cross-entropy loss function.
1. Gradient norm regularization:  $l(f(x),y) + \lambda \| \nabla_{\mathbf{x}}f(\mathbf{x})\| _2^2$  with a regularization constant  $\lambda$
2. A smoothness penalty:  $l(f(x), y) + \lambda \mathbb{E}_{\epsilon \sim \mathcal{N}(0, \sigma^2)} \| f(x + \epsilon) - f(x) \|_2^2$  with a fixed noise level  $\sigma^2$  and a varying regularization constant  $\lambda$ .
3. Randomized Smoothing:  $\mathbb{E}_{\epsilon \sim \mathcal{N}(0,\sigma^2)}l(f(x + \epsilon),y)$  with a noise level  $\sigma^2$
4. Adversarial Robust Training:  $l(f(\tilde{x}), y)$  where  $\tilde{x} = \arg \max_{\tilde{x} \in B_{\epsilon}(x)} l(f(\tilde{x}), y)$  and  $\tilde{x}$  was obtained from the  $\epsilon$ -ball around  $x$  using projected gradient descent.
B.2 Training Details
On CIFAR-10, we trained Resnet18 models for 200 epochs with an initial learning rate of 0.025. When training with gradient norm regularization or the smoothness penalty and large regularization constants we reduced the learning rate proportional to the increase in the regularization constant. After 150 and 175 epochs, we decayed the learning rate by a factor of 10.
On ImageNet-64x64, we trained Resnet18 models for 90 epochs with a batch size of 4096 and an initial learning rate of 0.1 that was decayed after 30 and 60 epochs, respectively. We used the same parameters for projected gradient descent (PGD) as in [29], that is we took 3 steps with a step size of  $2\epsilon /3$
On the MNIST dataset with a distractor, we trained a Resnet18 model for 9 epochs with an initial learning rate of 0.1 that was decayed after 3 and 6 epochs, respectively. We also trained an  $l_{2}$ -adversarily robust Resenet18 with projected gradient descent (PGD). We randomly chose the perturbation budget  $\epsilon \in \{1,4,8\}$  and took 10 steps with a step size of  $\alpha = 2.5\epsilon /10$ .
B.3 Diffusion Models
On CIFAR-10, we use the unconditional diffusion model edm-cifar10-32x32-uncond-vp. For comparison, Figure Supplement Figure 17 presents the results with the conditional diffusion model edm-cifar10-32x32-cond-vp. On ImageNet-64x64, we use the conditional diffusion model edm-imagenet-64x64-cond-adm. All models are available at https://github.com/NVlabs/ edm. The noise levels  $\sigma = 0.5$  and  $\sigma = 1.2$  were chosen to maximize the perceptual quality of the estimate of the score obtained from the diffusion model before running the experiments with the LPIPS metric.
B.4 Conditional- and Unconditional Diffusion Models
Based on our theory, there are two principled ways to evaluate diffusion models and discriminative classifiers. The first is to take the input gradient with respect to a single class and compare it to a conditional diffusion model. The second is to sum the input gradients of all the classes and compare it to an unconditional diffusion model (i.e., computing marginal probabilities). The main paper presents the first approach on ImageNet and ImageNet-64x-64 and the second approach on CIFAR-10. Supplement Figure 17 presents the first approach on CIFAR-10. We observe that there is no qualitative difference, i.e. both approaches give the same result.
B.5 Model Gradients
With the unconditional diffusion model, we sum the input gradients across all classes. With the conditional diffusion model, we consider the input gradient with respect to the predicted class. We consider input gradients before the softmax [20].
Figure 7: Left: Images from CIFAR10. Middle: Random perturbations on the data manifold. Right: Random perturbations off the data manifold.
B.6 CIFAR-10 Autoencoder
We use https://github.com/clementchadebec/benchmark_VAE to train an autoencoder on CIFAR-10 with a latent dimension  $k = 128$ . We use a default architecture and training schedule. We then use the autoencoder to estimate, at each data point, a 128-dimensional tangent space. Figure 7 depicts random directions within the estimated tangent spaces.
B.7 Pre-Trained Robust Models on ImageNet