Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness
Authors: Suraj Srinivas
Sebastian Bordt
Himabindu Lakkaraju
Venue: NeurIPS 2023 spotlight (2023)
Keywords: robustness
generative models
perceptually aligned gradients
bayes optimality
manifold hypothesis

Abstract: One of the remarkable properties of robust computer vision models is that their input-gradients are often aligned with human perception, referred to in the literature as perceptually-aligned gradients (PAGs). Despite only being trained for classification, PAGs cause robust models to have rudimentary generative capabilities, including image generation, denoising, and in-painting. However, the underlying mechanisms behind these phenomena remain unknown. In this work, we provide a first explanation of PAGs via \emph{off-manifold robustness}, which states that models must be more robust off- the data manifold than they are on-manifold. We first demonstrate theoretically that off-manifold robustness leads input gradients to lie approximately on the data manifold, explaining their perceptual alignment. We then show that Bayes optimal models satisfy off-manifold robustness, and confirm the same empirically for robust models trained via gradient norm regularization, randomized smoothing, and adversarial training with projected gradient descent. Quantifying the perceptual alignment of model gradients via their similarity with the gradients of generative models, we show that off-manifold robustness correlates well with perceptual alignment. Finally, based on the levels of on- and off-manifold robustness, we identify three different regimes of robustness that affect both perceptual alignment and model accuracy: weak robustness, bayes-aligned robustness, and excessive robustness. Code is available at https://github.com/tml-tuebingen/pags.

Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness
Suraj Srinivas' Harvard University Cambridge, MA
Sebastian Bordt*
University of Tübingen, Tübingen Al Center
Tübingen, Germany
ssrinivas@seas harvard. haw.aaasssaaaa..aarr.eeeeeee
sebastian. sebastian.bordeee Saaastiia.bordduui--teeee bburbbuunituuebingn .e
Himabindu Lakkaraju Harvard University
Cambridge, MA h hlakkaraju@hbs. edu
Abstract
One of the remarkable properties of robust computer vision models is that their input-gradients are often aligned with human perception, referred in the literature as perceptually--lignee gradients (PAGs). Despite only being trained for classi- fication, PAGs cause robust models to have rudimentary generative capabilities, including image generation, denoising, and in in-painting. However, the underlying mechanisms behind these phenomena remain unknown. this work, we provide first explanation of PAGs via off-manifold robusiness, which states that models must be more robust off- the data manifold than they are on-manifold. We first demonstrate theoretically that off-manifol robustness leads input gradients to lie approximately on the data manifold, explaining their perceptual alignment. We then show that Bayes optimal models satisfy off-manifol robustness, and confirm the same empirically for robust models trained via gradient norm regu- larization, randomized smoothing, and adversarial training with projected gradi- ent descent. Quantifying the perceptual alignment of model gradients via their similarity with the gradients of generative models, we show that off-maninee mfanfold robustness correlates well with perceptual alignment. Finally, based on the lev- els of on- and off-manifold robustness, we identify three different regimes of robustness that affect both perceptual alignment and model accuracy: weak ro- bustness, bayes-aligned robustness, and excessive robustness. Code is available https:///githu: com com/tml- seal--uubinggn/aagg
1 Introduction
An important desideratum for machine learning models is robustness, which requires that models be insensitive small amounts of noise added to the input. In particular, adversarial robustness requires models to be insensitive to adversarially chosen perturbations of the input. Tsipras et al. [I] first observed an unexpected benefit of such models, namely that their auput-gradients were "significantly more human-all -alineed" (see Figure for examples of ceaptuulll--alinnee gradients). Santurkar et al. built on this observation to show that robust models could be used to perform rudimentary image synthesis an unexpected capability of models trained in purely discriminative manner. Subsequent works have made use of the perceptual alignment of robust model gradients improve zero-shot
Equal Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

CIFAR-10
ImageNet
Image
Score
Robust Gradient (PAG)
Standard Gradient
of robust Figure I: A demonstration of the perceptual alignment phenomenon. The classifiers ("robust gradient") are perceptually similar the score of diffusion models (IO], while being qualitatively distinct from input-gradients of standard models ("standard gradient"). Best viewed in digital format.
object localization 3 perform conditional image synthesis 4 and improve classifier robustness 5 6]
Kaur al. coined the term perceptually-aligreed gradients (PAGs), and showed that it occurs not just with adversarial training (8) but also with randomized smoothed models (9]. Recently, Ganz al. (6) showed that the relationship between robustness and perceptual alignment can also work in the opposite direction: approximately enforeing perceptual alignment of input gradients can increase model robustness.
Despite these advances, the underlying mechanisms behind the phenomenon of perceptually aligned gradients in robust models still unclear. Adding the confusion, prior works have used the same term, PAGs, to refer to slightly different phenomena. We ground our discussion by first identifying variations of the same underlying phenomenon.
Phenomenon (Perceptual Alignment). The gradients of robust models highlight perceptually relevant features [Z]. and highlight discriminative input regions while ignoring distractors [Ily.
Phenomenon 2 (Generative Capabilities). Robust models have rudimentary image synthesis capabil- ities, where samples are generated by maximizing a specific output class by iteratively following the direction dictated by the gradd-gradeetss [2].
Phenomenon 3 (Smoothgrad Interpretability). Smoothgrad visualizations (which are gradients of randomized smooth models) tend to be visually sharper than the gradients of standard models [[2].
While Kaur et al. 7 refer to Phenomenon 2 as PAGs, Ganz et al. 6 use this term to refer to Phenomenon 1 Since they are all related to the gradients of robust models, choose to collectively refer to all of these phenomena as PAGs.
While the various phenomena arising from PAGs are now well-documented, there is little to work that attempts to explain the underlying mechanism. Progress on this problem has been hard to achieve because PAGs have been described via purely qualitative criteria, and it has been unclear how to make these statements quantitative. In this work, we address these gaps and make one of the first attempts at explaining the mechanisms behind PAGs. Crucially, we ground the discussion by attributing PAGs to gradients lying on a manifold, based on an analysis of Bayes optimal classifiers. We make the following contributions:
1. We establish the first-known theoretical connections between Bayes optimal predictors and the perceptual alignment of classifiers via off-manifold robustness. We also identify the manifold w.r.t. which this holds, calling it the signal manifold.
2. We experimentally verify that models trained with gradient-norm regularization, noise augmentation and randomized smoothing all exhibit off-manifold robustness.
3. We quantify the perceptual alignment of gradients by their perceptual similarity with the score of the data distribution, and find that this measure correlates well with off-manifold robustness.

---

4.
We identify three regimes of robustness: weak robustness, bayes-aligned robustness and excessive robustness, that differently affect both perceptual alignment and model accuracy.
2 Related Work
Robust training of neural networks Prior works have considered two broad classes of model robustness: adversarial robustness and robustness to normal noise. Adversarial robustness is achieved by training with adversarial perturbations generated using project gradient descent [8], or by random- ized smoothing [9) which achieves certified robustness to adversarial attacks by locally averaging with normal noise. Robustness to normal noise is achieved by explicitly training with noise, technique that is equivalent to Tikhonov regularization for linear models [3]. For non-linear models, gradient- regularization [14] is equivalent training with normal noise under the limit of
training with infinitely many noise samples [15]. In this paper, we make use of all of these robust training approaches and investigate their relationship PAGs.
Gradient-based model explanations Several popular post hoc explanation methods [66] 17 12
estimate feature importances by computing gradients of the output with respect to input features and aggregating them over local neighborhoods (18). However, the visual quality criterion used to evaluate these explanations has given rise to methods that produce visually striking attribution maps, while being independent of model behavior [9]]. While (0) attribute visual quality implicit score-based generative modeling of the data distribution, [21] propose that it depends on explanations lying on the data manifold. While prior works have attributed visual quality to generative modeling of data distribution or explanations lying on data manifold, our work demonstrates for the first time that (1) off- off-manifold robustness is crucial factor, and (2) it not the data manifold distribution, rather the signal manifold / distribution (defined in Section 3.2 that is the critical factor in explaining the phenomenon of PAGs.
3 Explaining Perceptually-Aligned Gradients
Our goal in this section is to understand the mechanisms behind PAGs. We first consider PAGs as lying on a low-dimensional manifold, and show theoretically that such on-manifold alignment of gradients is equivalent to off-manifold robustness of the model. We then argue that Bayes optimal models achieve both off-manifold robustness and on-manifold gradient alignment. In doing so, we introduce the distinction between the data and the signal manifold, which is key understanding the manifold structure of PAGs. Finally, we present arguments for why empirical robust models also exhibit off off-manifold robustness.
Notation Throughout this paper, we consider the task of image classification with inputs Rd where x ~ X' and E [1,2,...C] with C-classes. We consider deep neural networks f : Rd + A C-1 which map inputs onto C-class probability simplex. We assume that the input data lies k-dimensional manifold in the d-dimensional ambient space. Formally, a k-dimensional differential manifold M Rdd is locally Euclidean in R At every point E M. we can define projection matrix P. E ddd that projects points onto the k-dimensional tangent space at x. We denote p-d. I- P2 as the projection matrix to the subspace orthogonal to the tangent space.
3.1 Off-Manifold Robustness + On-Manifold Gradient Alignment
We now show via geometric arguments that off-manifold robustness of models and on-manifold alignment of gradients are identical. We begin by discussing definitions of on- and off-manifold noise. Consider a point x on manifold M, and a noise vector u. Then, u is off-manifold noise, which we denote u := Uoff and u is on-manifold noise, if (x + u) x +
(x + u) which we denote u == Usn- In other words, if the noise vector lies on the tangent space then it is on-manifold, otherwise it is off-manifold. Given this definition, we can define relative off-manifold robustness. For simplicity, we consider a scalar valued function, which can correspond to one of the C' output classes of the model.
3

---

Definition 1. (Relative off-manifold robustness) A model f: R - IR is pr ooffmmanifold robust, wrr some noise distribution if
Eus (f(x + Uzg) - f(x)) En (x - (x))) S where Unf unf (u)
+
While this definition states that the model more robust off-manifold than overall, can equivalently be interpreted as being more robust off- than on-manifold (with factor _ Let us now define on-manifold gradient alignment.
Definition 2. (On-manifold gradient alignment) A model f B + RR has pz-on-manifold aligned gradients if Vxf(x) - f(x)1 [x)(()]|
P2 where
This definition captures the idea that the difference between the gradients and its on-manifold projection is small. If P2 = O, then the gradients exactly lie on the tangent space. We are now ready state the following theorem.
Theorem (Equivalence between off-manifold robustness and on-manifold alignment). A function f Rd + R exhibits on-manifold gradient alignment if and only if it is off-manifold robust wrt normal noise u NN00 f + (with p1 = P2).
Proof Idea. Using Taylor series expansion, in terms of gradients. We further use linear algebraic identities to simplify that yield the on- manifold gradient alignment objective. The full proof is given in the supplementary material.
re- write the off- off-manifold robustness objective
When the noise is small, on-manifold gradient alignment and off-manifold robustness are identical. To extend this to larger noise levels, would need to make assumptions regarding the curvature of the data manifold. Broadly speaking, the less curved the underlying manifold is (the closer is to being linear), the larger the off-manifold noise can add, without intersecting another point on the manifold. In practice, we expect image manifolds to be quite smooth, indicating relative off- manifoll robustness larger noise levels [22]. In this next part, will specify the properties of the specific manifold w.r.t. which these notions hold.
3.2 Connecting Bayes Optimal Classifiers and Off-Manifold Robustness
In this subsection, we aim to understand the manifold structure characterizing the gradients of Bayes optimal models. We proceed by recalling the concept of Bayes optimality.
Bayes optimal classifiers. we perfectly knew the data generating distributions for all the classes, ie.. p(x y i) for all C classes, could write the Bayes optimal classifier as p(y x) p y =)=
p(x y = j). This an oracle classifier that represents the "best possible" model one create from data with perfect knowledge of the data generating process. Given assumption that the data lies on a low-dimensional data manifold, the Bayes optimal classifier is uniquely defined on the data manifold. However, outside this manifold. its behaviour undefined as all of the class-conditional probabilities p(x y) are zero or undefined themselves. In order to link Bayes-optimality and robustness, which inherently about the behavior of the classifier outside the data manifold, we introduce ground- ground-truth perturbed data distribution that is also defined outside the data manifold. While we might consider many perturbed distributions, is convenient to consider the data generating distributions that are represented by denoising auto-encoders with a stochastic decoder (or equivalently, score-based generative models (23) ). Here, the stochasticity ensures that data- seiiaaggneratingggg probabilities are defined everywhere, not just on the data manifold. This approach allows us to estimate the data manifold, assuming that the autoencoder has bottleneck layer with features for d-dimensional ambient space (2)

---

signal
input
distractor
ply | signal) p(y Input)
y "blue flycatcher"
ply distractor p
Figure An illustration signal-distractor decomposition for bird classification task. The signal represents the discriminative parts of the input, while the distractor represents the non-discriminative
parts.
Distinguishing Between the Data and the Signal Manifold. Given a classification problem, one can often decompose the inputs into signal component and distractor component. For intuition, consider the binary task of classifying cats and dogs. Given an oracle data generating distribution of cats and dogs in diverse backgrounds, the label must be statistically independent of the background and depend purely on the object (cat or dog) in the image. In other words, there exist no spurious correlations between the background and the output label. In such casc,
call the object the signal and the background the distractor. Formally, for every input x there exists a binary mask m(x) (0, 1)dd such that the signal is given by sxx d(x) = x @ (1 m(x)). The signal and distractor components are orthogonal to one another (x(x) d(x) = 0), and we can decompose any input in this manner (x s(x) + d(x)). We provide now define the signal distal-ditrrcttor distributions.
m(x) and the distractor is given by
an illustrative figure in Figure 2 Using this,
Definition 3. Given a data distribution p(x y) for E [1, C?, we have masking functions m(x) such that the resulting distribution p(x @ (1 m) y) p(x (1-m)) is statistically independent of y. The sparsest such masking function (such that m = arg minm Ex~p(xls) [|m(()) llo). yields corresponding distribution p(x @ (1 m² (x))), which the distractor distribution, and iss counterpart p(x C m" y) the signal distribution.
While it is possible define signals and distractors more generally any linear projections of the data, in this work, we restrict ourselves to masking to relate to feature attributions (II). While the subject of finding such optimal masks is the topic of feature attribution (24), in this discussion, we shall assume that the optimal masks m and the corresponding signal and distractor distributions are known. Similar to decomposing any point on the manifold, we can also decompose any vector on the tangent space on the data manifold into signal and distractor components using the optimal mask m In other words, we can write
V Vxp(x|1)
VxP(x (2)x)11)00 () + mx)))
xxp(x
mmm mx(x))
signal axx) (has information about
distractor d(x) (indopendent yf
Finally, we note that this signal-distractor decomposition does not meaningfully exist for all classi- fication problems in that it can be trivial with the entire data distribution being equal to the signal, with zero distractors. Two examples of such cases are: (1) ordinary MNIST classification has no distractors due to the simplicity of the task, as the entire digit is predictive of the true label and the background zero. (2) multi-class classification with a large set of classes with diverse images also have no distractors due to the complexity of the task, as the background can already be correlated with class information. For example, if the dataset mixes natural images with deep space images, there is no single distractor distribution one can find via masking that is independent of the class label. This disadvantage arises partly because we aim to identify important pixels, whereas a more general definition based on identifying subspaces can avoid these issues. We leave the exploration of this more general class of methods to future work.
Given this definition of the signal and distractor distributions, we are ready to make the following theorem: the input-g input-gradient of Bayes optimal classifier lie the signal manifold, as opposed to the general data manifold.
5

---

Theorem 2. The input- grantteradents of Bayes optimal classifiers lie on the signal manifold.
Proof Idea. We first show that the gradients of the Bayes optimal predictor lie in the linear span of the gradients of the gradients of the data distribution, thus lying on the tangent space of the data distribution. Upon making a signal distradctt decomposition of the gradients of the Bayes optimal classifier, we find that the distractor term always zero, indicating that the gradients of the Bayes optimal classifier always lie on the signal manifold. The proof is given in the supplementary material.
The full proof is given in the Supplementary material. This theorem can be intuitively thought of as follows: the optimal classifier only needs to look at discriminative regions of the input in order to classify optimally. In other words, changing the imput values at discriminative signal regions likely to have larger effect on model output than changing the inputs slightly at unimportant distractor regions, indicating that the gradients of the Bayes optimal classifier highlight the signal. Thus, the Bayes optimal classifier does not need to model the distractor; only the signal is sufficient. This fact inspires us to make the following hypothesis to help us ground the discussion on the perceptual alignment of gradients:
Hypothesis 1. The input gradients of classifiers are perceptually aligned if and only if they lie on the signal manifold, thus highlighting only discriminative parts of the input.
This indicates that Bayes optimal model's gradients are perceptually aligned. Recall that the signal component only provides information about discriminative components of the input, and thus, we expect this to connect to perceptual alignment from Phenomenon [ Next, we ask whether the gradients of practical models, particularly robust models, are also off-manifold robust.
3.3 Connecting Robust Models and Off-Manifold Robustness
Previously, we saw that Bayes optimal classifiers have the property of relative off-manifold robustness. Here, we argue that off-manifold robustness also holds for robust models in practice. This is non- trivial and perhaps surprising claim: common robustness objectives such adversarial training, gradient-norm norm regularization, etc are isotropic, meaning that they do not distinguish between on- and off-manifold directions.
Hypothesis 2. Robust models are off off-manifold robust w.r.f. the signal manifold.
We provide two lines of argument in support for this hypothesis. Ultimately, however, our evidence is empirical and presented in the next section.
Combined robust objectives are non-isotropic. While robustness penalties itself are isotropic and do not prefer robustness in any direction, they are combined with the cross-entropy loss on data samples, which lie on the data manifold. Let us consider the example of gradient norm regularization. The objective given by:
Argument 1:
E (((f(x) + (()))) = E {(f(x),(()) + {(x)) +A f())()))
on-manifold objective
otI-manifold objective
|| [(x)] due to V fx) f = O, which is possible because the on-manifold and off-manifold parts of the gradient are orthogonal to each other. Assuming that we are able to decompose models into on- manifold and off-manifold parts, these two objectives apply to these decompositions independently. This argument states that in this robust training objective, there exists trade-off between the cross-entropy loss and the on-manifold robustness term, whereas there is no such trade-off for the off-manifold term, indicating that it is much easier to minimize off-manifold robustness than on- manifold. This argument also makes the prediction that increased on- on-manifold robustness must be accompanied by higher train loss and decreased out-of-sample performance, and we will test this in the experiments section.
Here we have used, ==== Voll /=

---

Gradient Norm (CIFAR-10]
Projected Gradient Descent (CIFAR-10)
Randomized Smoothing (CIFAR-10)
1.0
12-Norm
0.5
0.0
1.00
0.85
Accuracy
0.75
0.80
0.50
0.25
0.70
10-4
Regularization Constant
10-2
10a
Adv. Perturbation Budget (Epsilon)
10:4
10
10P 10
10 10
10
Noise Level
10"
10
10a
Figure 3: Top Row: Robust models are off-manifold robust. The figure depicts the inverse on- and off-manifold robustness of Resnet18 8 models trained with different objectives on CIFAR-10 (larger values correspond to less robustness). As we increase the importance of the robustness term in the training objective, the models become increasingly robust to off-manifold perturbations. At the same time, their robustness to mn-manifold perturbations stays approximately constant. This means that the models become off- manifold robust. As we further increase the degree of robustness, both on- and off- off-manifold robustness increase. Bottom Row: The input gradients of robust models are perceptually similar to the score of the probability distribution, as measured by the LPIPS metric. We can also identify the models that have the most perceptually-aligne the yellow curves). Figures depict mean and minimum/maximumn across 10 different random seeds. Additional results including the smoothness penalty can be found Supplement Figure
gradients (the global maxima of
However, there are nuances to be observed here: while the data lies on the data manifold, the gradients of the optimal model lie on the signal manifold so this argument may not be exact. Nonetheless, for cases where the signal manifold and data manifold are identical, this argument holds and can explain preference for off-manifold robustness over te-manifold robustness.
Argument 2: Robust linear models are off-mmanifol robust. It is well-known result in machine learning (from, for example the representer theorem) that the linear analogue of gradient norm regularization, i.e., weight decay causes model weights to lie in the linear span of the data. In other words, given a linear model f(x) w x, its mput-gradien are the weights fxf(x)
w, and when trained with the objective L Fxx (f(x) - yxx)) 2 + A||w|| it follows that the weights have the following property: = E a,X,, i.e., the weights lie in the span of the data. In particular, if the data lies on a linear subspace, then so do the weights. Robust linear models are also infinitely off-manifold robust: for any perturbation uoft orthogonal to the data subspace, (x + unft) w thus they are completely robust to off-manifold perturbations.
In addition, if we assume that there are input co-ordinates x; that are uncorrelated with the output label, then w,X, is also uncorrelated with the label. Thus the only way to minimize the mean-squared error is to set w; 0 (i.c., solution which sets w, 0 has strictly better -quared error than one that docsn't '),, in which case the weights lie in the signal subspace, which consists of the subspace of all features correlated with the label. This shows that notions of signal-distractor decomposition transfer to the case of linear models.
4 Experimental Evaluation
In this section, we conduct extensive empirical analysis confirm our theoretical analyses and additional hypotheses. We first demonstrate that robust models exhibit relative off- menifold robustness (Section 4.1. We then show that off-manifold robustness correlates with the perceptual alignment of gradients (Section 4.2). Finally, we show that robust models exhibit a signal-distractor decomposition, that is they are relatively robust to noise on a distractor rather than the signal (Section (43). Below we detail experimental setup. Any additional details can be found in the Supplementary material.
7

---

ImageNct
ImageNet-6445 64x664
0.70
0:65
0.60
0.4
0.85
00555
Accuracy
0.50-
0:50
Accuracy
0.3
0.5
0:45
0.2
0.55
0:40
0.4
0.50
0.35
0.1
0.3
0:30
0.45
0.01
0.03
0:05
0.1 0.25
0.5
10
10-1
10"
10
102
10a
Adversarial Perturbation Budget (Epsilon)
Adversarial Perturbation Budget (Epsilon)
Figure 4: The input gradients of robust models trained with projected gradient descent on ImageNet and Imagenet-64x64 are perceptually similar to the score of the probability distribution, as measured by the LPIPS metric. On Imagenet-64x64, we also trained excessively robust models.
Data sets and Robust Models. We use CIFAR-10 [25]. ImageNet and ImageNet-64 (26), and an
MNIST dataset (27] with distractor, inspired by [I]. We train robust Resnet18 models (28) with (i) gradient norm regularization, (ii) randomized smoothing, (iii) a smoothness penalty, and (iv) Ig-adversarial robust training with projected gradient descent. The respective loss functions are given in the Supplementary material. On ImageNet, we use pre-trained robust models from [29].
Measuring On- and Off-Manifold Robustness. We measure on- and off-manifold robustness by perturbing data points with on- and off-manifold noise (Section B.T. I. For this, we estimate the tangent 3.11
space of the data manifold with auto-encoder, similar to 22 30 21. We then draw random noise vector and project it onto the tangent space. Perturbation of the input the tangent direction is used to measure on-manifold robustness. Perturbation of the input in the orthogonal direction is used to measure off-manifold robustness. To measure the change in the output of a classifier with C classes, we compute the L2 Lo-norm [[f(z) - f(x + u)ll2.
Measuring Perceptual Alignment. To estimate the perceptual alignment of model gradients, we would ideally compare them with the gradients of the Bayes optimal classifier. Since we do not have access to the Bayes optimal model's gradients VxP(y x). we use the score Vx log p(x y) as proxy, as both lie on the same data manifold. Given the gradient of the robust model and the score, we use the Learned Perceptual Image Patch Similarity (LPIPS) metric BT to measure the perceptual similarity between the two. The LPIPS metric computes the similarity between the activations of an AlexNet 32] and has been shown to match human perception well (31). In order to estimate the score Vx log p(x y). we make use of the diffusion-based generative models from Karras et al. (TO). Concretely, if D(x, or) is a denoiser function for the noisy probability distribution p(x, σ)) (compare Section 3.2 then the score is given by Vx loggpxx p(x, σ)) (D(x, a) x)/o² [10] Equation 3]. We use noise levels a 0.5 and a = 1.2 on CIFAR- 10 and ImageNet-64, respectively.
4.1 Evaluating On- vs. Off-manifold Robustness of Models
We measure the on- and off-manifold robustness of different Resnet18 models on CIFAR-10, using the procedure described above. We measure how much the model output changes in response an input perturbation of a fixed size (results for different perturbation sizes can be found in Supplement Figure D). The models were trained with three different robustness objectives and various levels of robustness. The results are depicted in the top row of Figure 3 Larger values in the plots correspond to larger perturbation in the output, that is less robustness. For little to no regularization (the left end of the plots), the models are less robust to random changes off- than to random changes on- the data manifold (the red curves lie above the green curves). Increasing the amount of robustness makes the models increasingly off-manifold robust (the red curves decrease monotonically). At the same time, the robustness objectives do not affect the on- mo-manifold robustness of the model (the green curves stay roughly constant). This means that the robust models become relatively off- robust (Definition [ At some point, the robustness objectives also start to affect the on-manifold behavior of the model, so that the models become increasingly on- on-manifold robust (the green curves start to fall). As can seen by a comparison with the accuracy curves in the bottom row of Figure 3 increasing manifold robustness mirrors steep fall in the accuracy of the trained models (the
8

---

Image
Score
Model Gradients
0.25
944
0:5
0.1
0:7 E4T
944
Weakly Robust
Bayes-Aligned Robust
Excessively Robust
Figure 6: The figure depicts the input gradients of models belonging different regimes of robustness. The numbers above the images indicate the perceptual similarity with the score, as measured by the LPIPS metric. Top row: ImageNNet-44x64. Bottom row: CIFAR-10. Weakly robust models are accurate but not off- offifoaifold robust. Bayes-aligned robust models are accurate and off- off-manifold robust. These are exactly the models that have perceptually aligned gradients. Excessively robust models are excessively on- on-manifold robust which makes them inaccurate. Best viewed in digital format.
green and blue curves fall in tandem). Remarkably, these results are consistent across the different types of regularization.
4.2 Evaluating the Perceptual Alignment of Robust Model Gradients
We now show that the input gradients of an intermediate regime of accurate and relatively off- manifold robust models (" B"Bayes-aligned robust models") are perceptually aligned, whereas the input gradients of weakly robust and excessively robust models are not. As discussed above, we measure the perceptual similarity of input gradients with the score of the probability distribution. The bottom row of Figure 3 depicts our results on CIFAR-10. For all three robustness objectives, the perceptual similarity of input gradients with the score, as measured by the LPIPS metric, gradually increases with robustness (the orange curves gradually increase). The perceptual similarity then peaks for an intermediate amount of robustness, after which begins to decrease. Figure 4 depicts our results on ImageNet and maageNet-64x64 Again, the perceptual similarity of input gradients with the score gradually increases with robustness. On ImageNet-644) 64x64, we also trained excessively robust models that exhibit decline both in accuracy and perceptual alignment.
To gain intuition for these results, Figure as well as Supplementary Figures 10 15
provide visualization of model gradients. In particular, Figure oo confirrms that the model gradients belonging to the left and right ends of the curves in Figure are indeed not perceptually aligned, whereas the model gradients around the peak (depicted in the middle columns of Figure 6) are indeed perceptually similar to the score.
While we use the perceptual similarity of input gradi- ents with the score as a useful proxy for the percep- tual alignment of input gradients, we note that this approach has theoretical foundation in the energy- based perspective on discriminative classifiers [33] In
Relative Noise Robustness (MNIST)
Resnet18
Robust Resnet18
particular, Srinivas and Fleuret (20). Zhu et al. 34 have suggested that the input gradients of softmax- based discriminative classifiers could be related to the score of the probability distribution. To the best of our knowledge, our work is the first to quantitatively compare the input gradients of robust models with independent estimates of the score.
10-7
10a
20'
Noise Level
Figure 5: Robust Models are relatively ro- bust to noise distractor.
4.3 Evaluating Signal vs. Distractor Robustness for Robust Models
We now show that robust models are relatively robust to noise on a distractor: Since a distractor is by definition not part of the signal manifold (Section 3.2) this serves evidence that the input

---

gradients of robust models are aligned with the signal manifold. In order to control the signal and the distractor, we create variant of the MNIST dataset where we artificially add the letter "A" as distractor. This construction is inspired by Shah et al. [II] for details see Supplement Figure T6 Because we know the signal and the distractor by design, we can add noise to only the signal or only the distractor and then measure the robustness of different models towards either type of noise. We call the ratio between these two robustness values the relative noise robustness of the model. Figure 5 depicts the relative noise robustness both for standard- and adversarially robust Resnet 88. From Figure we see that the standard model is already more robust to noise on the distractor. The robust model, however, is relatively much more robust to noise on the distractor. Since distractor noise is by definition off off-manifold robust w.r.t. the signal manifold, this result serves as evidence for Hypothesis 2. Morcover, the input gradients of the robust model (depicted in the supplement) are perceptually aligned. Hence, this experiment also serves as evidence for Hypothesis 1.
5
Discussion
Three Regimes of Robustness. Our experimental results show that different robust training methods show similar trends in terms of how they achieve robustness. For small levels of robustness regu- larization, we observe that the classifiers sensitivity to off-mmnifold perturbations slowly decreases (weak robustness), eventually falling below the on-manifold sensitivity, satisfying our key property of (relative) off-manifold robustness, as well as alignment with the Bayes classifier (Bayes-aligned ro- bustness). Excessive regularization causes models become insensitive to on-manifold perturbations, which often corresponds to a sharp drop in accuracy (excessive robustness).
The observation that robust training consists of different regimes (weak-, Bayes-aligned-, and excessive robustness) calls us to rethink standard robustness objectives and benchmarks [35), which do not distinguish on- and off-manifole robustness. An important guiding principle here can be not to exceed the robustness of the Bayes optimal classifier.
The Limits of Gradient-based Model Explanations. While Shah et al. [1]] find experimentally that gradients of robust models highlight discriminative input regions, we ground this discussion in the signal-distractor decomposition. Indeed, as the gradients of the Bayes optimal classifier are meaningful in that they lie tangent to the signal manifold, we can also expect gradients of robust models to mimic this property. Here the zero gradient values indicates the distractor distribution. However, if meaningful distal-d decomposition does not exist for dataset, i.e., the data and the signal distribution are identical, then we cannot expect gradients even the gradients of the Bayes optimal classifier to be "interpretable" in the sense that they highlight all the input features as the entire data is the signal.
Limitations. First, while our experimental evidence strongly indicates support for the hypothesis that fff-manifold robustness emerges from robust training, we do not provide a rigorous theoretical explanation for this. We suspect that it may be possible to make progress on formalizing Argument I in Section 3.3 upon making suitable simplifying assumptions about the model class. Second, we only approximately capture the alignment to the Bayes optimal classifier using the score-gres greddeerdints from a SOTA diffusion model as proxy, as we do not have access to a better proxy for a Bayes optimal classifier.
Acknowledgements
The authors would like thank Maximilian Müller, Yannie Neuhaus, Usha Bhalla and Ulrike von Luxburg for helpful comments. Sebastian Bordt is supported by the German Research Foundation through the Cluster of Excellence "Machine Learning New Perspectives for Science" (EXC 2064/1 number 390727645) and the International Max Planck Research School for Intelligent Systems (IMPRS-IS). Suraj Srinivas and Hima Lakkaraju are supported in part by the NSF awards 11S-2008461, IIS-2040989, IIS-2238714, 4, and research awards from Google, JP Morgan, Amazon, Harvard Data Science Initiative, and the Digital, Data, and Design (D$) Institute Harvard. The views expressed here are those of the authors and do not reflect the official policy or position of the funding agencies.
10

---

References
[1] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. In International Conference Learning Representations, 2019.
[2] Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with single (robust) classifier. In Advances in Neural Information Processing Systems, 2019.
[3] Gunjan Aggarwal, Abhishek Sinha, Nupur Kumari, and Mayank Singh. On the benefits of models with perceptually--aigee gradients. In Workshop at International Conference on Learning Representations, 2020.
[4] Bahjat Kawar, Roy Ganz, and Michael Elad. Enhancing diffusion-based image synthesis with robust classifier guidance. Transactions on Machine Learning Research, 2023.
[5] Tsachi Blau, Roy Ganz, Chaim Baskin, Michael Elad, and Alex Bronstein. Classifier robustness enhancement via test-time transformation. arXiv preprint arXiv:2303.15500 2023.
[6] Roy Ganz, Bahjat Kawar, and Michael Elad. Do perceptually aligned gradients imply adversarial robustness? In International Conference on Machine Learning. 2023.
[7] Simran Kaur, Jeremy Cohen, and Zachary C Lipton. Are perceptuall--llgneed gradients general property of robust classifiers? Workshop at Advances in Neural Information Processing Systems, 2019.
[8] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[9] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, 2019.
[10] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of iiitusion-based generative models. Advances in Neural Information Processing Systems, 2022.
[11] Harshay Shah, Prateek Jain, and Pranceth Netrapalli, Do input gradients highlight discriminative features? Advances Neural Information Processing Systems, 2021.
[12] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv preprint arXiv: 0ee::/006033222 2017.
[13] Chris M Bishop. Training with noise equivalent tikhonov regularization. Neural computa- lion, 1995.
[14] Harris Drucker and Yann Le Cun. Improving generalization performance using double back- propagation. IEEE transactions on neural networks, 1992.
[15] Suraj Srinivas and François Fleuret. Knowledge transfer with jacobian matching. In Interna- tional Conference on Machine Learning, 2018.
[16] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. 2014.
[17] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, In International Conference on Machine Learning. 2017.
[18] Tessa Han, Suraj Srinivas, and Himabindu Lakkaraju. Which explanation should choose? function approximation perspective to characterizing post hoc explanations. In Advances in Neural Information Processing Systems, 2022.
[19] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, 2018.

---

[20] Suraj Srinivas and Francois Fleuret. Rethinking the role of gradient- attribution methods for model interpretability. In International Conference on Learning Representations, 2021.
[21] Sebastian Bordt, Uddeshya Upadhyay, Zeynep Akata, and Ulrike von Luxburg. The manifold hypothesis for gradient-basee based explanations. In Conference Computer Vision and Puttern Recognition (CVPR) Workshops, 2023.
[22] Hang Shao, Abhishek Kumar, and Thomas Fletcher. The riemannian geometry of deep generative models. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018.
[23] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 2011.
[24] Neil Jethani, Mukund Sudarshan, Yindalon Aphinyanaphongs. and Rajesh Ranganath. Have we learned to explain?: How interpretability methods can learn to encode predictions in their interpretations. In International Conference on Artificial Intelligence and Statistics, 2021.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. University of Toronto, 2009.
[26] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and patlern recognition, 2009.
[27] Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2012.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[29] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Alcksander Madry. Do ad- versarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 2020.
[30] Christopher Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus-Robert Müller, and Pan Kessel. Fairwashing explanations with off-manifold detergent. In International Conference on Machine Learning.
[31] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, 2018.
[32] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012.
[33] Will Grathwohl, Kuan-Chieh Wang, Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly energy based model and you should treat it like one. In International Conference on Learning Representations, 2020.
[34] Yao Zhu, Jiacheng Ma, Jiacheng Sun, Zewei Chen, Rongxin Jiang. Yaowu Chen, and Zhenguo Li. Towards understanding the generative capability of adversarially robust classifiers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7728-7737, 2021.
[35] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. In Workshop at Advances in Neural Information Processing Systems, 2020.
12

---

A
Additional Proofs
Theorem A.1 (Equivalence between off-manifold robustness and on on-manifold alignment). A function R - R exhibits on-manifold gradient ali alignment if and only if it is off off-manifold robust wrr normal noise u ~ ((0, f + (with p1 = P2))
Proof. We proceed by observing that can decompose the input-gradient into on-manifold and off- manifold components by projecting onto the tangent space and its orthogonal component respectively, Le., (x) = PxV fx) Py Vxf(x).
We also observe that we can write the gradient norm terms of an expected dot product, i.e.,
o 2)) (Vx (x) u)² z (x) B(uu B(uuJ)V Vx = [x)x(x)]
Using these facts we can compute the norm of the off-manifold component as follows,
- [x)]
Vx)))(
Vxf(x)]
xx)|21)
On-manifold gradient alignment
Eia-~(00 ( (Vxf(x) u
5 Cov(uon) PP
1 Bu-N(0, Vxf(xx) u)²
a²
(n.r~~(0,0- (f + Uoll) f(x))²
lim
a o-+0
(f + - f(x))²
Off-manifold robustness
The second line is obtained by using the fact above regarding re- re-writing the gradient norm in terms of the expected dot product, and the final line obtained by using a first order Taylor expansion, which is exact in the limit of small sigma. From the equality of first and last terms, we have that the on-manifold gradient alignment <> the off-manifold robustness.
Theorem A.2. The input-gru gradients of Bayes optimal classifiers on the signal manifold <> -ayes optimal classifiers are relative off-manifold robust.
Proof. From definition 3 it is clear that given a classification problem, there exists a single distractor distribution d(x). Now, take gradients of log probabilities of the Bayes optimal classifiers, which results in:
Vx logp(y = - = Vx logp(x y y= - Σ ply = j xx)x logp(x 1y
We notice first that the vectors Vx lox y) all lie tangent the data manifold by definition, as this data generating process p(x y) itself defines the data manifold. As Vx logp(y x) a linear combination of the class-condiiional generative model gradients, follows that the imput-gradien of the Bayes optimal model also lie tangent to the data manifold. Now. like any vector on the tangent space at x, can be decomposed into signal and distractor components. Computing the distractor, we find that
Vx logp(y
x) ② m² = d(x) ΣP(== = x))(x) = 0
This happens because the distractor is independent of the label. thus the distractor component is zero, and the aaput-gradient of the Bayes optimal model lies entirely on the signal manifold. From Theorem A. it follows that when model gradients lie manifold, it is also off-manifold robust wrt that manifold.
13

---

B Experimental Details
B.1 Robust Training Objectives
We consider the following robust training objectives, where ((x, y) denotes the cross-entropy loss function.
1. Gradient norm regularization: 1(f(z), v) + xx)x with regularization constant À.
2. A smoothness penalty: 1(f(z)) y) + XE,-N((.03) If + + - f()) with a fixed noise level o² and a varying regularization constant A.
3. Randomized Smoothing: E.-.v(..=))(((( + c). with a noise level o².
4. Adversarial Robust Training: (z) y) where arg max B. ((f()), v) and i was obtained from the e-ball around using projected gradient descent.
B.2 Training Details
On CIFAR-10, trained Resnet18 models for 200 epochs with initial learning rate of 0.025.
When training with gradient norm regularization or the smoothness penalty and large regularization constants reduced the learning rate proportional to the increase in the regularization constant. After 150 and 175 epochs, we decayed the learning rate by factor of 10.
On ImageNet-64x64, we trained Resnet18 8 models for 90 epochs with batch size of 4096 and an initial learning rate of 0.1 that was decayed after and 60 epochs, respectively. We used the same parameters for projected gradient descent (PGD) as in 22)] that is we took 3 steps with step size of 23/3.
On the MNIST dataset with distractor, we trained Resnet18 model for epochs with an initial learning rate of 0.1 that was decayed after and 6 epochs. respectively. We also trained an l2- adversarially robust Resenet18 with projected gradient descent (PGD). We randomly chose the perturbation budget c E {1, 4, 8) and took 10 steps with a step size of 2.5c/1..
B.3 Diffusion Models
On CIFAR-10, use the unconditional diffusion model edm- www.frroa3fa.....
For comparison, Figure Supplement Figure 17 17 presents the results with the conditional diffusion model cdm-ciffar11-32232-cottt cdmn-imagenet--66664-comm
the conditional diffusion model All models are available at withtts:/// /ttps://gtn thub. com aVbs The noise levels 0.5 and 1.2 were chosen maximize the perceptual quality of the estimate of the score obtained from the diffusion model before running the experiments with the LPIPS metric.
On 644464,t-66464,,
edm
B.4
Conditional- and Unconditional Diffusion Models
Based theory, there classifiers. The first to take the input gradient with respect to single class and compare it to conditional diffusion model. The second is to sum the input gradients of all the classes and compare to an unconditional diffusion model (i.c., computing marginal probabilitics). The main paper present the first approach ImageNet and ImageNet-64x-44 and the second approach CIFAR- 10. Supplement Figure 17 presents the first approach on CIFAR-10. We observe that there no qualitative difference, i.e. both approaches give the same result.
principled ways evaluate diffusion models and discriminative
B.5 Model Gradients
With the unconditional diffusion model, we sum the input gradients across all classes. With the conditional diffusion model, we consider the input gradient with respect to the predicted class. We consider input gradients before the softmax (20).
14

---

Figure Left: Images from CIFARIO. Middle: Random perturbations on the data manifold. Right:
Random perturbations off the data manifold.
B.6 CIFAR-10 Autoencoder
We use https: httt:p///ggtttto..m.mc/c/cttttte www.m///eeeeeetcccccceeeeeeeeeee __tetakk_________ to train an autoeoncoder on CIFAR-I0 10 with a latent dimension & = 128. We use default architecture and training schedule. We then use the autoencoder to estimate, each data point, 128-dimensional tangent space. Figure] depicts random directions within the estimated tangent spaces.
B.7 Pre-Trained Robust Models on ImageNet
On ImageNet, use the pre-trained robust Resnetl8 models form https: com/ Aacrraoff/robas----deee--ttrrrre http://www.....o.oeoce/a//aaaaaa
load these models, we use the robustness library
B.8
Estimating the Score on ImageNet
We estimate the score on ImageNet using the diffusion model for ImageNet-64xx6. To estimate the score, we simply down-scale an image to 64x64.
B.9 MNIST with a Distractor
The MNIST data set with distractor is inspired by (L]. The data consists of gray-scale images of size 56x28. Every image contains a single MNIST digit and the distractor. We choose the fixed letter "" as the distractor. On every image, we randomly place the distractor top below the MNIST digit. In order to estimate the relative noise robustness, we separately add different levels of noise the signal or distractor. Figure [6 depicts images and models gradients on this data set.
I6
B.10 The LPIPS metric
The LPIPS metric measures the perceptual similarity between two different images. The metric itself corresponds loss, meaning that lower values correspond more similar images [31]. The figures in the main paper depict I-LPIPS, that is higher values correspond to more similar images.
C Additional Plots
The figures below depict the model gradients of different types of models, ranging from weakly robust to excessively robust. The figures depict the relationship between model gradients and the score qualitatively. This complements the quantitative results in the main paper.
15

---

Image
Perturbation Nor 1.4
Perturatioo Norm 2.6
Perturiation Norm 4.8
Pesurbation
Figure 8: Perturbations of different size, measured by the 12-norm of the perturbation. The Figure depicts the original images with random perturbations of the given L2-norm.
Gradent (0s00nt
Randoomeed Smoothhng
Smoothoess
0.100
Otiiaiic Sveturerie
0.15
0.10
0.000
9.05
0.05
0.000
10th
2014
SPP
10
10
10"d
12"
10"4
107
10
10"h
10'
Perturbation norm 1.4
Grodlont (CRSAR11)
Protecter Gradient Descent (CIFAR
Sandombeed Smoothing (CIF5810)
Smootinegt Peralty (CIFAR
0.4
Ne Buld aaiiker
L2.Norn
0.10
10
Regularization
10nt
10M
10
0.0
Adv. Perturbation Busqee (epsiion)
0.0
Noisa Level
0.00
20
13th
(b) Perturbation norm Le 2.6
Pro ected Gradtent Despent (CFAR-10 (0)
Nandomized Smoothing (CBAR-19)
Smoothoose Penaity (00
0.4
0.15
8.75
07
025
0.00
Regulartation Constant
13th
12"
30
12
10
8:00
0.0
Nobse Level
(c) Perturbation norm L = 4.8
Projected Descent (CFA4-11))
Randombbed Smoothing (CIF5R10)
Smocthness Penaly (CIFAR-10)
1.3
retutaation
3.5
12-
0.5
0.25
8.0
Regularization Constant
10th 10 10
6.
Neise Lerval
Regultrcctiinn Constant
(d) Perturbation norm Tog 8.9
Figure 9: On- and off off-manifold robustness of Resnet18 models trained with different objectives on CIFAR-IO (larger values correspond less robustness). The Figure depicts the on- and off-manifold robustness for perturbations of different size, as measured by the Lo-norm of the pertubation. Compare Figure in the main paper.
16

---

Increasing Robustness
Figure 10: The input gradients of different models trained with gradient norm regularization CIFAR-10. The top rows depict the image, the score, and the input gradients of unrobust models. The middle rows depict the perceptually aligned input gradients of robust models. The bottom rows depict the input gradients of excessively robust models. Best viewed digital format.
17

---

Increasing Robustness
Figure 11: The input gradients of different models trained with projected gradient descent CIFAR-10. The top rows depict the image, the score, and the input gradients of unrobust models. The lower middle rows depict the perceptually aligned input gradients of robust models. The bottom rows depict the input gradients of excessively robust models. Best viewed digital format.
18

---

Increasing Robustness
The input gradients of different models trained with smoothness penalty CIFAR-10. The top rows depict the image, the score, and the input gradients unrobust models. The middle rows depict the perceptually aligned input gradients of robust models. The bottom rows depict the input gradients of excessively robust models. Best viewed in digital format.
Figure 12:
19

---

Robustness
Increasing
Figure 13: The input gradients of different models trained with randomized smoothing CIFAR-
10. The top rows depict the image, the score, and the input gradicnts of unrobust models. The middle rows depict the perceptually aligned input gradients of robust models. The bottom rows depict the input gradients of excessively robust models. Best viewed in digital format.
20

---

Increasing Robustness
Figure 14: The input gradients of different models trained with projected gradient descent ImageNet-64x64. The top rows depict the image, the score, and the input gradients of unrobust models. The middle rows depict the perceptually aligned input gradients of robust models. The bottom rows depict the input gradients of excessively robust models. Best viewed digital format.
21

---

Increasing Robustness
Figure 15: The input gradients of different models trained with projected gradient descent on ImageNet. The models are from 29]. The top rows depict the image, the score, and the input gradients of unrobust models. The bottom rows depict the perceptually aligned input gradients of robust models. Best viewed in digital format.
22

---

A
2
/
A
A
/
4
9
6
A
7
A
A
O
4
A
A
A
A
9
(a) Images from the data set.
A
2
/
A
A
4
6
6
A
7
A
A
O
4
A
A
A
A
9
(b) Noise on the signal.
A
2
/
A
A
/
4
9
6
A
7
A
A
O
4
A
A
A
A
9
Noise on the distractor.
(d) Input gradients of a Resnet1 88
(c) Input gradients of an adversarially robust R8snet18,
Figure 16: The MNIST dataset with distractor used to create Figure in the main paper.
23

---

Gradient Norm (CIFAR-10)
Projected Gradient Descent (CIFAR-10)
Randomized Smoothing (CIFAR-10)
0.85
Accuracy
0.75
0.80
1-LPIPS
0.50
0.75
0.4
0.25
10:
10
10
10a
10-4
10
10 10a
10
10
10-1
0.2
Regularization Constant
Adv. Perturbation Budiet (Epsilon)
Naise
Figure 17: Measuring the perceptual alignment between the input gradients of Resnet18 18 models and the score as estimated with a conditional diffusion model. Compare Supplement Section B.4 and Figure in in the main paper. The Figure in the main paper depicts results with unconditional diffusion model.
24


Reviews:
[{"review_id": "pMIeCFzciO", "reviewer": "Reviewer_nd2c", "summary": "The paper titled \"Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness\" investigates the phenomenon of perceptually-aligned gradients (PAGs) in robust computer vision models. PAGs refer to the alignment of model gradients with human perception, enabling these models to exhibit generative capabilities such as image generation, denoising, and in-painting. The paper aims to explain the underlying mechanisms behind PAGs and provides insights into the relationship between off-manifold robustness, gradient alignment, and model accuracy.\n\nThe paper's contributions can be summarized as follows:\n\n1. Theoretical Explanation: The paper presents a theoretical explanation of PAGs through the concept of off-manifold robustness. It demonstrates that the alignment of gradients with the data manifold is equivalent to off-manifold robustness. It also introduces the distinction between the data manifold and the signal manifold, which helps in understanding the input gradients of robust discriminative models.\n\n2. Connection to Bayes Optimal Models: The paper establishes a connection between Bayes optimal models and off-manifold robustness. It shows that Bayes optimal models achieve both off-manifold robustness and on-manifold gradient alignment. The input gradients of Bayes optimal classifiers lie on the signal manifold, indicating their perceptual alignment.\n\n3. Empirical Analysis: Extensive empirical analysis is conducted to validate the theoretical findings. Robust models trained with various techniques are evaluated on different datasets. The experiments confirm the relative off-manifold robustness of robust models, the correlation between off-manifold robustness and perceptual alignment, and the presence of signal-distractor decomposition in robust models.\n\nThe paper's main contribution lies in providing a theoretical framework and empirical evidence for understanding the mechanisms behind PAGs in robust computer vision models. It sheds light on the importance of off-manifold robustness and its connection to gradient alignment and model accuracy. The findings have implications for developing more explainable and generative models in computer vision tasks. By identifying different regimes of robustness, the paper also calls for rethinking standard robustness objectives and benchmarks.\n\nOverall, the paper advances our understanding of the properties and behavior of robust models, paving the way for further research in improving model interpretability and generative capabilities in the field of computer vision.", "questions": "Clarification on the choice of perturbations: Can the authors explain the rationale behind the specific choice of on- and off-manifold perturbations? How representative are these perturbations of real-world scenarios? Are there other types of perturbations that could be considered in future research?\n\nExtension to non-linear models: The paper primarily focuses on linear models and their relationship to off-manifold robustness. Can the authors discuss how their findings and hypotheses can be extended to non-linear models? Are there any unique considerations or challenges in analyzing non-linear models in this context?\n\n", "limitations": "- Limited exploration of perturbation space: The choice of on- and off-manifold perturbations in the experiments might not cover the entire space of possible perturbations. There could be a wide range of perturbations that lie between the two extremes. Exploring a more comprehensive set of perturbations and their effects on on- and off-manifold robustness could enhance the depth of the analysis. \n- Evaluation metrics: The paper primarily focuses on measuring robustness based on the change in model output. While this is a common approach, it may not capture all aspects of robustness. Considering additional evaluation metrics, such as adversarial examples or other robustness measures, could provide a more comprehensive understanding of model behavior.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The strength of the paper lies in its combination of theoretical analysis and empirical evaluation. It provides a comprehensive examination of the phenomenon of perceptually-aligned gradients (PAGs) in robust computer vision models. By presenting a theoretical framework based on off-manifold robustness and its connection to gradient alignment, the paper offers a solid foundation for understanding the underlying mechanisms behind PAGs.\n\nFurthermore, the empirical analysis conducted in the paper strengthens its findings. The authors perform extensive experiments using different datasets and robust training techniques, validating their theoretical explanations and hypotheses. The empirical results consistently support the theoretical claims, providing evidence for the presence of off-manifold robustness, gradient alignment, and signal-distractor decomposition in robust models.\n\nThe paper's contribution also extends to the exploration of various aspects related to PAGs, such as the connection to Bayes optimal models, the trade-off between on- and off-manifold robustness, and the correlation between off-manifold robustness and perceptual alignment. These insights broaden our understanding of the behavior and properties of robust models in computer vision tasks.\n\nOverall, the strength of the paper lies in its rigorous analysis, clear explanations, and compelling empirical evidence, making a significant contribution to the field of computer vision and model interpretability.", "weaknesses": "One potential weakness of the paper is that the evaluation of on- and off-manifold perturbations does not cover the entire space of perturbations. There is a significant portion of perturbations that fall in-between the two categories, which are not thoroughly explored in the current research. It would be valuable for future work to address these intermediate cases and provide a more comprehensive understanding of the model's behavior in those scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and encouraging review! We are glad that you found our analyses rigorous and our explanations clear. We answer specific questions below.\n\n1.*“One potential weakness of the paper is that the evaluation of on- and off-manifold perturbations does not cover the entire space of perturbations. There is a significant portion of perturbations that fall in-between the two categories, which are not thoroughly explored in the current research.”* *“Clarification on the choice of perturbations: Can the authors explain the rationale behind the specific choice of on- and off-manifold perturbations?”* *“Limited exploration of perturbation space: The choice of on- and off-manifold perturbations in the experiments might not cover the entire space of possible perturbations.”*\n\nPlease note that by definition, every perturbation can be written as a linear combination of one on- and one off-manifold perturbation (since the tangent space is simply a linear subspace). This is also crucial to our experiments -- we create random Gaussian perturbations and then proceed to project them onto on- and off-manifold parts for analysis in Figure 2. Therefore, the results in our paper already incorporate the intermediate cases mentioned. In case we have misunderstood your questions, please let us know, we are happy to clarify.\n\n---\n\n\n2.*“Extension to non-linear models: The paper primarily focuses on linear models and their relationship to off-manifold robustness. Can the authors discuss how their findings and hypotheses can be extended to non-linear models? Are there any unique considerations or challenges in analyzing non-linear models in this context?”*\n\nWe believe this is a misunderstanding – both the theory and the experiments in our paper are for standard deep neural networks, that is, highly non-linear models. Except for argument 2, which discusses linear models, the rest of the paper is about non-linear neural network models. \n\n---\n\n3.*“Evaluation metrics: The paper primarily focuses on measuring robustness based on the change in model output. While this is a common approach, it may not capture all aspects of robustness. Considering additional evaluation metrics, such as adversarial examples or other robustness measures, could provide a more comprehensive understanding of model behavior.”*\n\nWe are interested in robustness insofar as it is related to the phenomenon of perceptual-aligned gradients. We provide additional results on adversarial noise in the one-page PDF, please see the global comment for details. \n\n"}, {"review_id": "o4grnjoVrH", "reviewer": "Reviewer_58Dj", "summary": "This paper explores the connection between manifold alignment, perceptually-aligned gradients, and model robustness, deriving a number of novel results that explain previously observed phenomena in the explainability and robustness literatures. Crucially the paper grounds its contributions in Bayes optimal classifiers, arguing that the gradients of such classifiers are aligned with a “signal manifold”. The paper leverages denoising autoencoders to estimate the data manifold and diffusion models to estimate the gradients of a Bayes optimal classifier. Using these quantities, the paper demonstrates a number of novel results, including a strong relationship between manifold alignment and model accuracy.", "questions": "1. Why is Theorem 2 stated with an if and only if? Don’t you prove that both statements are true for Bayes optimal classifiers?\n2. Would it be possible to add an adversarially-trained model for CIFAR-10?\n3. On line 228 you say “We elaborate on this argument in the Supplementary material.” Where?\n4. On line 255, does “$l_2$-adersarial robust training” mean PGD applied during training?\n5. In the legend for Figure 2, is “perturbation” really the right term? Isn’t the plot essentially showing the inverse of robustness? Why not use the relative robustness metric defined in Definition 1? Why is an $l_1$ divergence used here rather than the $l_2$ divergence used in Definiton 1? Can $\\rho_1$ here be connected to Brier score to provide a proper score function?\n6. Are error bars standard error?\n7. Do you have any comment on the point where the lines cross in Figure 2 and why there is a different trade off depending on the training objective?\n8. At line 274, how were these specific $\\sigma$ values chosen for each dataset to parameterize the diffusion model?", "limitations": "The acknowledged limitations throughout the paper and the supplemental material are superb (e.g. the statement “We provide two lines of argument in support for this hypothesis [Hypothesis 2]. Ultimately, however, our evidence is empirical”).", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper presents a timely and likely impactful theoretical contribution with implications not only for robust learning but also for explainability and generalization. The paper is very well organized and well-written. The introduction provides an excellent summary of the literature and current open questions. The paper disambiguates between three closely-related notions of “perceptually-aligned gradients” and provides a powerful taxonomy through the definition of Phenomena 1-3.\n\nDrawing on prior work, the paper introduces quantitative metrics to evaluate robustness and gradient alignment with respect to a data manifold. The theoretical contributions are somewhat geometrically simplistic, but powerful. Theorem 1 has been assumed intuitively in prior work, but this paper introduces the formalisms to quantify the connection between off-manifold robustness and on-manifold gradient alignment. Argument 1-2 outline very compelling circumstantial evidence for the broader Hypotheses and will hopefully lead to future theoretical developments.\n\nThe paper provides extensive experimental evidence, including useful error bars. Figure 2 provides gorgeous plots and the correlation between manifold alignment and model accuracy is striking.\n\nFinally, the paper makes a number of practical advances related to effective adversarial training. Figure 5 intuitively summarizes the paper’s crucial claims about three regimes of robust training.", "weaknesses": "* It does not seem right that the mask defining distractor vs signal features is constrained to be binary (Definition 3). This means the signal and distractor distributions are not invariant to linear transformations of $x$. Shouldn’t the only requirement be the statistical independence on line 163?\n* All experiments evaluating perceptual alignment used the LPIPS metric. Wouldn’t another metric, such as cosine distance, be appropriate. This would allow for distractors that aren’t defined by a binary mask.\n* Unfortunately (probably due to computational cost) different results are presented for CIFAR-10 and ImageNet, reducing the global consistency of the paper.\n* The experiment shown in Figure 4 is not well explained, would it be possible to explain the setup in a little more detail using previously-established notation?\n\nSmall issues:\n* The definition of Bayes optimal classifiers assumes an equal class distribution (i.e. uniform p(y)). This most likely will not be the case in most applications. Note that this does not invalidate any of the paper’s findings.\n* Type on line 200 “Bayes optimal models gradients”\n* Missing article on line 220: “there exists a trade-off between [the] cross-entropy loss and on-manifold robustness term”\n* Missing pronoun at line 351: “meaningful in that [they] lie tangent”\n* At line 354, “gradients is repeated, and do you mean the double negative “cannot be uninterpretable”?\n* Figures 4 and 5 are out of order.\n* The numbering of Theorems in the Supplemental material is off (the proofs continue as Theorem 3, 4 rather than 1, 2).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review! We're glad that you liked our paper overall, including the acknowledgement of our limitations. We respond to individual questions below.\n\n1.*“It does not seem right that the mask defining distractor vs signal features is constrained to be binary (Definition 3)”*\n\nWe agree that Definition 3 can be extended such that the signal and distractor are any orthogonal complements of each other, and need not be axis aligned to the inputs. The reason we define these in this manner is to explain the observations of Shah et al. (“Do input gradients highlight discriminative features?”, NeurIPS 2021) who observe that input gradients of robust models highlight discriminative features, which they define as input pixels that can be used to predict the label. Having said that, we agree that generalizing definition 3 in this manner makes the theory potentially more interesting and general. We will add a note about this in the paper.\n\n---\n\n2.*“All experiments evaluating perceptual alignment used the LPIPS metric. Wouldn’t another metric, such as cosine distance, be appropriate. This would allow for distractors that aren’t defined by a binary mask.”*\n\nSee our response 7 to reviewer aEc6 for the rationale behind the LPIPS metric. We are not sure that we understand the point of how the LPIPS metric does not allow for distractors that aren’t defined by a binary mask whereas the cosine distance does. We would be happy to clarify if the reviewer could elaborate on this.\n\n---\n \n3.*“Unfortunately (probably due to computational cost) different results are presented for CIFAR-10 and ImageNet, reducing the global consistency of the paper.” / “Would it be possible to add an adversarially-trained model for CIFAR-10?”*\n\nWe partly address this by providing results for an adversarially-trained model on CIFAR-10 in the additional pdf file. \n\n---\n\n4.*“The experiment shown in Figure 4 is not well explained, would it be possible to explain the setup in a little more detail using previously-established notation?”*\n\nCertainly! We apologize for the lack of detail, and we will rewrite this setup accordingly. Essentially, we create a variant of the MNIST dataset where the signal and distractor are known by design. We then proceed to train robust models and find that they are indeed more robust to the distractor as opposed to the signal. \n\n---\n\n5.*“Why is Theorem 2 stated with an if and only if?”*\n\nThe “if and only if” part of theorem 2 is indeed redundant, as we do prove that off-manifold robustness and on-manifold alignment are identical in theorem 1. We only write this again for clarity.\n\n---\n\n6.*“On line 228 you say “We elaborate on this argument in the Supplementary material.” Where?”*\n\nWe apologize for omitting this discussion in the supplementary due to oversight. The elaboration essentially consists of a slightly more rigorous treatment of the argument presented in the main paper – we can decompose any model f(x) into on-manifold and off-manifold components globally by projecting on the tangent space at each point. Thus in the decomposition in argument 1, the on-manifold objective applies purely on the on-manifold model and similarly to the off-manifold parts. Now, if the behavior of the model is purely independent on- and off-manifold, then the stationary points of the separate objectives applied to the two independent models imply off-manifold robustness of the final model.\n\n---\n\n7.*“On line 255, does “adversarial robust training” mean PGD applied during training?”*\n\nYes.\n\n---\n\n8.*“In the legend for Figure 2, is “perturbation” really the right term? Isn’t the plot essentially showing the inverse of robustness? Why not use the relative robustness metric defined in Definition 1? Why is an divergence used here rather than the divergence used in Definiton 1?\"*\n\nGreat point! The additional pdf file shows that our results are robust to the choice of divergence. We will modify Figure 2 to use the same divergence as in Definition 1. We will also exchange the term “perturbation” with “sensitivity”. \n\nRegarding plotting relative robustness vs plotting on- and off-manifold robustness separately, we plot separately to maximize clarity wrt to the underlying phenomenon. We believe relative robustness would simply scale both these curves by the overall robustness value. Nonetheless, we will add a note about this in the paper, and these to our supplement. \n\n---\n\n9.*“Are error bars standard error?”*\n\nThe error bars in Figure 2 depict the minimum- and maximum observed values when training 10 models with different random seeds (including different regularization terms for the randomized penalties). This will be clarified in the description of the figure. The reason these are min/max instead of standard deviations is that standard deviations across 10 observations might not be very meaningful, and re-training these models is computationally expensive.\n\n---\n\n10.*“Do you have any comment on the point where the lines cross in Figure 2 and why there is a different trade off depending on the training objective?”*\n\nGreat question! At this point, we don’t have any specific insights except for the general observation that while the overall behavior of different regularization objectives is comparable, these are still different training objectives that do lead to different optima. To given an example, while gradient norm regularization and randomized smoothing both lead to perceptually aligned gradients, a close qualitative comparison of the respective input gradients (Supplementary Figures 7 and 8) reveals that the input gradients resulting from randomized smoothing have a tendency to be perceptually more ‘smooth’.\n\n---\n\n11.*“At line 274, how were these specific values chosen for each dataset to parameterize the diffusion model?”*\n\nThese values were chosen to maximize the perceptual quality of the estimate of the score obtained from the diffusion model.\n"}, {"review_id": "Jfm6hoe2Au", "reviewer": "Reviewer_avWq", "summary": "The paper seeks to provide a condition that leads models to have gradients that are aligned with human perception, i.e. which highlight relevant features of the image while ignoring distractor features. The paper first proposes that for any manifold, if a model is robust to perturbations which are off-manifold, then its gradients will be aligned with the manifold. The paper then defines a \"signal\" manifold as a hypothetical manifold of features which relevant to the classification task, defined as the multiplication of the data with a masking function. The paper then states and proves that a Bayes optimal classifier is robust to perturbations perpendicular to this manifold, and hence its gradients lie on the signal manifold. The paper further hypothesizes that gradients on the signal manifold are perceptually aligned. An argument is then provided that models robust to adversarial perturbations which have similar off-manifold robustness as the Bayes optimal classifier have perceptually aligned gradients. Finally, the paper presents empirical evidence in support of their theoretical claims and hypotheses.  ", "questions": "1. How are on- and off-manifold perturbations generated exactly?\n2. Is there any way to provide more empirical evidence for Hypothesis 1?", "limitations": "The authors address their limitations in sections 3 and 4, where they describe how much of an approximation their empirical setup and theorems are.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an interesting hypothesis and designs a robust experimental methodology to validate it within the set of assumptions. \n2. The notion of weakly robust, optimally robust and overly robust models and their connection to PAGs is novel, and interesting. Comparing such classifiers with score functions is interesting in its own right, and can be a good tool in analysing models empirically.\n ", "weaknesses": "1. It is not clear from the experiments what the signal manifold is. Line 278 talks about perturbing about 10\\% of the input, but it would be good to clarify this further.\n2. Hypothesis 1 is never validated empirically in the paper, but the experiments build upon this hypothesis by comparing the perceptual similarity with the gradients of the score function.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review! We address specific concerns below.\n\n1.*“It is not clear from the experiments what the signal manifold is. Line 278 talks about perturbing about 10% of the input, but it would be good to clarify this further.”*\n\nThe concept of the signal manifold arises from the observation that the input gradients of discriminate models have a tendency to highlight parts of the image that are discriminative. This should be seen in contrast to the score, which models the entire data distribution (the data manifold), including class-irrelevant background information. The signal manifold is not explicitly known. The perturbations applied in this paper are with respect to the data manifold. In Line 278 we should have been more clear. When we measure how much the model output changes in response to a change in the input, we fix the $l_2$-norm of the perturbation that is applied to the input (just as we would do with adversarial perturbations). In our experiments, the $l_2$-norm of the perturbation is about 10% of the $l_2$-norm of the input image. We will clarify this in the paper.\n\n---\n\n2.*“Hypothesis 1 is never validated empirically in the paper, but the experiments build upon this hypothesis by comparing the perceptual similarity with the gradients of the score function.” “Is there any way to provide more empirical evidence for Hypothesis 1?”*\n\nOur results in Figure 2 provide partial evidence for Hypothesis 1. Here we observe that the off-manifold robustness of models increases in the top row to increasing robustness, and simultaneously we observe a corresponding increase in the LPIPS metric. This shows that perceptual alignment and lying on the data manifold go hand-in-hand. We also provide results specific to the signal manifold (which lies within the data manifold) in Figure 4.\n\n---\n\n3.*“How are on- and off-manifold perturbations generated exactly?”*\n\nWe estimate the tangent space of the data manifold using an auto-encoder. Then we draw a random normal vector and project it onto the tangent space. The part of the vector that lies in the tangent space is the on-manifold perturbation. The part of the vector that is orthogonal to the tangent space is the off-manifold perturbation. Line 257 in the main paper goes into more detail about this.\n"}, {"review_id": "07Y3beddry", "reviewer": "Reviewer_aEc6", "summary": "This paper studies Perceptually Aligned Gradients (PAGs), a phenomenon where the input gradients are semantically meaningful and aligned with human perception.\nWhile this trait gained research attention, we do not truly understand it.\nTo this end, the paper proposes an explanation via off-manifold robustness.\nNamely, they show empirically and theoretically that models that are off-manifold robust have perceptually aligned gradients for different robustification techniques.\nMoreover, they propose a quantitative way to assess PAG, while prior to this work, PAG was evaluated qualitatively only.\nIn addition, they identify three levels of robustness and analyze them in terms of PAG.", "questions": "* Is the score function being modeled in the measuring Perceptual Alignment paragraph for clean images or noisy images? From my understanding, diffusion networks model the noisy score function and not the clean one.\n\n* What is the rationale for measuring the LPIPS for gradients? It is usually used for images, and input-gradient distribution is significantly different. Why does it make sense to use this rather than cosine similarity or other closed-form metrics?\n\n* Figure 2 shows that there are points where models are good in off-manifold robustness before being on-manifold robust (for example, the left figure in the top row of Figure 2, around a noise level of 0.5). Can this analysis help us find robust models that have better robustness-accuracy tradeoffs?\n\n* Why are the results of Imagenet 64 so low?\n\nIn general, I find this paper interesting and would like to increase my score given the authors' response.\n", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Perceptually Aligned Gradients is a fascinating phenomenon, and despite gaining substantial research attention, we do not understand it. Shedding light on this trait is an interesting and important research goal. The paper empirically and theoretically verifies the relationship between PAG and off-manifold robustness. To strengthen their findings, they do so for different robustification techniques.\n\n* Decomposing the gradients of a model to on and off-manifold directions is very interesting thinking. Besides being theoretically logical, they also develop a way of doing so.\n\n* Excellent and clear writing. The authors have done a good job of providing motivation for this work. In addition, while several papers use this term to describe different behaviors, this paper formulates the definitions and makes an order in the various definitions.\n", "weaknesses": "* The connection between a conditional score function and PAG was made in [1]. Despite not being used for assessing PAG, they show that a classifier trained to mimic the score function possesses PAG. Thus, I do not find linking the two for assessing PAG very novel.\n\n* Overstatements and bit trivial claims - “In this work, we provide 7 a first explanation of PAGs” [abstract], but [4] offers an explanation of the generative capabilities of adversarial training using energy-based models. Additionally, I find some claims that the authors made efforts to explain quite straightforward. For example, robust models are off-manifold robust. We know robust models have PAG, meaning their gradients are aligned on the manifold. This means that the change of loss in off manifold direction is smaller than the on-manifold (because otherwise, the gradients weren’t aligned), making such models off-manifold robust.\n\n**Minor weaknesses and requests**\n\n* I find this important to report the LPIPS results in Figure 5 to demonstrate that the used metric indeed makes sense and aligns with the human evaluation of PAG.\n\n* Missing citations - I think that in order to motivate the significance of studying PAG for the general audience, it is important to include more works that study and utilize PAG. For instance, [2,3].\n\n[1] Do Perceptually Aligned Gradients Imply Robustness?\n\n[2] On the benefits of models with perceptually-aligned gradients.\n\n[3] BIGRoC: Boosting Image Generation via a Robust Classifier.\n\n[4] Towards Understanding the Generative Capability of Adversarially Robust Classifiers.\n\n\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your great review, and we're glad you found our paper interesting! We address specific comments below.\n\n1.*“The connection between a conditional score function and PAG was made in [1]. Despite not being used for assessing PAG, they show that a classifier trained to mimic the score function possesses PAG. Thus, I do not find linking the two for assessing PAG very novel.”*\n\nThe reviewer is right that score-functions have been used in the context of PAGs before [1], but to the best of our knowledge, we are the first to use score-functions to provide a quantitative metric to evaluate PAGs.\n\n---\n\n2.*“Overstatements and bit trivial claims - “In this work, we provide a first explanation of PAGs” [abstract], but [4] offers an explanation of the generative capabilities of adversarial training using energy-based models.\"*\n \nThanks for the reference! While we believe that the provided reference [4] is indeed a valuable contribution, it misses a critical component of the PAGs phenomenon, that gradients of robust models highlight only the discriminative components (see Phenomenon 1, line 38 in our paper) while ignoring the rest. However we agree that it can be a partial explanation for Phenomenon 2, i.e., generative capabilities, and hence we shall re-word our claims accordingly.   \n\n---\n\n3.*\"Additionally, I find some claims that the authors made efforts to explain quite straightforward. For example, robust models are off-manifold robust. We know robust models have PAG, meaning their gradients are aligned on the manifold. This means that the change of loss in off manifold direction is smaller than the on-manifold (because otherwise, the gradients weren’t aligned), making such models off-manifold robust.”*\n\nTo the best of our knowledge, it has not been well-established in prior works that PAGs imply gradients lie on-manifold (Hypothesis 2), and our paper takes a step in this direction. While theorem 1 is intuitive, it is critical to explaining PAGs as it helps link PAGs to model robustness, which is our main objective. In addition, theorem 1 only applies in very limited settings (under the limit of small Gaussian noise), and the contribution of our work is to show that this also holds for realistic noise levels and for real models.\n\n---\n\n4.*“I find this important to report the LPIPS results in Figure 5 to demonstrate that the used metric indeed makes sense and aligns with the human evaluation of PAG.”*\n\nGreat point. Please see the pdf file provided as part of the rebuttal. The reviewer can also compare Figure 2 with Supplement Figures 7-11.\n\n---\n\n5.*“Missing citations - I think that in order to motivate the significance of studying PAG for the general audience, it is important to include more works that study and utilize PAG. For instance, [2,3].”*\n\n[2] is already cited in the paper, thanks for pointers to [3,4]. We shall discuss applications of PAGs in the paper. \n\n---\n\n6.*\"Is the score function being modeled in the measuring Perceptual Alignment paragraph for clean images or noisy images? From my understanding, diffusion networks model the noisy score function and not the clean one.”*\n\nDiffusion models provide a family of score functions parametrized by the noise level sigma. In theory, noise training in these models ensures scores are well-defined everywhere. Once trained, the score-function is applicable to all points in the input with an appropriate sigma. We choose the parameter sigma to maximize the perceptual alignment of the score.\n\n---\n\n7.*“What is the rationale for measuring the LPIPS for gradients? It is usually used for images, and input-gradient distribution is significantly different. Why does it make sense to use this rather than cosine similarity or other closed-form metrics?”*\n\nWe choose the LPIPS metric because we find that it adequately quantifies the observed variation in the perceptual similarity between the score and the input gradients of differently robust models. Independently of the chosen metric, this variation is observed in the qualitative depictions in Supplement Figures 7-11. We agree with the reviewer that the LPIPS metric is not necessarily perfect and that other metrics could be considered. Empirically, however, we found that the LPIPS metric worked best. For example, we did initial experiments with the cosine similarity and found the results, while often qualitatively similar, to be less robust and less aligned with our human perception. A potential reason for this might be that the cosine similarity is rather sensitive to the fact that the score aligns with the data manifold, whereas model gradients align with the signal manifold.\n\n---\n\n8.*“Figure 2 shows that there are points where models are good in off-manifold robustness before being on-manifold robust (for example, the left figure in the top row of Figure 2, around a noise level of 0.5). Can this analysis help us find robust models that have better robustness-accuracy tradeoffs?”*\n\nGreat question! We would argue that an analysis of the on- and off-manifold robustness curves can help us identify models that are relatively accurate and also have perceptually-aligned gradients (or off-manifold robustness). Considering the example pointed out by the reviewer (noise level of 0.5), this corresponds to a model with relatively high accuracy and perceptually-aligned gradients. \n\nWe can also help find models with better robustness-accuracy tradeoffs in the sense that we identify that it is preferable for models to be only off-manifold robust as opposed to being robust on-manifold. Thus by choosing methods that prefer only off-manifold robustness, we may be able to achieve a better accuracy-robustness tradeoff. \n\n---\n\n9.*“Why are the results of Imagenet 64 so low?”*\n\nOn ImageNet-64x64, a performance drop is expected due to the significant downsampling of the images. We train the Resnet 18 models from scratch using standard training procedures without any significant hyperparameter optimization.\n"}, {"review_id": "KK3Ngw0a14", "reviewer": "Reviewer_eaFT", "summary": "This paper presents a theoretical characterization and some theorems supporting previous empirical finidings on perceptually aligned gradients (PAG) of classification neural network models. Specifically, it provides the first rigorous definitions for the previously qualitative definitions of \"PAG\", and provides the first attempt at a theoretical framework connecting PAG and model robustness, which has been observed empricially in previous work.", "questions": "- In the abstract, is noise augmentation and randomized smoothing the same thing? If so, I would suggest listing them as one item and not two.\n- Lines 272-274: Is the score-based model used here class-conditional? If so, it should be made clear by defining p(x|y) and not p(x) in the score function definitions. If not, further explanation is needed. Why would we characterize perceptual alignment using a class-unaware model? \"PAG\" usually refers to salient class-specific features.", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Very well-written and well-presented paper. Covers and taxonomizes the relevant literature very nicely.\n- Math and definitions are intuitive, relevant, and sound. I especially like the fact that the theorem assumes almost nothing on the model behavior or training objective.\n- Overall, this paper addresses a very important and relevant topic, characterizes a lot of themes in it, and provides a great theoretical explanation for it. This theoretical justification was notably missing from previous PAG literature.\n- Empirical evaluation metrics are carefully picked and well-thought-out. Comparing PAG to score-based models is especially relevant (this was first suggested in [6], I think a reference is due on lines 264-274).", "weaknesses": "- lines 106-107: noise can also have both on- and off- manifold elements (think of a linear combination of two on- and off- manifold noise vectors). Therefore, the \"otherwise\" on line 107 is incorrect.\n- Minor issue: Defnition 1 is missing a definition of x (for formality).\n- While the math (theorem 1) is nice, it does not consider adversarially chosen noise. It would be nice to mention that for normal noise with $\\sigma \\rightarrow 0$, this covers adversarial noise as well, but it should be made clear that more work is needed to specifically consider adversarial attacks.\n- Line 151: Paragraph title mentions difference between data and signal manifolds, but the paragraph does not discuss \"data manifold\". Typo?\n- The definition of signal vs distractor is too simplistic. Signals and distractors are not necessarily separated by a pixel mask only. The underlying signal manifold could lie on a linear or non-linear projection of x, not necessarily on the pixel space itself. This weakens the resulting math (theorem 2), unless the authors can rewrite their proofs w.r.t. a determinstic \"signal\" function that is generalized beyond simple point-wise masking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful review! We're glad you liked the paper and found the theory intuitive and relevant. We address your questions below.\n\n1.*“While the math (theorem 1) is nice, it does not consider adversarially chosen noise. It would be nice to mention that for normal noise with $\\sigma \\rightarrow 0$, this covers adversarial noise as well, but it should be made clear that more work is needed to specifically consider adversarial attacks.”*\n\nGood point! We think adversarial noise are definitely an important case, but slightly tricky to handle as it represents the worst-case perturbation and has measure zero. For example, it can still happen that models may be less robust off-manifold compared to on-manifold, but the only adversarial noise is on-manifold. Essentially, this can happen because adversarial noise is about misclassification, but our notions of robustness are about change in output value. But, note that this theory nonetheless holds for adversarially-trained models in practice, which have on-manifold gradients and hence off-manifold robustness in the average case. Please let us know in case you have more thoughts about this problem!\n\n---\n\n2.*“The definition of signal vs distractor is too simplistic. Signals and distractors are not necessarily separated by a pixel mask only. The underlying signal manifold could lie on a linear or non-linear projection of x, not necessarily on the pixel space itself. This weakens the resulting math (theorem 2), unless the authors can rewrite their proofs w.r.t. a determinstic \"signal\" function that is generalized beyond simple point-wise masking.”*\n\nWe agree that Definition 3 can be extended such that the signal and distractor are any pointwise orthogonal complements of each other, and need not be axis aligned to the input pixels. The reason we define these in this manner is to explain the observations of Shah et al. (“Do input gradients highlight discriminative features?”, NeurIPS 2021) who observe that input gradients of robust models highlight discriminative features, which they define as input pixels that can be used to predict the label, which serves as the basis for \"phenomenon 1\" in our paper. Having said that, we agree that generalizing definition 3 makes the theory potentially more interesting and general. We will add a note about this in the paper.\n\n---\n\n3.*“In the abstract, is noise augmentation and randomized smoothing the same thing? If so, I would suggest listing them as one item and not two.”*\n\nBy “noise augmentation” we mean the smoothness penalty detailed in Section C.1 in the Supplement, which is different from randomized smoothing. Essentially, this involves training models with an explicit regularization encouraging the model behaviour to be similar with and without noise, i.e., regularizing $ || f(x)  - f(x + \\epsilon) ||^2$ to be small, where $\\epsilon \\sim \\mathcal{N}(0, \\sigma^2)$. \n\n---\n\n4.*“Lines 272-274: Is the score-based model used here class-conditional? If so, it should be made clear by defining p(x|y) and not p(x) in the score function definitions. If not, further explanation is needed. Why would we characterize perceptual alignment using a class-unaware model? \"PAG\" usually refers to salient class-specific features.”*\n\nThe diffusion model is unconditional. Interestingly, we found little difference between the scores of class-conditional and unconditional diffusion models on natural images (comparing “cifar10-32x32-uncond-vp” and “cifar10‑32x32‑cond‑vp” from [10]). Empirically, our results are robust towards using conditional and unconditional diffusion models. \n\nThank you also for all the suggestions with the typos, we will make these corrections in the draft."}]

Author rebuttal:
We thank all reviewers for taking the time to review our paper and providing constructive feedback. We are encouraged by their positive assessment of the paper, and we are committed to incorporating reviewer feedback to further strengthen the paper. 

Based on the reviewers’ comments, we have conducted additional experiments and provide the following plots in a one-page PDF:
1. Results with adversarial training on CIFAR-10 (in response to a question by reviewer 58Dj). We provide plots in the format of Figure 2 in the main paper and visualizations of input gradients as in the Supplement. We trained with projected gradient descent (PGD) and a $l_2$-budget of varying size $\epsilon$, taking 10 steps of size $\alpha = 2 * \epsilon / 10$. We used our default learning rate schedule on CIFAR-10 and proportionally decreased the initial learning rate for very large perturbation budgets (compare Supplement B.2).
2. Robustness of our evaluation with respect to different divergence measures (in response to a question by reviewer 58Dj).
3. Adding the LPIPS metric to Figure 5 (in response to reviewer aEc6).


Meta-review: The reviewers were unanimously positive about this paper, noting that the paper addresses PAGs are an interesting but poorly understood phenomenon. This paper provides a theoretical explanation and extensive empirical validation thereof. Reviewers praised the clarity of exposition and mathematical rigour and clarity. As such I gladly recommend this paper for acceptance.
Decision: Accept (spotlight)