Title: Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness
Venue: NeurIPS 2023
Accepted: yes
Keywords: robustness

generative models

perceptually aligned gradients

bayes optimality

manifold hypothesis
Abstract: One of the remarkable properties of robust computer vision models is that their input-gradients are often aligned with human perception, referred to in the literature as perceptually-aligned gradients (PAGs). Despite only being trained for classification, PAGs cause robust models to have rudimentary generative capabilities, including image generation, denoising, and in-painting. However, the underlying mechanisms behind these phenomena remain unknown. In this work, we provide a first explanation of PAGs via \emph{off-manifold robustness}, which states that models must be more robust off- the data manifold than they are on-manifold. We first demonstrate theoretically that off-manifold robustness leads input gradients to lie approximately on the data manifold, explaining their perceptual alignment. We then show that Bayes optimal models satisfy off-manifold robustness, and confirm the same empirically for robust models trained via gradient norm regularization, randomized smoothing, and adversarial training with projected gradient descent. Quantifying the perceptual alignment of model gradients via their similarity with the gradients of generative models, we show that off-manifold robustness correlates well with perceptual alignment. Finally, based on the levels of on- and off-manifold robustness, we identify three different regimes of robustness that affect both perceptual alignment and model accuracy: weak robustness, bayes-aligned robustness, and excessive robustness. Code is available at https://github.com/tml-tuebingen/pags.

Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness

Abstract

One of the remarkable properties of robust computer vision models is that their input-gradients are often aligned with human perception, referred to in the literature as perceptually-aligned gradients (PAGs). Despite only being trained for classification, PAGs cause robust models to have rudimentary generative capabilities, including image generation, denoising, and in-painting. However, the underlying mechanisms behind these phenomena remain unknown. In this work, we provide a first explanation of PAGs via off-manifold robustness , which states that models must be more robust off- the data manifold than they are on-manifold. We first demonstrate theoretically that off-manifold robustness leads input gradients to lie approximately on the data manifold, explaining their perceptual alignment. We then show that Bayes optimal models satisfy off-manifold robustness, and confirm the same empirically for robust models trained via gradient norm regularization, randomized smoothing, and adversarial training with projected gradient descent. Quantifying the perceptual alignment of model gradients via their similarity with the gradients of generative models, we show that off-manifold robustness correlates well with perceptual alignment. Finally, based on the levels of on- and off-manifold robustness, we identify three different regimes of robustness that affect both perceptual alignment and model accuracy: weak robustness, bayes-aligned robustness, and excessive robustness. Code is available at https://github.com/tml-tuebingen/pags .

1 Introduction

An important desideratum for machine learning models is robustness , which requires that models be insensitive to small amounts of noise added to the input. In particular, adversarial robustness requires models to be insensitive to adversarially chosen perturbations of the input. Tsipras et al. [1] first observed an unexpected benefit of such models, namely that their input-gradients were 'significantly more human-aligned' (see Figure 1 for examples of perceptually-aligned gradients). Santurkar et al. [2] built on this observation to show that robust models could be used to perform rudimentary image synthesis - an unexpected capability of models trained in a purely discriminative manner. Subsequent works have made use of the perceptual alignment of robust model gradients to improve zero-shot

CIFAR-10

ImageNet

Image

Score

(PAG)

object localization [3], perform conditional image synthesis [4], and improve classifier robustness [5, 6]. Kaur et al. [7] coined the term perceptually-aligned gradients (PAGs), and showed that it occurs not just with adversarial training [8], but also with randomized smoothed models [9]. Recently, Ganz et al. [6] showed that the relationship between robustness and perceptual alignment can also work in the opposite direction: approximately enforcing perceptual alignment of input gradients can increase model robustness.

Despite these advances, the underlying mechanisms behind the phenomenon of perceptually aligned gradients in robust models are still unclear. Adding to the confusion, prior works have used the same term, PAGs, to refer to slightly different phenomena. We ground our discussion by first identifying variations of the same underlying phenomenon.

Phenomenon 1 (Perceptual Alignment) . The gradients of robust models highlight perceptually relevant features [1], and highlight discriminative input regions while ignoring distractors [11].

Phenomenon 2 (Generative Capabilities) . Robust models have rudimentary image synthesis capabilities, where samples are generated by maximizing a specific output class by iteratively following the direction dictated by the input-gradients [2].

Phenomenon 3 (Smoothgrad Interpretability) . Smoothgrad visualizations (which are gradients of randomized smooth models) tend to be visually sharper than the gradients of standard models [12].

While Kaur et al. [7] refer to Phenomenon 2 as PAGs, Ganz et al. [6] use this term to refer to Phenomenon 1. Since they are all related to the gradients of robust models, we choose to collectively refer to all of these phenomena as PAGs.

While the various phenomena arising from PAGs are now well-documented, there is little to no work that attempts to explain the underlying mechanism. Progress on this problem has been hard to achieve because PAGs have been described via purely qualitative criteria, and it has been unclear how to make these statements quantitative. In this work, we address these gaps and make one of the first attempts at explaining the mechanisms behind PAGs. Crucially, we ground the discussion by attributing PAGs to gradients lying on a manifold, based on an analysis of Bayes optimal classifiers. We make the following contributions:

We establish the first-known theoretical connections between Bayes optimal predictors and the perceptual alignment of classifiers via off-manifold robustness. We also identify the manifold w.r.t. which this holds, calling it the signal manifold .

We experimentally verify that models trained with gradient-norm regularization, noise augmentation and randomized smoothing all exhibit off-manifold robustness.

We quantify the perceptual alignment of gradients by their perceptual similarity with the score of the data distribution, and find that this measure correlates well with off-manifold robustness.

We identify three regimes of robustness: weak robustness, bayes-aligned robustness and excessive robustness, that differently affect both perceptual alignment and model accuracy.

2 Related Work

Robust training of neural networks Prior works have considered two broad classes of model robustness: adversarial robustness and robustness to normal noise. Adversarial robustness is achieved by training with adversarial perturbations generated using project gradient descent [8], or by randomized smoothing [9], which achieves certified robustness to adversarial attacks by locally averaging with normal noise. Robustness to normal noise is achieved by explicitly training with noise, a technique that is equivalent to Tikhonov regularization for linear models [13]. For non-linear models, gradient-norm regularization [14] is equivalent to training with normal noise under the limit of training with infinitely many noise samples [15]. In this paper, we make use of all of these robust training approaches and investigate their relationship to PAGs.

Gradient-based model explanations Several popular post hoc explanation methods[16, 17, 12] estimate feature importances by computing gradients of the output with respect to input features and aggregating them over local neighborhoods [18]. However, the visual quality criterion used to evaluate these explanations has given rise to methods that produce visually striking attribution maps, while being independent of model behavior [19]. While [20] attribute visual quality to implicit score-based generative modeling of the data distribution, [21] propose that it depends on explanations lying on the data manifold. While prior works have attributed visual quality to generative modeling of data distribution or explanations lying on data manifold, our work demonstrates for the first time that (1) off-manifold robustness is a crucial factor, and (2) it is not the data manifold / distribution, rather the signal manifold / distribution (defined in Section 3.2) that is the critical factor in explaining the phenomenon of PAGs.

3 Explaining Perceptually-Aligned Gradients

Our goal in this section is to understand the mechanisms behind PAGs. We first consider PAGs as lying on a low-dimensional manifold, and show theoretically that such on-manifold alignment of gradients is equivalent to off-manifold robustness of the model. We then argue that Bayes optimal models achieve both off-manifold robustness and on-manifold gradient alignment. In doing so, we introduce the distinction between the data and the signal manifold, which is key to understanding the manifold structure of PAGs. Finally, we present arguments for why empirical robust models also exhibit off-manifold robustness.

Notation Throughout this paper, we consider the task of image classification with inputs x ∈ R d where x ∼ X and y ∈ [1 , 2 , ...C ] with C -classes. We consider deep neural networks f : R d →△ C -1 which map inputs x onto a C -class probability simplex. We assume that the input data x lies on a k -dimensional manifold in the d -dimensional ambient space. Formally, a k -dimensional differential manifold M⊂ R d is locally Euclidean in R k . At every point x ∈ M , we can define a projection matrix P x ∈ R d × d that projects points onto the k -dimensional tangent space at x . We denote P ⊥ x = I -P x as the projection matrix to the subspace orthogonal to the tangent space.

3.1 Off-Manifold Robustness ⇔ On-Manifold Gradient Alignment

We now show via geometric arguments that off-manifold robustness of models and on-manifold alignment of gradients are identical. We begin by discussing definitions of on- and off-manifold noise. Consider a point x on manifold M , and a noise vector u . Then, u is off-manifold noise, if P x ( x + u ) = x , which we denote u := u off, and u is on-manifold noise, if P x ( x + u ) = x + u , which we denote u := u on. In other words, if the noise vector lies on the tangent space then it is on-manifold, otherwise it is off-manifold. Given this definition, we can define relative off-manifold robustness. For simplicity, we consider a scalar valued function, which can correspond to one of the C output classes of the model.

Definition 1. (Relative off-manifold robustness) A model f : R d → R is ρ 1 -off-manifold robust, wrt some noise distribution u ∼ U if

While this definition states that the model is more robust off-manifold than it is overall, it can equivalently be interpreted as being more robust off-manifold than on-manifold (with a factor ρ 1 1 -ρ 1 ). Let us now define on-manifold gradient alignment.

Definition 2. (On-manifold gradient alignment) A model f : R d → R has ρ 2 -on-manifold aligned gradients if

This definition captures the idea that the difference between the gradients and its on-manifold projection is small. If ρ 2 = 0 , then the gradients exactly lie on the tangent space. We are now ready to state the following theorem.

Theorem 1 (Equivalence between off-manifold robustness and on-manifold alignment) . A function f : R d → R exhibits on-manifold gradient alignment if and only if it is off-manifold robust wrt normal noise u ∼ N (0 , σ 2 ) for σ → 0 (with ρ 1 = ρ 2 ).

Proof Idea. Using a Taylor series expansion, we can re-write the off-manifold robustness objective in terms of gradients. We further use linear algebraic identities to simplify that yield the on-manifold gradient alignment objective. The full proof is given in the supplementary material.

When the noise is small, on-manifold gradient alignment and off-manifold robustness are identical. To extend this to larger noise levels, we would need to make assumptions regarding the curvature of the data manifold. Broadly speaking, the less curved the underlying manifold is (the closer it is to being linear), the larger the off-manifold noise we can add, without it intersecting at another point on the manifold. In practice, we expect image manifolds to be quite smooth, indicating relative off-manifold robustness to larger noise levels [22]. In this next part, we will specify the properties of the specific manifold w.r.t. which these notions hold.

3.2 Connecting Bayes Optimal Classifiers and Off-Manifold Robustness

In this subsection, we aim to understand the manifold structure characterizing the gradients of Bayes optimal models. We proceed by recalling the concept of Bayes optimality.

Bayes optimal classifiers. If we perfectly knew the data generating distributions for all the classes, i.e., p ( x | y = i ) for all C classes, we could write the Bayes optimal classifier as p ( y = i | x ) = p ( x | y = i ) / ∑ C j =1 p ( x | y = j ) . This is an oracle classifier that represents the 'best possible' model one can create from data with perfect knowledge of the data generating process. Given our assumption that the data lies on a low-dimensional data manifold, the Bayes optimal classifier is uniquely defined on the data manifold . However, outside this manifold, its behaviour is undefined as all of the class-conditional probabilities p ( x | y ) are zero or undefined themselves. In order to link Bayes-optimality and robustness, which is inherently about the behavior of the classifier outside the data manifold, we introduce a ground-truth perturbed data distribution that is also defined outside the data manifold. While we might consider many perturbed distributions, it is convenient to consider the data generating distributions that are represented by denoising auto-encoders with a stochastic decoder (or equivalently, score-based generative models [23]). Here, the stochasticity ensures that the data-generating probabilities are defined everywhere, not just on the data manifold. This approach allows us to estimate the data manifold, assuming that the autoencoder has a bottleneck layer with k features for a d -dimensional ambient space [22].

signal

input

distractor

p(y|signal)=p(y|input)

y="blueflycatcher"

p(y|distractor)=p(y)

Distinguishing Between the Data and the Signal Manifold. Given a classification problem, one can often decompose the inputs into a signal component and a distractor component. For intuition, consider the binary task of classifying cats and dogs. Given an oracle data generating distribution of cats and dogs in diverse backgrounds, the label must be statistically independent of the background and depend purely on the object (cat or dog) in the image. In other words, there exist no spurious correlations between the background and the output label. In such a case, we can call the object the signal and the background the distractor. Formally, for every input x there exists a binary mask m ( x ) ∈ { 0 , 1 } d such that the signal is given by s ( x ) = x ⊙ m ( x ) and the distractor is given by d ( x ) = x ⊙ (1 -m ( x )) . The signal and distractor components are orthogonal to one another ( s ( x ) ⊤ d ( x ) = 0 ), and we can decompose any input in this manner ( x = s ( x ) + d ( x ) ). We provide an illustrative figure in Figure 2. Using this, we can now define the signal-distractor distributions.

Definition 3. Given a data distribution p ( x | y ) for y ∈ [1 , C ] , we have masking functions m ( x ) such that the resulting distribution p ( x ⊙ (1 -m ) | y ) = p ( x ⊙ (1 -m )) is statistically independent of y . The sparsest such masking function (such that m ∗ = arg min m E x ∼ p ( x | y ) ∥ m ( x ) ∥ 0 ), yields a corresponding distribution p ( x ⊙ (1 -m ∗ ( x ))) , which is the distractor distribution, and its counterpart p ( x ⊙ m ∗ ( x ) | y ) the signal distribution.

While it is possible to define signals and distractors more generally as any linear projections of the data, in this work, we restrict ourselves to masking to relate to feature attributions [11]. While the subject of finding such optimal masks is the topic of feature attribution [24], in this discussion, we shall assume that the optimal masks m ∗ and the corresponding signal and distractor distributions are known. Similar to decomposing any point on the manifold, we can also decompose any vector on the tangent space on the data manifold into signal and distractor components using the optimal mask m ∗ . In other words, we can write

Finally, we note that this signal-distractor decomposition does not meaningfully exist for all classification problems in that it can be trivial with the entire data distribution being equal to the signal, with zero distractors. Two examples of such cases are: (1) ordinary MNIST classification has no distractors due to the simplicity of the task, as the entire digit is predictive of the true label and the background is zero. (2) multi-class classification with a large set of classes with diverse images also have no distractors due to the complexity of the task, as the background can already be correlated with class information. For example, if the dataset mixes natural images with deep space images, there is no single distractor distribution one can find via masking that is independent of the class label. This disadvantage arises partly because we aim to identify important pixels, whereas a more general definition based on identifying subspaces can avoid these issues. We leave the exploration of this more general class of methods to future work.

Given this definition of the signal and distractor distributions, we are ready to make the following theorem: the input-gradients of a Bayes optimal classifier lie on the signal manifold, as opposed to the general data manifold.

Theorem 2. The input-gradients of Bayes optimal classifiers lie on the signal manifold.

Proof Idea. We first show that the gradients of the Bayes optimal predictor lie in the linear span of the gradients of the gradients of the data distribution, thus lying on the tangent space of the data distribution. Upon making a signal-distractor decomposition of the gradients of the Bayes optimal classifier, we find that the distractor term is always zero, indicating that the gradients of the Bayes optimal classifier always lie on the signal manifold. The proof is given in the supplementary material.

The full proof is given in the Supplementary material. This theorem can be intuitively thought of as follows: the optimal classifier only needs to look at discriminative regions of the input in order to classify optimally. In other words, changing the input values at discriminative signal regions is likely to have a larger effect on model output than changing the inputs slightly at unimportant distractor regions, indicating that the gradients of the Bayes optimal classifier highlight the signal. Thus, the Bayes optimal classifier does not need to model the distractor; only the signal is sufficient. This fact inspires us to make the following hypothesis to help us ground the discussion on the perceptual alignment of gradients:

Hypothesis 1. The input gradients of classifiers are perceptually aligned if and only if they lie on the signal manifold, thus highlighting only discriminative parts of the input.

This indicates that Bayes optimal model's gradients are perceptually aligned. Recall that the signal component only provides information about discriminative components of the input, and thus, we expect this to connect to perceptual alignment from Phenomenon 1. Next, we ask whether the gradients of practical models, particularly robust models, are also off-manifold robust.

3.3 Connecting Robust Models and Off-Manifold Robustness

Previously, we saw that Bayes optimal classifiers have the property of relative off-manifold robustness. Here, we argue that off-manifold robustness also holds for robust models in practice. This is a nontrivial and a perhaps surprising claim: common robustness objectives such as adversarial training, gradient-norm regularization, etc are isotropic, meaning that they do not distinguish between on- and off-manifold directions.

Hypothesis 2. Robust models are off-manifold robust w.r.t. the signal manifold.

We provide two lines of argument in support for this hypothesis. Ultimately, however, our evidence is empirical and presented in the next section.

Argument 1: Combined robust objectives are non-isotropic. While robustness penalties itself are isotropic and do not prefer robustness in any direction, they are combined with the cross-entropy loss on data samples, which lie on the data manifold. Let us consider the example of gradient norm regularization. The objective is given by:

Here we have used, ∥∇ x f ( x ) ∥ 2 = ∥∇ on x f ( x ) + ∇ off x f ( x ) ∥ 2 = ∥∇ on x f ( x ) ∥ 2 + ∥∇ off x f ( x ) ∥ 2 due to ∇ on x f ( x ) ⊤ ∇ off x f ( x ) = 0 , which is possible because the on-manifold and off-manifold parts of the gradient are orthogonal to each other. Assuming that we are able to decompose models into onmanifold and off-manifold parts, these two objectives apply to these decompositions independently. This argument states that in this robust training objective, there exists a trade-off between the cross-entropy loss and the on-manifold robustness term, whereas there is no such trade-off for the off-manifold term, indicating that it is much easier to minimize off-manifold robustness than onmanifold. This argument also makes the prediction that increased on-manifold robustness must be accompanied by higher train loss and decreased out-of-sample performance, and we will test this in the experiments section.

GradientNorm(CIFAR-10)

ProjectedGradientDescent(CIFAR-10)

RandomizedSmoothing（CIFAR-10)

On-Manifold Perturbation

1.0

Off-Manifold Perturbation

L2-Norm

0.5

0.0

0.90

1.00-

0.8

0.85

0.75

0.80

0.6

1-LPIPS

0.50

0.75

0.4

0.25

0.70

10-4

10-2

100

102

10-4

10-2

100

102

104

10-3

10-2

10-1

100

101

0.2

RegularizationConstant

Adv.PerturbationBudget(Epsilon)

NoiseLevel

However, there are nuances to be observed here: while the data lies on the data manifold, the gradients of the optimal model lie on the signal manifold so this argument may not be exact. Nonetheless, for cases where the signal manifold and data manifold are identical, this argument holds and can explain a preference for off-manifold robustness over on-manifold robustness.

Argument 2: Robust linear models are off-manifold robust. It is a well-known result in machine learning (from, for example the representer theorem) that the linear analogue of gradient norm regularization, i.e., weight decay causes model weights to lie in the linear span of the data. In other words, given a linear model f ( x ) = w ⊤ x , its input-gradient are the weights ∇ x f ( x ) = w , and when trained with the objective L = E x ( f ( x ) -y ( x )) 2 + λ ∥ w ∥ 2 , it follows that the weights have the following property: w = ∑ N i =1 α i x i , i.e., the weights lie in the span of the data. In particular, if the data lies on a linear subspace , then so do the weights. Robust linear models are also infinitely off-manifold robust: for any perturbation u off orthogonal to the data subspace, w ⊤ ( x + u off ) = w ⊤ x , thus they are completely robust to off-manifold perturbations.

In addition, if we assume that there are input co-ordinates x i that are uncorrelated with the output label, then w i x i is also uncorrelated with the label. Thus the only way to minimize the mean-squared error is to set w i = 0 (i.e., a solution which sets w i = 0 has strictly better mean-squared error than one that doesn't), in which case the weights lie in the signal subspace, which consists of the subspace of all features correlated with the label. This shows that even notions of signal-distractor decomposition transfer to the case of linear models.

4 Experimental Evaluation

In this section, we conduct extensive empirical analysis to confirm our theoretical analyses and additional hypotheses. We first demonstrate that robust models exhibit relative off-manifold robustness (Section 4.1). We then show that off-manifold robustness correlates with the perceptual alignment of gradients (Section 4.2). Finally, we show that robust models exhibit a signal-distractor decomposition, that is they are relatively robust to noise on a distractor rather than the signal (Section 4.3). Below we detail our experimental setup. Any additional details can be found in the Supplementary material.

0.01

0.03

0.05

0.1

0.25

0.5

1

3

5

Adversarial Perturbation Budget (Epsilon)

0.45

0.50

0.55

0.60

0.65

0.70

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

1-LPIPS

ImageNet

10 -2

10 -1

10 0

10 1

10 2

10 3

Adversarial Perturbation Budget (Epsilon)

0.0

0.1

0.2

0.3

0.4

0.3

0.4

0.5

0.6

1-LPIPS

ImageNet-64x64

Data sets and Robust Models. We use CIFAR-10 [25], ImageNet and ImageNet-64 [26], and an MNIST dataset [27] with a distractor, inspired by [11]. We train robust Resnet18 models [28] with (i) gradient norm regularization, (ii) randomized smoothing, (iii) a smoothness penalty, and (iv) l 2 -adversarial robust training with projected gradient descent. The respective loss functions are given in the Supplementary material. On ImageNet, we use pre-trained robust models from [29].

Measuring On- and Off-Manifold Robustness. We measure on- and off-manifold robustness by perturbing data points with on- and off-manifold noise (Section 3.1). For this, we estimate the tangent space of the data manifold with an auto-encoder, similar to [22, 30, 21]. We then draw a random noise vector and project it onto the tangent space. Perturbation of the input in the tangent direction is used to measure on-manifold robustness. Perturbation of the input in the orthogonal direction is used to measure off-manifold robustness. To measure the change in the output of a classifier with C classes, we compute the L 2 -norm || f ( x ) -f ( x + u ) || 2 .

Measuring Perceptual Alignment. To estimate the perceptual alignment of model gradients, we would ideally compare them with the gradients of the Bayes optimal classifier. Since we do not have access to the Bayes optimal model's gradients ∇ x p ( y | x ) , we use the score ∇ x log p ( x | y ) as a proxy, as both lie on the same data manifold. Given the gradient of the robust model and the score, we use the Learned Perceptual Image Patch Similarity (LPIPS) metric [31] to measure the perceptual similarity between the two. The LPIPS metric computes the similarity between the activations of an AlexNet [32] and has been shown to match human perception well [31]. In order to estimate the score ∇ x log p ( x | y ) , we make use of the diffusion-based generative models from Karras et al. [10]. Concretely, if D ( x , σ ) is a denoiser function for the noisy probability distribution p ( x , σ ) (compare Section 3.2), then the score is given by ∇ x log p ( x , σ ) = ( D ( x , σ ) -x ) /σ 2 [10, Equation 3]. We use noise levels σ = 0 . 5 and σ = 1 . 2 on CIFAR-10 and ImageNet-64, respectively.

4.1 Evaluating Onvs. Off-manifold Robustness of Models

We measure the on- and off-manifold robustness of different Resnet18 models on CIFAR-10, using the procedure described above. We measure how much the model output changes in response to an input perturbation of a fixed size (results for different perturbation sizes can be found in Supplement Figure 9). The models were trained with three different robustness objectives and to various levels of robustness. The results are depicted in the top row of Figure 3. Larger values in the plots correspond to a larger perturbation in the output, that is less robustness. For little to no regularization (the left end of the plots), the models are less robust to random changes off- than to random changes on- the data manifold (the red curves lie above the green curves). Increasing the amount of robustness makes the models increasingly off-manifold robust (the red curves decrease monotonically). At the same time, the robustness objectives do not affect the on-manifold robustness of the model (the green curves stay roughly constant). This means that the robust models become relatively off-manifold robust (Definition 1). At some point, the robustness objectives also start to affect the on-manifold behavior of the model, so that the models become increasingly on-manifold robust (the green curves start to fall). As can be seen by a comparison with the accuracy curves in the bottom row of Figure 3, increasing on-manifold robustness mirrors a steep fall in the accuracy of the trained models (the

Image

Score

ModelGradients

0.25

0.2

0.43

0.58

0.71

0.69

0.57

0.47

0.46

0.79

0.78

0.77

0.68

0.92

0.93

0.78

0.84

0.71

WeaklyRobust

Bayes-Aligned Robust

ExcessivelyRobust

green and blue curves fall in tandem). Remarkably, these results are consistent across the different types of regularization.

4.2 Evaluating the Perceptual Alignment of Robust Model Gradients

We now show that the input gradients of an intermediate regime of accurate and relatively offmanifold robust models (" Bayes-aligned robust models") are perceptually aligned, whereas the input gradients of weakly robust and excessively robust models are not. As discussed above, we measure the perceptual similarity of input gradients with the score of the probability distribution. The bottom row of Figure 3 depicts our results on CIFAR-10. For all three robustness objectives, the perceptual similarity of input gradients with the score, as measured by the LPIPS metric, gradually increases with robustness (the orange curves gradually increase). The perceptual similarity then peaks for an intermediate amount of robustness, after which it begins to decrease. Figure 4 depicts our results on ImageNet and ImageNet-64x64. Again, the perceptual similarity of input gradients with the score gradually increases with robustness. On ImageNet-64x64, we also trained excessively robust models that exhibit a decline both in accuracy and perceptual alignment.

To gain intuition for these results, Figure 6, as well as Supplementary Figures 10-15, provide a visualization of model gradients. In particular, Figure 6 confirms that the model gradients belonging to the left and right ends of the curves in Figure 3 are indeed not perceptually aligned, whereas the model gradients around the peak (depicted in the middle columns of Figure 6) are indeed perceptually similar to the score.

While we use the perceptual similarity of input gradients with the score as a useful proxy for the perceptual alignment of input gradients, we note that this approach has a theoretical foundation in the energybased perspective on discriminative classifiers [33]. In particular, Srinivas and Fleuret [20], Zhu et al. [34] have suggested that the input gradients of softmaxbased discriminative classifiers could be related to the score of the probability distribution. To the best of our knowledge, our work is the first to quantitatively compare the input gradients of robust models with independent estimates of the score.

4.3 Evaluating Signal vs. Distractor Robustness for Robust Models

10 -2

10 -1

10 0

10 1

Noise Level

1

4

7

10

Signal/Distractor

Relative Noise Robustness (MNIST)

Resnet18

Robust Resnet18

We now show that robust models are relatively robust to noise on a distractor. Since a distractor is by definition not part of the signal manifold (Section 3.2), this serves as evidence that the input

gradients of robust models are aligned with the signal manifold. In order to control the signal and the distractor, we create a variant of the MNIST dataset where we artificially add the letter 'A' as a distractor. This construction is inspired by Shah et al. [11], for details see Supplement Figure 16. Because we know the signal and the distractor by design, we can add noise to only the signal or only the distractor and then measure the robustness of different models towards either type of noise. We call the ratio between these two robustness values the relative noise robustness of the model. Figure 5 depicts the relative noise robustness both for a standard- and an adversarially robust Resnet18. From Figure 5, we see that the standard model is already more robust to noise on the distractor. The robust model, however, is relatively much more robust to noise on the distractor. Since distractor noise is by definition off-manifold robust w.r.t. the signal manifold, this result serves as evidence for Hypothesis 2. Moreover, the input gradients of the robust model (depicted in the supplement) are perceptually aligned. Hence, this experiment also serves as evidence for Hypothesis 1.

5 Discussion

Three Regimes of Robustness. Our experimental results show that different robust training methods show similar trends in terms of how they achieve robustness. For small levels of robustness regularization, we observe that the classifiers sensitivity to off-manifold perturbations slowly decreases ( weak robustness), eventually falling below the on-manifold sensitivity, satisfying our key property of (relative) off-manifold robustness , as well as alignment with the Bayes classifier ( Bayes-aligned robustness). Excessive regularization causes models to become insensitive to on-manifold perturbations, which often corresponds to a sharp drop in accuracy ( excessive robustness).

The observation that robust training consists of different regimes (weak-, Bayes-aligned-, and excessive robustness) calls us to rethink standard robustness objectives and benchmarks [35], which do not distinguish on- and off-manifold robustness. An important guiding principle here can be not to exceed the robustness of the Bayes optimal classifier.

The Limits of Gradient-based Model Explanations. While Shah et al. [11] find experimentally that gradients of robust models highlight discriminative input regions, we ground this discussion in the signal-distractor decomposition. Indeed, as the gradients of the Bayes optimal classifier are meaningful in that they lie tangent to the signal manifold, we can also expect gradients of robust models to mimic this property. Here the zero gradient values indicates the distractor distribution. However, if a meaningful signal-distractor decomposition does not exist for a dataset, i.e., the data and the signal distribution are identical, then we cannot expect gradients even the gradients of the Bayes optimal classifier to be "interpretable" in the sense that they highlight all the input features as the entire data is the signal.

Limitations. First, while our experimental evidence strongly indicates support for the hypothesis that off-manifold robustness emerges from robust training, we do not provide a rigorous theoretical explanation for this. We suspect that it may be possible to make progress on formalizing Argument 1 in Section 3.3 upon making suitable simplifying assumptions about the model class. Second, we only approximately capture the alignment to the Bayes optimal classifier using the score-gradients from a SOTA diffusion model as a proxy, as we do not have access to a better proxy for a Bayes optimal classifier.

References

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations , 2019.

Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with a single (robust) classifier. In Advances in Neural Information Processing Systems , 2019.

Gunjan Aggarwal, Abhishek Sinha, Nupur Kumari, and Mayank Singh. On the benefits of models with perceptually-aligned gradients. In Workshop at International Conference on Learning Representations , 2020.

Bahjat Kawar, Roy Ganz, and Michael Elad. Enhancing diffusion-based image synthesis with robust classifier guidance. Transactions on Machine Learning Research , 2023.

Tsachi Blau, Roy Ganz, Chaim Baskin, Michael Elad, and Alex Bronstein. Classifier robustness enhancement via test-time transformation. arXiv preprint arXiv:2303.15409 , 2023.

Roy Ganz, Bahjat Kawar, and Michael Elad. Do perceptually aligned gradients imply adversarial robustness? In International Conference on Machine Learning , 2023.

Simran Kaur, Jeremy Cohen, and Zachary C Lipton. Are perceptually-aligned gradients a general property of robust classifiers? Workshop at Advances in Neural Information Processing Systems , 2019.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018.

Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning , 2019.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 2022.

Harshay Shah, Prateek Jain, and Praneeth Netrapalli. Do input gradients highlight discriminative features? Advances in Neural Information Processing Systems , 2021.

Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 , 2017.

Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation , 1995.

Harris Drucker and Yann Le Cun. Improving generalization performance using double backpropagation. IEEE transactions on neural networks , 1992.

Suraj Srinivas and François Fleuret. Knowledge transfer with jacobian matching. In International Conference on Machine Learning , 2018.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. 2014.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning , 2017.

Tessa Han, Suraj Srinivas, and Himabindu Lakkaraju. Which explanation should i choose? a function approximation perspective to characterizing post hoc explanations. In Advances in Neural Information Processing Systems , 2022.

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems , 2018.

Suraj Srinivas and Francois Fleuret. Rethinking the role of gradient-based attribution methods for model interpretability. In International Conference on Learning Representations , 2021.

Sebastian Bordt, Uddeshya Upadhyay, Zeynep Akata, and Ulrike von Luxburg. The manifold hypothesis for gradient-based explanations. In Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , 2023.

Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deep generative models. In IEEE Conference on Computer Vision and Pattern Recognition Workshops , 2018.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 2011.

Neil Jethani, Mukund Sudarshan, Yindalon Aphinyanaphongs, and Rajesh Ranganath. Have we learned to explain?: How interpretability methods can learn to encode predictions in their interpretations. In International Conference on Artificial Intelligence and Statistics , 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Toronto , 2009.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition , 2009.

Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 2012.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition , 2016.

Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? Advances in Neural Information Processing Systems , 2020.

Christopher Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus-Robert Müller, and Pan Kessel. Fairwashing explanations with off-manifold detergent. In International Conference on Machine Learning .

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2018.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems , 2012.

Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations , 2020.

Yao Zhu, Jiacheng Ma, Jiacheng Sun, Zewei Chen, Rongxin Jiang, Yaowu Chen, and Zhenguo Li. Towards understanding the generative capability of adversarially robust classifiers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7728-7737, 2021.

Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In Workshop at Advances in Neural Information Processing Systems , 2020.

A Additional Proofs

Theorem A.1 (Equivalence between off-manifold robustness and on-manifold alignment) . A function f : R d → R exhibits on-manifold gradient alignment if and only if it is off-manifold robust wrt normal noise u ∼ N (0 , σ 2 ) for σ → 0 (with ρ 1 = ρ 2 ).

Proof. We proceed by observing that we can decompose the input-gradient into on-manifold and offmanifold components by projecting onto the tangent space and its orthogonal component respectively, i.e., ∇ x f ( x ) = P x ∇ x f ( x ) + P ⊥ x ∇ x f ( x ) .

We also observe that we can write the gradient norm in terms of an expected dot product, i.e., 1 σ 2 E u ∼N (0 ,σ 2 ) ( ∇ x f ( x ) ⊤ u ) 2 = 1 σ 2 ∇ x f ( x ) ⊤ E ( uu ⊤ ) ∇ x f ( x ) = ∥∇ x f ( x ) ∥ 2 .

Using these facts we can compute the norm of the off-manifold component as follows,

The second line is obtained by using the fact above regarding re-writing the gradient norm in terms of the expected dot product, and the final line is obtained by using a first order Taylor expansion, which is exact in the limit of small sigma. From the equality of first and last terms, we have that the on-manifold gradient alignment ⇔ the off-manifold robustness.

Theorem A.2. The input-gradients of Bayes optimal classifiers lie on the signal manifold ⇔ Bayes optimal classifiers are relative off-manifold robust.

Proof. From definition 3, it is clear that given a classification problem, there exists a single distractor distribution d ( x ) . Now, we take gradients of log probabilities of the Bayes optimal classifiers, which results in:

We notice first that the vectors ∇ x log p ( x | y ) all lie tangent to the data manifold by definition, as this data generating process p ( x | y ) itself defines the data manifold. As ∇ x log p ( y | x ) is a linear combination of the class-conditional generative model gradients, it follows that the input-gradient of the Bayes optimal model also lie tangent to the data manifold. Now, like any vector on the tangent space at x , it can be decomposed into signal and distractor components. Computing the distractor, we find that

This happens because the distractor is independent of the label, thus the distractor component is zero, and the input-gradient of the Bayes optimal model lies entirely on the signal manifold. From Theorem A.1, it follows that when a model gradients lie on a manifold, it is also off-manifold robust wrt that manifold.

B Experimental Details

B.1 Robust Training Objectives

We consider the following robust training objectives, where l ( x, y ) denotes the cross-entropy loss function.

Gradient norm regularization: l ( f ( x ) , y ) + λ ∥∇ x f ( x ) ∥ 2 2 with a regularization constant λ .

A smoothness penalty: l ( f ( x ) , y ) + λ E ϵ ∼N (0 ,σ 2 ) ∥ f ( x + ϵ ) -f ( x ) ∥ 2 2 with a fixed noise level σ 2 and a varying regularization constant λ .

Randomized Smoothing: E ϵ ∼N (0 ,σ 2 ) l ( f ( x + ϵ ) , y ) with a noise level σ 2 .

Adversarial Robust Training: l ( f (˜ x ) , y ) where ˜ x = arg max ˜ x ∈ B ϵ ( x ) l ( f (˜ x ) , y ) and ˜ x was obtained from the ϵ -ball around x using projected gradient descent.

B.2 Training Details

On CIFAR-10, we trained Resnet18 models for 200 epochs with an initial learning rate of 0.025. When training with gradient norm regularization or the smoothness penalty and large regularization constants we reduced the learning rate proportional to the increase in the regularization constant. After 150 and 175 epochs, we decayed the learning rate by a factor of 10.

On ImageNet-64x64, we trained Resnet18 models for 90 epochs with a batch size of 4096 and an initial learning rate of 0.1 that was decayed after 30 and 60 epochs, respectively. We used the same parameters for projected gradient descent (PGD) as in [29], that is we took 3 steps with a step size of 2 ϵ/ 3 .

On the MNIST dataset with a distractor, we trained a Resnet18 model for 9 epochs with an initial learning rate of 0.1 that was decayed after 3 and 6 epochs, respectively. We also trained an l 2 -adversarially robust Resenet18 with projected gradient descent (PGD). We randomly chose the perturbation budget ϵ ∈ { 1 , 4 , 8 } and took 10 steps with a step size of α = 2 . 5 ϵ/ 10 .

B.3 Diffusion Models

On CIFAR-10, we use the unconditional diffusion model edm-cifar10-32x32-uncond-vp . For comparison, Figure Supplement Figure 17 presents the results with the conditional diffusion model edm-cifar10-32x32-cond-vp . On ImageNet-64x64, we use the conditional diffusion model edm-imagenet-64x64-cond-adm . All models are available at https://github.com/NVlabs/ edm . The noise levels σ = 0 . 5 and σ = 1 . 2 were chosen to maximize the perceptual quality of the estimate of the score obtained from the diffusion model before running the experiments with the LPIPS metric.

B.4 Conditional- and Unconditional Diffusion Models

Based on our theory, there are two principled ways to evaluate diffusion models and discriminative classifiers. The first is to take the input gradient with respect to a single class and compare it to a conditional diffusion model. The second is to sum the input gradients of all the classes and compare it to an unconditional diffusion model (i.e., computing marginal probabilities). The main paper present the first approach on ImageNet and ImageNet-64x-64 and the second approach on CIFAR-10. Supplement Figure 17 presents the first approach on CIFAR-10. We observe that there is no qualitative difference, i.e. both approaches give the same result.

B.5 Model Gradients

With the unconditional diffusion model, we sum the input gradients across all classes. With the conditional diffusion model, we consider the input gradient with respect to the predicted class. We consider input gradients before the softmax [20].

B.6 CIFAR-10 Autoencoder

We use https://github.com/clementchadebec/benchmark_VAE to train an autoeoncoder on CIFAR-10 with a latent dimension k = 128 . We use a default architecture and training schedule. We then use the autoencoder to estimate, at each data point, a 128-dimensional tangent space. Figure 7 depicts random directions within the estimated tangent spaces.

B.7 Pre-Trained Robust Models on ImageNet

On ImageNet, we use the pre-trained robust Resnet18 models form https://github.com/ microsoft/robust-models-transfer . To load these models, we use the robustness library https://github.com/MadryLab/robustness .

B.8 Estimating the Score on ImageNet

We estimate the score on ImageNet using the diffusion model for ImageNet-64x64. To estimate the score, we simply down-scale an image to 64x64.

B.9 MNIST with a Distractor

The MNIST data set with a distractor is inspired by [11]. The data set consists of gray-scale images of size 56x28. Every image contains a single MNIST digit and the distractor. We choose the fixed letter "A" as the distractor. On every image, we randomly place the distractor on top or below the MNIST digit. In order to estimate the relative noise robustness, we separately add different levels of noise to the signal or distractor. Figure 16 depicts images and models gradients on this data set.

B.10 The LPIPS metric

The LPIPS metric measures the perceptual similarity between two different images. The metric itself corresponds to a loss, meaning that lower values correspond to more similar images [31]. The figures in the main paper depict 1-LPIPS, that is higher values correspond to more similar images.

C Additional Plots

The figures below depict the model gradients of different types of models, ranging from weakly robust to excessively robust. The figures depict the relationship between model gradients and the score qualitatively. This complements the quantitative results in the main paper.

Image

Perturbation Norm =1.4

PerturbationNorm =2.6

PerturbationNorm=4.8

PerturbationNorm=8.9

(a) Perturbation norm

L

2

= 1

.

4

(b) Perturbation norm

L

2

= 2

.

6

(c) Perturbation norm

L

2

= 4

.

8

(d) Perturbation norm

L

= 8

.

9

Gradient Norm(CIFAR-10)

Projected GradientDescent(CIFAR-10)

Randomized Smoothing（CIFAR-10)

Smoothness Penalty(CIFAR-10)

On-Manifold Perturbation

Off-Manifold Perturbation

0.20

0.15

0.100

2-Norm

0.10

0.15

0.10

0.075

0.10

0.050

2

0.05

0.05

0.05

0.025

0.00

0.00

0.00

0.000

10-4

102

100

102

10-4

102

100

102

104

10-2

100

102

103

101

101

103

RegularizationConstant

Adv.Perturbation Budget(Epsilon)

Noise Level

Regularization Constant

GradientNorm （CIFAR-10)

Projected GradientDescent(CIFAR-10)

RandomizedSmoothing（CIFAR-10)

SmoothnessPenalty(CIFAR-10)

0.4

On-Manifold Perturbation

Off-Manifold Perturbation

0.15

2-Norm

0.3

0.4

0.4

0.10

0.2

2

0.1

0.2

0.2

0.05

0.0

10-4

Regularization Constant

102

100

102

0.0

Adv.Perturbation Budget (Epsilon)

10-4

10-2

100

102

104

0.0

10-2

Noise Level

100

102

0.00

10-3

ReaularizationConstant

10-1

101

103

1.00

GradientNorm （CIFAR-10)

On-Manifold Perturbation

1.00

Projected GradientDescent(CIFAR-10)

1.00

Randomized Smoothing(CIFAR-10)

0.4

SmoothnessPenalty(CIFAR-10)

lorm

0.75

Off-Manifold Perturbation

0.75

0.75

0.3

N-

0.50

0.50

0.50

0.2

L2

0.25

0.25

0.25

0.1

0.00

10-4

RegularizationConstant

10-2

100

102

0.00

Adv.Perturbation Budget(Epsilon)

10-4

10-2

100

102

104

0.00

102

Noise Level

100

102

0.0

103

Regularization Constant

10-1

101

103

Gradient Norm (CIFAR-10)

Projected GradientDescent(ClFAR-10)

RandomizedSmoothing(CIFAR-10)

SmoothnessPenalty(CIFAR-10)

On-Manifold Perturbation

1.00

L2-Norm

1.0

Off-Manifold Perturbation

1.0

1.0

0.75

0.5

0.5

0.5

0.50

0.25

0.0

10-4

Regularization Constant

10-2

100

102

0.0

Adv.Perturbation Budget (Epsilon)

10-4

102

100

102

104

0.0

102

Noise Level

100

102

0.00

10-3

Regularization Constant

10-1

101

103

2

Increasing Robustness

Increasing Robustness

Increasing Robustness

Increasing Robustness

Increasing Robustness

JNOLRGROUND

Increasing Robustness

A

7

A

A

7

2

A

A

A

A

2

2

A

A

h

A

A

A

A

A

人

4

-

A

A

A

A

A

6

A

A

9

空

A

GradientNorm(CIFAR-10)

ProjectedGradientDescent(CIFAR-10)

RandomizedSmoothing(CIFAR-10)

0.90

1.00

0.8

0.85

0.75

0.80

0.6

S

P

P

0.50

0.75

0.4

0.25

0.70

0.2

10-4

10-2

100

102

10-4

10-2

100

102

104

10-3

10-2

10-1

100

101

RegularizationConstant

Adv.PerturbationBudget(Epsilon)

NoiseLevel

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical characterization and some theorems supporting previous empirical finidings on perceptually aligned gradients (PAG) of classification neural network models. Specifically, it provides the first rigorous definitions for the previously qualitative definitions of \"PAG\", and provides the first attempt at a theoretical framework connecting PAG and model robustness, which has been observed empricially in previous work.\n\nSTRENGTHS:\n- Very well-written and well-presented paper. Covers and taxonomizes the relevant literature very nicely.\n- Math and definitions are intuitive, relevant, and sound. I especially like the fact that the theorem assumes almost nothing on the model behavior or training objective.\n- Overall, this paper addresses a very important and relevant topic, characterizes a lot of themes in it, and provides a great theoretical explanation for it. This theoretical justification was notably missing from previous PAG literature.\n- Empirical evaluation metrics are carefully picked and well-thought-out. Comparing PAG to score-based models is especially relevant (this was first suggested in [6], I think a reference is due on lines 264-274).\n\nWEAKNESSES:\n- lines 106-107: noise can also have both on- and off- manifold elements (think of a linear combination of two on- and off- manifold noise vectors). Therefore, the \"otherwise\" on line 107 is incorrect.\n- Minor issue: Defnition 1 is missing a definition of x (for formality).\n- While the math (theorem 1) is nice, it does not consider adversarially chosen noise. It would be nice to mention that for normal noise with $\\sigma \\rightarrow 0$, this covers adversarial noise as well, but it should be made clear that more work is needed to specifically consider adversarial attacks.\n- Line 151: Paragraph title mentions difference between data and signal manifolds, but the paragraph does not discuss \"data manifold\". Typo?\n- The definition of signal vs distractor is too simplistic. Signals and distractors are not necessarily separated by a pixel mask only. The underlying signal manifold could lie on a linear or non-linear projection of x, not necessarily on the pixel space itself. This weakens the resulting math (theorem 2), unless the authors can rewrite their proofs w.r.t. a determinstic \"signal\" function that is generalized beyond simple point-wise masking.\n\nQUESTIONS:\n- In the abstract, is noise augmentation and randomized smoothing the same thing? If so, I would suggest listing them as one item and not two.\n- Lines 272-274: Is the score-based model used here class-conditional? If so, it should be made clear by defining p(x|y) and not p(x) in the score function definitions. If not, further explanation is needed. Why would we characterize perceptual alignment using a class-unaware model? \"PAG\" usually refers to salient class-specific features.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies Perceptually Aligned Gradients (PAGs), a phenomenon where the input gradients are semantically meaningful and aligned with human perception.\nWhile this trait gained research attention, we do not truly understand it.\nTo this end, the paper proposes an explanation via off-manifold robustness.\nNamely, they show empirically and theoretically that models that are off-manifold robust have perceptually aligned gradients for different robustification techniques.\nMoreover, they propose a quantitative way to assess PAG, while prior to this work, PAG was evaluated qualitatively only.\nIn addition, they identify three levels of robustness and analyze them in terms of PAG.\n\nSTRENGTHS:\n* Perceptually Aligned Gradients is a fascinating phenomenon, and despite gaining substantial research attention, we do not understand it. Shedding light on this trait is an interesting and important research goal. The paper empirically and theoretically verifies the relationship between PAG and off-manifold robustness. To strengthen their findings, they do so for different robustification techniques.\n\n* Decomposing the gradients of a model to on and off-manifold directions is very interesting thinking. Besides being theoretically logical, they also develop a way of doing so.\n\n* Excellent and clear writing. The authors have done a good job of providing motivation for this work. In addition, while several papers use this term to describe different behaviors, this paper formulates the definitions and makes an order in the various definitions.\n\nWEAKNESSES:\n* The connection between a conditional score function and PAG was made in [1]. Despite not being used for assessing PAG, they show that a classifier trained to mimic the score function possesses PAG. Thus, I do not find linking the two for assessing PAG very novel.\n\n* Overstatements and bit trivial claims - “In this work, we provide 7 a first explanation of PAGs” [abstract], but [4] offers an explanation of the generative capabilities of adversarial training using energy-based models. Additionally, I find some claims that the authors made efforts to explain quite straightforward. For example, robust models are off-manifold robust. We know robust models have PAG, meaning their gradients are aligned on the manifold. This means that the change of loss in off manifold direction is smaller than the on-manifold (because otherwise, the gradients weren’t aligned), making such models off-manifold robust.\n\n**Minor weaknesses and requests**\n\n* I find this important to report the LPIPS results in Figure 5 to demonstrate that the used metric indeed makes sense and aligns with the human evaluation of PAG.\n\n* Missing citations - I think that in order to motivate the significance of studying PAG for the general audience, it is important to include more works that study and utilize PAG. For instance, [2,3].\n\n[1] Do Perceptually Aligned Gradients Imply Robustness?\n\n[2] On the benefits of models with perceptually-aligned gradients.\n\n[3] BIGRoC: Boosting Image Generation via a Robust Classifier.\n\n[4] Towards Understanding the Generative Capability of Adversarially Robust Classifiers.\n\nQUESTIONS:\n* Is the score function being modeled in the measuring Perceptual Alignment paragraph for clean images or noisy images? From my understanding, diffusion networks model the noisy score function and not the clean one.\n\n* What is the rationale for measuring the LPIPS for gradients? It is usually used for images, and input-gradient distribution is significantly different. Why does it make sense to use this rather than cosine similarity or other closed-form metrics?\n\n* Figure 2 shows that there are points where models are good in off-manifold robustness before being on-manifold robust (for example, the left figure in the top row of Figure 2, around a noise level of 0.5). Can this analysis help us find robust models that have better robustness-accuracy tradeoffs?\n\n* Why are the results of Imagenet 64 so low?\n\nIn general, I find this paper interesting and would like to increase my score given the authors' response.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper seeks to provide a condition that leads models to have gradients that are aligned with human perception, i.e. which highlight relevant features of the image while ignoring distractor features. The paper first proposes that for any manifold, if a model is robust to perturbations which are off-manifold, then its gradients will be aligned with the manifold. The paper then defines a \"signal\" manifold as a hypothetical manifold of features which relevant to the classification task, defined as the multiplication of the data with a masking function. The paper then states and proves that a Bayes optimal classifier is robust to perturbations perpendicular to this manifold, and hence its gradients lie on the signal manifold. The paper further hypothesizes that gradients on the signal manifold are perceptually aligned. An argument is then provided that models robust to adversarial perturbations which have similar off-manifold robustness as the Bayes optimal classifier have perceptually aligned gradients. Finally, the paper presents empirical evidence in support of their theoretical claims and hypotheses.\n\nSTRENGTHS:\n1. The paper presents an interesting hypothesis and designs a robust experimental methodology to validate it within the set of assumptions. \n2. The notion of weakly robust, optimally robust and overly robust models and their connection to PAGs is novel, and interesting. Comparing such classifiers with score functions is interesting in its own right, and can be a good tool in analysing models empirically.\n\nWEAKNESSES:\n1. It is not clear from the experiments what the signal manifold is. Line 278 talks about perturbing about 10\\% of the input, but it would be good to clarify this further.\n2. Hypothesis 1 is never validated empirically in the paper, but the experiments build upon this hypothesis by comparing the perceptual similarity with the gradients of the score function.\n\nQUESTIONS:\n1. How are on- and off-manifold perturbations generated exactly?\n2. Is there any way to provide more empirical evidence for Hypothesis 1?\n\nLIMITATIONS:\nThe authors address their limitations in sections 3 and 4, where they describe how much of an approximation their empirical setup and theorems are.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the connection between manifold alignment, perceptually-aligned gradients, and model robustness, deriving a number of novel results that explain previously observed phenomena in the explainability and robustness literatures. Crucially the paper grounds its contributions in Bayes optimal classifiers, arguing that the gradients of such classifiers are aligned with a “signal manifold”. The paper leverages denoising autoencoders to estimate the data manifold and diffusion models to estimate the gradients of a Bayes optimal classifier. Using these quantities, the paper demonstrates a number of novel results, including a strong relationship between manifold alignment and model accuracy.\n\nSTRENGTHS:\nThis paper presents a timely and likely impactful theoretical contribution with implications not only for robust learning but also for explainability and generalization. The paper is very well organized and well-written. The introduction provides an excellent summary of the literature and current open questions. The paper disambiguates between three closely-related notions of “perceptually-aligned gradients” and provides a powerful taxonomy through the definition of Phenomena 1-3.\n\nDrawing on prior work, the paper introduces quantitative metrics to evaluate robustness and gradient alignment with respect to a data manifold. The theoretical contributions are somewhat geometrically simplistic, but powerful. Theorem 1 has been assumed intuitively in prior work, but this paper introduces the formalisms to quantify the connection between off-manifold robustness and on-manifold gradient alignment. Argument 1-2 outline very compelling circumstantial evidence for the broader Hypotheses and will hopefully lead to future theoretical developments.\n\nThe paper provides extensive experimental evidence, including useful error bars. Figure 2 provides gorgeous plots and the correlation between manifold alignment and model accuracy is striking.\n\nFinally, the paper makes a number of practical advances related to effective adversarial training. Figure 5 intuitively summarizes the paper’s crucial claims about three regimes of robust training.\n\nWEAKNESSES:\n* It does not seem right that the mask defining distractor vs signal features is constrained to be binary (Definition 3). This means the signal and distractor distributions are not invariant to linear transformations of $x$. Shouldn’t the only requirement be the statistical independence on line 163?\n* All experiments evaluating perceptual alignment used the LPIPS metric. Wouldn’t another metric, such as cosine distance, be appropriate. This would allow for distractors that aren’t defined by a binary mask.\n* Unfortunately (probably due to computational cost) different results are presented for CIFAR-10 and ImageNet, reducing the global consistency of the paper.\n* The experiment shown in Figure 4 is not well explained, would it be possible to explain the setup in a little more detail using previously-established notation?\n\nSmall issues:\n* The definition of Bayes optimal classifiers assumes an equal class distribution (i.e. uniform p(y)). This most likely will not be the case in most applications. Note that this does not invalidate any of the paper’s findings.\n* Type on line 200 “Bayes optimal models gradients”\n* Missing article on line 220: “there exists a trade-off between [the] cross-entropy loss and on-manifold robustness term”\n* Missing pronoun at line 351: “meaningful in that [they] lie tangent”\n* At line 354, “gradients is repeated, and do you mean the double negative “cannot be uninterpretable”?\n* Figures 4 and 5 are out of order.\n* The numbering of Theorems in the Supplemental material is off (the proofs continue as Theorem 3, 4 rather than 1, 2).\n\nQUESTIONS:\n1. Why is Theorem 2 stated with an if and only if? Don’t you prove that both statements are true for Bayes optimal classifiers?\n2. Would it be possible to add an adversarially-trained model for CIFAR-10?\n3. On line 228 you say “We elaborate on this argument in the Supplementary material.” Where?\n4. On line 255, does “$l_2$-adersarial robust training” mean PGD applied during training?\n5. In the legend for Figure 2, is “perturbation” really the right term? Isn’t the plot essentially showing the inverse of robustness? Why not use the relative robustness metric defined in Definition 1? Why is an $l_1$ divergence used here rather than the $l_2$ divergence used in Definiton 1? Can $\\rho_1$ here be connected to Brier score to provide a proper score function?\n6. Are error bars standard error?\n7. Do you have any comment on the point where the lines cross in Figure 2 and why there is a different trade off depending on the training objective?\n8. At line 274, how were these specific $\\sigma$ values chosen for each dataset to parameterize the diffusion model?\n\nLIMITATIONS:\nThe acknowledged limitations throughout the paper and the supplemental material are superb (e.g. the statement “We provide two lines of argument in support for this hypothesis [Hypothesis 2]. Ultimately, however, our evidence is empirical”).", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper titled \"Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness\" investigates the phenomenon of perceptually-aligned gradients (PAGs) in robust computer vision models. PAGs refer to the alignment of model gradients with human perception, enabling these models to exhibit generative capabilities such as image generation, denoising, and in-painting. The paper aims to explain the underlying mechanisms behind PAGs and provides insights into the relationship between off-manifold robustness, gradient alignment, and model accuracy.\n\nThe paper's contributions can be summarized as follows:\n\n1. Theoretical Explanation: The paper presents a theoretical explanation of PAGs through the concept of off-manifold robustness. It demonstrates that the alignment of gradients with the data manifold is equivalent to off-manifold robustness. It also introduces the distinction between the data manifold and the signal manifold, which helps in understanding the input gradients of robust discriminative models.\n\n2. Connection to Bayes Optimal Models: The paper establishes a connection between Bayes optimal models and off-manifold robustness. It shows that Bayes optimal models achieve both off-manifold robustness and on-manifold gradient alignment. The input gradients of Bayes optimal classifiers lie on the signal manifold, indicating their perceptual alignment.\n\n3. Empirical Analysis: Extensive empirical analysis is conducted to validate the theoretical findings. Robust models trained with various techniques are evaluated on different datasets. The experiments confirm the relative off-manifold robustness of robust models, the correlation between off-manifold robustness and perceptual alignment, and the presence of signal-distractor decomposition in robust models.\n\nThe paper's main contribution lies in providing a theoretical framework and empirical evidence for understanding the mechanisms behind PAGs in robust computer vision models. It sheds light on the importance of off-manifold robustness and its connection to gradient alignment and model accuracy. The findings have implications for developing more explainable and generative models in computer vision tasks. By identifying different regimes of robustness, the paper also calls for rethinking standard robustness objectives and benchmarks.\n\nOverall, the paper advances our understanding of the properties and behavior of robust models, paving the way for further research in improving model interpretability and generative capabilities in the field of computer vision.\n\nSTRENGTHS:\nThe strength of the paper lies in its combination of theoretical analysis and empirical evaluation. It provides a comprehensive examination of the phenomenon of perceptually-aligned gradients (PAGs) in robust computer vision models. By presenting a theoretical framework based on off-manifold robustness and its connection to gradient alignment, the paper offers a solid foundation for understanding the underlying mechanisms behind PAGs.\n\nFurthermore, the empirical analysis conducted in the paper strengthens its findings. The authors perform extensive experiments using different datasets and robust training techniques, validating their theoretical explanations and hypotheses. The empirical results consistently support the theoretical claims, providing evidence for the presence of off-manifold robustness, gradient alignment, and signal-distractor decomposition in robust models.\n\nThe paper's contribution also extends to the exploration of various aspects related to PAGs, such as the connection to Bayes optimal models, the trade-off between on- and off-manifold robustness, and the correlation between off-manifold robustness and perceptual alignment. These insights broaden our understanding of the behavior and properties of robust models in computer vision tasks.\n\nOverall, the strength of the paper lies in its rigorous analysis, clear explanations, and compelling empirical evidence, making a significant contribution to the field of computer vision and model interpretability.\n\nWEAKNESSES:\nOne potential weakness of the paper is that the evaluation of on- and off-manifold perturbations does not cover the entire space of perturbations. There is a significant portion of perturbations that fall in-between the two categories, which are not thoroughly explored in the current research. It would be valuable for future work to address these intermediate cases and provide a more comprehensive understanding of the model's behavior in those scenarios.\n\nQUESTIONS:\nClarification on the choice of perturbations: Can the authors explain the rationale behind the specific choice of on- and off-manifold perturbations? How representative are these perturbations of real-world scenarios? Are there other types of perturbations that could be considered in future research?\n\nExtension to non-linear models: The paper primarily focuses on linear models and their relationship to off-manifold robustness. Can the authors discuss how their findings and hypotheses can be extended to non-linear models? Are there any unique considerations or challenges in analyzing non-linear models in this context?\n\nLIMITATIONS:\n- Limited exploration of perturbation space: The choice of on- and off-manifold perturbations in the experiments might not cover the entire space of possible perturbations. There could be a wide range of perturbations that lie between the two extremes. Exploring a more comprehensive set of perturbations and their effects on on- and off-manifold robustness could enhance the depth of the analysis. \n- Evaluation metrics: The paper primarily focuses on measuring robustness based on the change in model output. While this is a common approach, it may not capture all aspects of robustness. Considering additional evaluation metrics, such as adversarial examples or other robustness measures, could provide a more comprehensive understanding of model behavior.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}