The Manifold Hypothesis for Gradient-Based Explanations
Authors: Sebastian Bordt
Uddeshya Upadhyay
Zeynep Akata
Ulrike von Luxburg
Venue: ICLR 2022 Submitted (2022)
Keywords: Interpretability
Explainability

Abstract: When are gradient-based explanations meaningful? We propose a necessary criterion: explanations need to be aligned with the tangent space of the data manifold. To test this hypothesis, we employ autoencoders to estimate and generate data manifolds. Across a range of different datasets -- MNIST, EMNIST, CIFAR10, X-ray pneumonia and Diabetic Retinopathy detection -- we demonstrate empirically that the more an explanation is aligned with the tangent space of the data, the more interpretable it tends to be. In particular, popular post-hoc explanation methods such as Integrated Gradients and SmoothGrad tend to align their results with the data manifold. The same is true for the outcome of adversarial training, which has been claimed to lead to more interpretable explanations. Empirically, alignment with the data manifold happens early during training, and to some degree even when training with random labels. However, we theoretically prove that good generalization of neural networks does not imply good or bad alignment of model gradients with the data manifold. This leads to a number of interesting follow-up questions regarding gradient-based explanations.



Under review as a conference paper at ICLR 2022
THE MANIFOLD HYPOTHESIS FOR GRADIENT-BASHE EXPLANATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
When are maadient-based explanations meaningful? We propose necessary crite- rion: explanations need to be aligned with the tangent space of the data manifold. To test this hypothesis, we employ autoencoders to estimate and generate data manifolds. Across a range of different datasets MNIST, EMNIST, CIFARIO I0, X-ray pneumonia and Diabetic Retinopathy detection cally that the more an explanation is aligned with the tangent space of the data, the more interpretable it tends be. In partieular, popular post-hoc explanation methods such as Integrated Gradients and SmoothGrad tend to align their results with the data manifold. The same is true for the outcome of adversarial training, which has been claimed to lead to more interpretable explanations. Empirically, alignment with the data manifold happens early during training, and to some de- gree even when training with random labels. However, we theoretically prove that good generalization of neural networks does imply good bad alignment of model gradients with the data manifold. This leads to number of interesting follow-up questions regarding gradient- based explanations.
demonstrate empiri-
INTRODUCTION
A large number of algorithms aim to provide post-hoc explanations for the output of neural net- works (Simonyan al., 2014; Bach al., 2015; Shrikumar al., 2017; Ancona et al., 2018; Lim et al., 2021). Many of them are, directly or indirectly, based on the gradient with respect to the input (Smilkov et al., 2017; Sundararajan al., 2017: Garreau Mardaoui, 2021; Agarwal al., 2021). A particularly interesting application for gradient-based input attribution methods is natural image classification. Despite recent attempts to provide a priori interpretable image classifiers, neu- ral networks remain exceptionally successful at classification (Chen et al., 201 8a). However, recent work has demonstrated that post-hoc explanation methods fail various sanity checks (Adebayo al., 201 8b;a; Kindermans et al., 2019; Arun et al., 2020), and some have even suggested that they should not be used at all (Rudin, 2019).
In this paper, try to understand when and why gradient- based input attribution methods can be meaningful. To this end, we propose the following hypothesis:
Fradieet-based explanations are more meaningful the more they are aligned with the tangent space of the data manifold.
Consider image classification. It is widely believed that natural image data concentrates around low-dimensional image manifold (Goodfellow et al., 2016, Section 5.11.3). This image manifold captures the geometric structure of the data. In particular, the tangent space captures all components of an image that can changed while still staying within the realm of natural images. If gradient- based explanation approximately lies in the tangent space, this means that it highlights mcaningful way in which the different components of an image contribute to the prediction. a gradient based explanation lies orthogonal to the tangent space, this means that it points in some direction that would not lead to realistic images, and human would have a hard time to understand its meaning.
Our motivation for proposing this hypothesis is twofold. First, we believe that it is intuitive, and we provide empirical evidence in its support. Second, we hope that the hypothesis can provide a perspective why obtaining gradient-based explanations might be more difficult than elassification.

---

Under review as a conference paper at ICLR 2022
Sumples E Rd
2'
&1
5
Pd
q
Classffer
AT Dienpas
(()
E Explanation allgged tangent space
Eg: Explanaion not aligued with saggen
dimensional manifold In R
.. f()) Gradient
Ingur
Figure 1: Conceptual overview of our approach. We first estimate the data manifold of an existing dataset with variational autoencoder, then the decoder generative model. On the generated data, we train a classifier f. For this classifier, we evaluate whether different gradient based expla- nations& &, align with the tangent space of the data manifold. Moving along an explanation aligned with the tangent space keeps us in the manifold, whereas moving along an orthogonal explanation takes us out of manifold. Our hypothesis that the latter does not lead meaningful explanations because it describes changes that lead to unnatural images.
To evaluate the hypothesis empirically, we employ autoencoders to estimate the data manifolds of five different datasets MNIST, EMNIST, CIFARIO, X-ray pncumonia and diabetic retinopathy detection. As depicted in Figure 1, we also use variational autoencoders as generative models. This allows us to generate datasets with completely known manifold structure. With this approach, provide qualitative and quantitative evidence that explanations that are more aligned with the tangent space of the data more interpretable.
To study when and why model gradients are aligned with the tangent space of the data, we first show that the gradients of neural networks initialization unrelated to the structure of the data manifold. This means that the learning algorithm picks up some aspects of the structure of the data manifold during training. We show that this happens early during training, and to some extent even when training with random labels. Moreover, under standard training procedures, the alignment between model gradients and the data manifold deteriorates as the model increasingly fits the labels. This is avoided by la adversarial training, which significantly aligns the model's gradients with the tangent space of the data. Is it always the case that neural network that generalizes necessarily adapts its gradients to the data manifold, at least to some degree? The answer is no, we show theoretically. Without further assumptions, the relation between data manifold and model gradients is ambiguous: The alignment between the two quantities can be arbitrarily good bad.
The organization of the paper is as follows. Sec.2 formally introduces the manifold hypothesis and outlines our conceptual approach. Sec.3 evaluates the hypothesis five datasets. Sec.4 discusses the effects of adversarial training and the evolution of model gradients over the course of training. Sec.5 contains formal proof that generalization does not imply alignment with the data manifold, Sec.6 discusses the related work and 7 discusses the implications of our results.
2 THE MANIFOLD HYPOTHESIS
Our goal is to evaluate the following hypothesis: A gradient-based explanation & E TRd at a point M is meaningful the more it is aligned with the tangent space of the data manifold at x. Below we first give background on data manifolds, tangent spaces and model gradients; then we detail our evaluation approach.
2.1 BACKGROUND
Data manifolds and tangent spaces. A k-dimensional differentiable manifold M Rd is subset d-dimensional space that locally resembles R At every point x E M, the tangent space T. is k-dimensional subspace of Rdd The tangent space T. consists of all directions such that x + v, for ||>||| small, is again close to the manifold. Manifolds and tangent spaces are the subject of differential geometry, to which we refer for comprehensive introduction.
The lonn-standing hypothesis that natural image data concentrates around low-dimensional im- age manifold is supported by a number of empirical studies (Weinberger & Saul, 2006; Fefferman
2

---

Under review as a conference paper at ICLR 2022
et al., 2016). However, accurately learning the data manifolds of natural image datasets manifold learning - is difficult and the exact properties of these manifolds remain unknown (Cayton, 2005; Aamari & Levrard, 2019). Shao et al. (2018) investigate the properties of manifolds generated by deep generative models and find that they have mostly low curvature.
Model gradients and the fraction of the gradient in tangent space. We consider neural net- works that learn differentiable functions f : R - R Here C' is the number of classes and the model prediction is given by arg max, f(z);- The gradient of class with respect to the input is given by grad, (2) = ((f()))) Unless mentioned otherwise, we always consider the model gradient with respect to the predicted class and before the softmax.
At every point z that lies on the data manifold M. we can decompose the gradient into a part that lies in tangent space and a part that is orthogonal to it. Formally, we have grad 01 + U2 with Vi Ta U2 T and V1 22. Here U1 is the part of the gradient that lies in the tangent space, and 02 is the part of the gradient that is orthogonal to the tangent space. If the gradient completely lies in the tangent space, we have d22 O. If the gradient is completely orthogonal to the tangent space, we have 11 O. In practice, some part of the gradient will lie in the tangent space and another part be orthogonal it, that is we have i & and 12 o O. To quantitatively measure how well the gradient is aligned with the tangent space, we compute the
| [grad, (x)]]
Fraction of the Gradient in Tangent Space
E [0, 1].
(1)
2.2 HOw DO WE KNOW THE DATA MANIFOLD?
To estimate whether an explanation aligned with the tangent space make use of autoen- coders. The various variants of variational au- toencoders (Kingma & Welling, 2013; Higgins et al., 2017), allow to estimate the data man- ifolds of existing datasets. Importantly, they also allow to generate datasets with completely known manifold structure (Algorithm 1). We make use of two related approaches that we term the generative approach and the recon- structive approach. In both approaches, we first train an autoencoder on the orignal dataset.
Construct classification problem with known manifold structure.
Algorithm 1
Require: Dataset X = (x,, y:)
Require: Dimension of latent space & E [d].
Train variational autoencoder 4a (P(((a) (a)) on X.
Latent 2, ~ po (Ir), Reconstruction d; ~ qo (2:).
Let e R*+d - [C] solve (&,, f:)) 1/i- Sample n-times from the prior & (O Ic).
The dataset is ã. go(x), Vi = c(,, &.). for i E [n] do
Compute tangent spaces
for I [d] do
for m E [k] do
The generative approach to create datasets with a completely known manifold strue- ture. To generate dataset with completely known manifold structure, we have to train a variational- another generative autoencoder (Tagasovska et al., 2019). After training. we pass the original dataset through the autoen- coder. Then we train an additional classifier that reproduces the original labels from latent codes and reconstructed images. Equipped with this labeling function, we sample from the prior and use decoder and labeling function to generate a dataset. If the decoder is differentiable, we can compute the tangent space at each datapoint x (Shao et al., 2018; Anders et al., 2020).
fp..m.
a(usli Fzm
(2:)
end for end for
11,100
ti
Ts. span
t4,a,0
ts,d,k
end for
The reconstructive approach create datasets with an estimated manifold structure. The main limitation of the generative approach is that we might not be able to obtain high-quality samples with reasonably small latent spaces. While there have been great advances generative modeling, state- of-the-art models like hierarchical variational autoencoders (Vahdat & Kautz, 2020) require very large latent spaces, i.e. k R d. For our analysis, it is however critical that Vk/d is small with k == d, the fraction of the gradient in tangent space is always 1. To evaluate our hypothesis on real- world nigh-dimensional image data where it is difficult obtain realistic samples with not-too- latent spaces, we rely on estimating the tangent space. That is we simply pass the original dataset through the autoencoder and take the reconstucted images with the original labels our dataset.
3

---

Under review as a conference paper at ICLR 2022
3 PUTTING THE HYPOTHESIS TO THE TEST
Explanation algorithms. We consider four gradient-based input attribution methods: The gra- dient (Simonyan et al., 2014), Integrated Gradients (Sundararajan et al., 2017), Input x Gradient (Ancona et al., 2018), 8), and SmoothGrad (Smilkov et al., 2017). The motivation behind Integrated Gradients is axiomatic. The motivation behind SmoothGrad to reduce noise the gradient. All four methods provide explanations as vectors in IR We can evaluate how each method is aligned with the tangent space of the data manifold by computing the fraction of the explanation method in tangent space. While other methods also provide explanations as vectors in R restrict our- selves to these four methods because they are directly related to the gradient with respect to the input, which is main object of investigation.
Experimental setting. Given dataset, obtained either with the generative or the reconstructive approach, we train a neural network to minimize the test error. For this network, we then eval- uate how gradients and other gradient-based explanation methods relate to the data manifold. To evaluate whether an explanation is meaningful, we use qualitative evaluations as demonstrated in (Simonyan et al., 2014; Bach et al., 2015; Sundararajan et al., 2017: Smilkov et al., 2017). We also rely on the literature that demonstrates the utility of Integrated Gradients and SmoothGrad for diabetic retinopathy detection (Sayres et al., 2019; Van Craenendonck et al., 2020).
When we quantitatively evaluate the fraction of an explanation in tangent space (1), we need to account for the fact that even a random vector has a non-zero fraction tangent space. random vector is by definition completely unrelated to the structure of the data manifold. The expected fraction of a random vector that lies in any k-dimensional subspace is /kyd. In our MNIST32 task, for example, d = 1024, k = 10 and /10/1024 0.1. Thus, we could only say that gradient- based explanation is systematically related to the tangent space of the data manifold if, on average, the fraction of the explanation in tangent space is significantly larger than 0.1.
Datasets. We evaluate our hypothesis several datasets. This includes (i) MNIST32 and MNIST256, two variants of the MNIST dataset (LeCunn et al., 1998) with 10 classes and 60000 grayscale training images and 10000 grayscale test images of size 32 x 32 and 256 256, respec- tively. (ii) EMNISTI28, a variant of the EMNIST dataset (Cohen et al., 2017) that extends MNIST with handwritten letters and has over 60 classes, (iii) the CIFAR 10 dataset (Krizhevsky al., 2009). We also evaluate our hypothesis on two real world high dimensional image datasets: X-ray Pneumo- nia (Kermany et al., 2018) and Diabetic Retinopathy Detection !. Both tasks have been used before to study the properties of post-hoc explanation methods (Rajaraman et al., 2019; Luján- Luján-García et al., 2020: Amyar et al., 2020; Arun et al., 2020; Chetoui & Akhloufi, 2020; Van Craenendonck et al., 2020). All further details on the datasets are provided in appendix A.
3.1 EVALUATION ON GRAYSACLE IMAGES: MNIST32, MNIST256 AND EMNISTI22
We first demonstrate on MNIST32 that the part of the gradient that lies in tangent space is meaning- ful, whereas the part of the gradient that is orthogonal to the tangent space is not meaningful. The MNIST32 dataset was generated with 3-TCVAE (Chen al., 2018b), trained on MNIST as de- scribed in Algorithm (details in appendix A). The data lies on completely known 10-dimensional image manifold in 1024-dimensional space. On this dataset, we train a simple neural network with two convolutional and two fully connected layers to test accuracy greater than 99%.
Figure 2 depicts model gradients for images from the test set of MNIST32 where the fraction of the gradient in tangent space is particularly large. The gradients are decomposed into the part of the gradient that lies tangent space (second row) and the part of the gradient that is orthogonal to the tangent space (third row). It can be seen quite clearly that the part of the gradient that lies in tangent space has meaningful structure that in particular relates to prediction that we aim to explain. In case of the number 3 (fourth column), regions that would complete an 8 have negative attribution. In case of the number 8 (ninth column), pixels that distinguish the number from a have a positive attribution instead. Similar patterns can be observed for other numbers, too. In contrast, the part of the gradient that is orthogonal to the tangent space does not have meaningful structure. In line with
Dataset at http:///www..aggecccooccccccccee
4

---

Under review as a conference paper at ICLR 2022
Prediction
Predictio: ]
Predicton: 2
Predictio:
Prediction 4
Prediction
Frediction:
Prediction
Prediction
Prediction: 9
O
I
2
3
4
5
6
9
8
9
3
6
8
3
1.2
100
2.5
Vinggted oradlets
Godlet
2.00
1.75
0.8
2.0
Broud Drud
1:30
1.25
8 1.00
0.4
1.0
0.75
0.2
0.5
0.30
0.25
0.0
0.0
0.00
0.05
0.25
0.30
0.35
0.00
0² o
Fraction tangent space
Fraction in tangent space
Fraction tangent spece
Figure 2: (First row) depicts images from the test set of MNIST32. (Second row) depicts the part of the gradient that lies in tangent space for these images. (Third row) depicts the part of gradient that is orthogonal to the tangent space for these images. Blue corresponds to negative attribution, red to positive attribution. (Bottom row) depicts the fraction of four different explanation methods in langent space. (Bottom left) MNIST32. (Bottom center) MNIST256. (Bottom right) EMNISTI28. Vertical gray line shows the expected fraction of a random vector in tangent space.
our hypothesis, it consists of seemingly unrelated spots of positive and negative attribution. We do not address the question whether the part of the gradient that is orthogonal to the tangent space at x is best deseribed as noise or has a structural component. Shah et al. (2021) show that model gradients are not able to differentiate instance-specifee features from other task-relevant features that are not pertinent to a given input instance. In this respect, note that part of the gradient that is orthogonal to the tangent space at might lie within the tangent space of other instances on the data manifold.
To provide quantitative evidence for our hypothesis we look at the fraction in tangent space (Equa- tion 1) of different gradient-based attribution methods. If the hypothesis is correct, we would expect attribution methods that are known to provide better explanations to have a higher fraction in tangent space. The bottom- of Figure 2 depicts the distribution of the fraction in tangent space for model gradients, SmoothGrad, Integrated Gradients and Input x Gradient over all images from the test set. All four methods provide explanations whose fraction in tangent space is considerably larger than random. In particular, the mean fraction of the gradient in tangent space is significantly larger than random (t-test, p < 0.001). As expected under the hypothesis, Integrated Gradients and Input x Gradient improve upon the gradient. SmoothGrad seems to have little effect in this task.
As an experiment, we increase the dimension of the MNIST32 images 256 x 256 by bilinear upsampling. This preserve the structure of the data manifold while increasing the dimension of the ambient space, i.c. on MNIST256 the ratio /k/d is only 0.012. To measure the effect on the fraction of model gradients in tangent space, we train a ResNet18 (He et al., 2016) to test accuracy greater than 99% and then compute again the fraction in tangent space of various gradient- based attribution methods (bottom-center of Figure 2). At first glance, the picture seems similar to that of MNIST32 on the left- - the only modification being that SmoothGrad now improves upon the gradient. However, looking on the x-axis reveals that the fraction in tangent space of all methods significantly reduced. Figure 8 in the Appendix shows that this also impacts the visual quality of explanations. We conjecture that, holding everything else fixed, aligning model gradients with the data manifold is harder as the ratio /k/d decreases.
Finally, we present our results EMNISTI28. To create this dataset, used the reconstructive approach. We resized the images form EMNIST to 128 x 128 and estimated the data manifold with latent dimension of k- 656 (details are in appendix A.3). EMNIST128 has 60 classes and serves
5

---

Under review as a conference paper at ICLR 2022
Dan potroon
Iqper Goal
Food
Duoa pobn on
God
Topar
Randam Freggated Drucknt SreadhGrvel Grudont
Frucionn
fresmmnt
Prostee tangentssees
Retnopaitty
Figure 3: Comparing different explanation methods on (Top row) CIFARIO, (Middle row) X-Ray Pneumonia and (Bottom row) Diabetic Retinopathy detection. (Left part) qualitatively shows differ- ent explanations. (Right part) shows the distribution of the fraction tangent space for all methods over the images from the test set. In high dimensional cases SmoothGrad has the highest component, whereas Input x Gradient has the highest component on CIFARIO.
as an example of a high-dimensional problem. We train a VGG16 network that yields an accuracy of 99%. Figure (right) depicts again the fraction in tangent space of various explanation methods. SmoothGrad has the best performance, while again all post-hoc methods improve upon the gradient.
3.2 EXPERIMENTS ON CIFARIO AND MEDICAL DIAGNOSIS DATASETS
We now turn to CIFARIO and two medical diagnosis datasets, using the reconstructive approach. For CIFARIO, we trained a convolutional autoencoder with h 144. A VGG16 classifier on the reconstructed images achieves an accuracy 94.1% For pneumonia and diabetic retinopathy detection, we resized the images to 1x 256x224 (i.e., d = 57344) and 3x224x224 (i.e.. d 150528), respectively. We used a convolutional autoencoder with a latent dimension of 7168 and 6272, respectively. For both tasks train ResNet18 classifier the reconstructed images. It achieves an accuracy of 89% for pneumonia and 92% for retinopathy. Details are in Appendix A.
The top of Figure shows the performance of various gradient-based explanation methods on the reconstructed CIFAR10 dataset. The quantitative plot on the right shows that all post-hoc explanation methods have higher fraction in tangent space than the gradient. Among all methods, Input x Gradient achieves the highest fraction in tangent space. The qualitative example on the left indicates that Input x Gradient indeed provides the most meaningful explanations. For instance, for the upper-left image of the frog, while other methods seem to focus on only some parts of frog along with the background, Input Gradient focuses on the central region covering the entire frog. This is similar to the low-dimensional MNIST task (MNIST32). where Input x Gradient is among the best performing methods, indicating that SmoothGrad has little impact lower dimensions.
Figure 3 middle row shows the performance of gradient-based explanation methods on the pneu- monia dataset. The quantitative plots on the right show the distribution of the fraction tangent space for different methods on all the test images. We observe that in general the fraction is highest for SmoothGrad and lowest for raw gradients. The qualitative examples on the left indicate that SmoothGrad indeed focuses the relevant region near the lungs to make the correct predictions, whereas raw gradients do not seem to focus on the lungs. Similarly, Figure bottom row shows the performance of gradient-based exissedexplann explanation methods on the diahetic retinopathy dataset. Again, from

---

Under review as a conference paper at ICLR 2022
1.0
tostor nargeet space
Sperrme Pm
Dro
0.6
0.6
0.6
0.4
0.
02
8.0
10"
30"
20a
20
0.0
207
10
100
207
10
0.0
19P
121
307
20"
10
Number of Grodiinn Stegs
Yumber Gradien: Stops
Number of Gridien Steps
Figure 4: Fraction of gradient in tangent space evolving over the course of training. Mean and 90% confidence bounds. Gray line shows the expected fraction of a random vector in tangent space. Logarithmic x-axis. (Left) Training with Adam (Kingma & Ba, 2014). (Center) PGD Adv. Robust Training. (Right) Training with Adam and random labels.
qualitative examples shown on left we see that SmoothGrad correctly focuses on the region away from the pupil to explain the diagnosis but other methods, such as gradients, wrongly focus the pupil to explain the decision. Both the qualitative and quantitative results indicate that SmoothGrad has strong impact in high dimensions.
In conclusion, we find that the fraction of gradient based explanations in the tangent space is much higher than the expectation of fraction of random vectors in the tangent space. We also observe that for complex data manifolds (as described in this section), the alignment of different explanation methods consistent with the literature (i.e., raw gradients being worse than other methods, with SmoothGrad being the best in high-dimensional setup).
4 WHEN AND WHY ARE GRADIENTS ALIGNED WITH THE DATA MANIFOLD?
When and why are model gradients aligned with the data manifold? That there is some degree of alignment is plausible: After all, finite differences between the points in the training data serve rough approximations of the data manifold. For more detailed perspective, we study how the fraction of model gradients in tangent space evolves over the course of training (Figure 4). The left part of Figure depicts the evolution of model gradients over the course of training on the MNIST32 dataset and the model from Section 3.1. First, we see that the fraction of model gradients in tangent space at initialization only as good random. Second, and somewhat surprisingly. we see that the fraction in tangent space increases rapidly during the early steps of training, almost before the model starts to fit the labels (note that the x-axis in Figure 4 is logarithmic). Moreover, the relation between the data manifold and model gradients deteriorates as the model increasingly fits the labels.
4.1 THE EFFECT OF ADVERSARIAL TRAINING
3.0
Robust Gradient
Gradient
Why does the relation between the data manifold and model gradients deteriorate as the model increasingly fits the labels? For insights on this question, turn to training with adversarial perturbations. Previous work has noted that adversarial training aligns model gradients with human perception and the data manifold (Tsipras al., 2018; Kim al., 2019). In line with our hypothesis, we expect that adversarial training increases the fraction of model gradients in tangent space. An evaluation on the MNIST32 dataset shows that this is indeed the case. Fig- ure depicts the fraction of model gradients in tangent space, both for the model from section 3. and for the same architecture trained with projected gra- dient descent (PGD) against an lz-adversary (Madry et al., 2017). Details regarding the adversarial training procedure are in Appendix A. The effect of adversarial training is quite remarkable: the mean fraction of gradients in tangent space is 0.68 for the adversarially trained model, compared with 0.31 for the standard model. The fraction of adversarially robust gradients in tangent space is not being improved by any of the post-hoe explanation methods considered in this paper. As depicted in the center part of Figure the fraction of model gradients in tangent space does not
2.0
Density
0.5
0²
0²
10
Fraction in tangent space
Figure 5: Adv. Robust Gradients.
7

---

Under review as a conference paper at ICLR 2022
deteriorate under adversarial training, but instead increases monotonically. We hypothesize that the observed behavior under standard training is due to the presence of non-robust features that are not aligned with the data manifold (Ilyas et al., 2019).
4.2 TRAINING WITH RANDOM LABELS
There are two different ways in which a model might align its gradients with the data manifold: (1) The model leverages information about the data manifold in an unsupervised way. After all, the data manifold is an unsupervised object that exists independently of the labels. (2) The model leverages information about the data manifold implicitly through the labels. After all, natural image classi- fication problems, class boundaries might be implicitly aligned with the data manifold. We attempt to distinguish between these two different possibilities by training with random labels. The result of this experiment is depicted in the right part of Figure 4. First, the fraction of gradients in tangent space increase even when training with random labels. Second, it reaches its peak just before the network starts to fit the labels. Third, as the network increasingly (over-)fitt the data, the relation between model gradients and the data manifold deteriorates until remains hardly better than ran- dom. This experiment shows that the model does indeed, at least in part, leverage information about the data manifold in an unsupervised way. However, it also indicates that this relation might be hard sustain if the labels unrelated to the structure of the data manifold.
GENERALIZATION DOES NOT IMPLY ALIGNMENT WITH THE MANIFOLD
We have already seen empirically that the relation between model gradients and the data manifold can exhibit considerable local variation (Figures 2 and 3). This is the case even for models that achieve an accuracy > 99% (Section 3.1). In this section, we are going to formally prove that a single neural network that achieves a test accuracy of 100% can exhibit an arbitrary amount of variation between its gradients and the data manifold (Theorem 1). This is illustrated in Figure 6. Figure 6 depicts the fraction of model gradients tangent space for a two-layer neural network trained solve classification problem that lives 1-dimensional data manifold in RS For this network, the distribution of model gradients in tangent space is bi bi-modal with two modes near 0 and 1. This means that (a) for 50% of observations, model gradients lie within the tangent space of the data manifold, and (b) for the remaining 50% of observations, model gradients are orthogonal to the tangent space of the data manifold.
To formally prove this result, we leverage the recently demonstrated connections between the training dynamics of infinite width neural net- works and Wasserstein gradient flow (Chizat & Bach, 2018; 2020). In particular, Chizat & Bach (2020) show that the limit of Wasser- stein gradient flow, conditional on convergence, takes the form of a maximum merimum--maggin classifier
Gradient
8.5
tangent space
arg max min y- (v, (2) DEP(34+1) (2.0)E
Figure 6: M3 and Simulation Results.
where ((v, x)) B (,))), wp(( a, .r) +b) and
6(x)= max xz 0}. We show that there exists manifold and corresponding classification such that the gradient of (2) is partly perfectly and partly not at all aligned with the manifold.
Theorem (Generalization does not imply alignment with the manifold). For every dimension > 1, there exists a I-dimensional manifold Ma C. Rd that does not lie in any proper subspace, a probability distribution 'D Ma 1- and maximum-mant margin classifier with zero test error given by equation (2) such that
(of(v,x)) dx
(af(Vr))) dx
P(a.a)~D
E Ts
> 0.49
and
P(aaa)-D
T
> 0.49.
The proof is in appendix B, follow Shah al. (2021). It would have been equally possible to provide examples where the gradient is always or never aligned with the data manifold. We choose this partieular example because believe that it is related to what observed in practice.
8

---

Under review as a conference paper at ICLR 2022
6 RELATED WORK
Explanation methods. Methods such as layer conductance (Dhamdhere et al., 2018), internal influ- ence (Leino et al., 2018), and GradCam (Selvaraju et al., 2017) attempt to attribute model predictions to a certain layer and are aptly called layer attribution methods. The work in (Singla et al., 2020) proposed a method to explain the classifier by generating visual perturbation of an image by exag- gerating or diminishing the semantic features that the classifier associates with target label. Work on causal explanation such as (Schwab & Karlen, 2019) formulates the problem of explaining machine learning model as causal learning task that estimates the influence of certain inputs on the outputs of another machine learning model. Moreover, work such as (Yang & Song, 2020) uses models based on differentiable inductive logic programming that learns first-order logic to explain the data. The work in (Yeh al., 2020) proposes method to find set of concepts that are "suffi cient" to explain predictions. Koh et al. (2020) build a model where concepts are built-in so that one can control influential concepts.
Evaluating explanations. It is important to objectively assess how good an explanation is. How- ever, unavailability of ground-truth explanations makes it difficult to evaluate them (Samek et al., 2021). Evaluation is further complicated by the fact that explanations may not be reliable as they are susceptible to adversarial attacks (Heo et al., 2019; Dombrowski et al., 2019). Some of the existing work in this direction focus on (i) Faithfulless//Sfffiiinnn on faithfulness propose that an explanation should reliably and comprehensively represent the local decision structure of the analyzed ML model. A practical technique to assess such property of the model, in case of image data, is "pixel-flipping (Samek et al., 2016), it tests whether removing the features highlighted by the explanation (as most relevant) leads to a strong decay of the net- work prediction abilities. Works on human interpretability emphasize that the presented explanation techniques deliver results that are meaningful to the human (Samek et al., 2021). It is difficult to de- fine human interpretabilityy as different users may have different capabilities at reading explanations and at making sense of the features that support them (Ribeiro al., 2016; Narayanan al., 2018 88; Miller, 2019).
and (ii) Human interpretabiliiy. Works
DISCUSSION
Recently, there has been lot of discussion about post-hoc explanation methods, with some authors claiming that they work relatively well (Samek et al., 2021), while others assert that they should not be used at all (Rudin, 2019). In this work, we focus on the particular case of gradient-bass based input attribution methods and propose necessary criterion: explanations should be aligned with the tangent space of the data manifold.
The objective of this paper is not to claim that the gradients of existing models provide good ex- planations or that any particular post-hoc explanation method works especially well. Instead, our goal is to convince the reader that the manifold hypothesis for gradient-based explanations deserves consideration, and to start exploring the potential implications of this hypothesis. We hope that our hypothesis can provide a novel perspective on gradient-based explanations, and to serve a criterion for future investigations. As we demonstrate in Section 5, the question of whether model gradients are aligned with the data manifold is amendable to theoretical analysis. Future work might ask under what conditions adversarial training causes model gradients to be aligned with the data manifold, or when Integrated Gradients can of improve the fraction of model gradients in tangent space. More- over, while current models and algorithms provide only imperfect alignment, it an open question whether this is due to the fact that we have not yet found the right algorithm, or because the problem harder than classification.
Although we are, to the best of our knowledge, the first to provide a systematic evaluation of our hypothesis, we believe that it is implicit in previous works. Projections of model gradients on esti- mates of the tangent space have long been performed in the literature (Kim et al., 2019; Dombrowski et al., 2019). Our criterion might also be implicitly related to other sanity checks for explanations (Adebayo et al.. 20)8 For example, since model gradients random initialization to the data manifold, an explanation that does not change when model weights are being randomized can also not lie in the tangent space of the data manifold.
unrelated
9

---

Under review as a conference paper at ICLR 2022
8 ETHICS AND REPRODUCABILITY STATEMENTS
Explainability is an important demand when it comes to applications of machine learning, for ex- ample medicine or in societal contexts. This has been stressed by the European Data Protection Regulation, and will be an important aspect of future Al legislation. However, the field of ex- plainable machine learning is still widely open, and formal guarantees about explanation algorithms rarely exist. Our work takes one step towards understanding conceptually how particular explana- tion methods work. Many more such steps will be necessary before we actually understand which kind of guarantees can possibly be given for explanation algorithms.
Regarding our experimental setup: The properties of the data manifolds of natural image datasets depend data collection practices and other design choices made during dataset creation. In this work, we use standard open-sourced datasets that do not require ethical approval.
All details regarding experimental setup be found in appendix A. The same is true for all model architectures, as well as the details of neural network training. Python code for computing the tangent space and model architectures can be found in appendix D. A formal proof of Theorem is in appendix B.
REFERENCES
Eddie Aamari and Clément Levrard. Nonasymptotic rates for manifold, tangent space and curvature estimation. The Annals of Statistics, 2019.
Julius Adebayo, Justin Gilmer, Ian Goodfellow, and Been Kim. Local explanation methods for deep neural networks lack sensitivity to parameter values. arXiv preprint arXiv: 201 8a.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. arXiv preprint urXiv:1810.0329) 20.
Sushant Agarwal, Shahin Jabbari, Chirag Agarwal, Sohini Upadhyay, Zhiwei Steven Wu, and Himabindu Lakkaraju. Towards the unification and robustness of perturbation and gradient based explanations. arXiv preprint arXiv:2102.10611 2021.
Amine Amyar, Romain Modzelewski, Hua Li, and Su Ruan. Multi-task deep learning based imaging analysis for covid-19 pneumonia: Classification and segmentation. Computers in Biology and Medicine, 2020.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient- attribution methods for deep neural networks. International Conference on Learning Representations, 2018.
Christopher Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus-Robed Müller, and Pan Kessel. Fairwashing explanations with off-manifold detergent. In International Conference Machine Learning. 2020.
Nishanth Arun, Nathan Gaw, Praveer Singh, Ken Chang, Mehak Aggarwal, Bryan Chen, Katharina Hoebel, Sharut Gupta, Jay Patel, Mishka Gidwani, et al. Assessing the (un) trustworthiness of saliency maps for localizing abnormalities in medical imaging, arXiv preprint arXiv:2008..2211 027666 2020.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On wixel-wise explanations for non-linear classifier decisions by layer-wise wise relevance propagation. PLOS ONE, 2015.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in 3-vae. arXiv preprint arXiv:1804.03500 0...80 2018.
Lawrence Cayton. Algorithms for manifold learning. Univ. of California at San Diego Tech. Rep, 2005.
10

---

Under review as a conference paper at ICLR 2022
Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. This looks like that: deep learning for interpretable image recognition. arXiv preprint arXiv:1806.100 10574, 2018a.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in variational autoencoders. arXiv preprini arXiv: /80t:.0409.9 2018b.
Mohamed Chetoui and Moulay A Akhloufi. Explainable diabetic retinopathy using efficientnet. In IEEE Engineering in Medicine & Biology Society (EMBC), 2020.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-
parameterized models using optimal transport. arXiv preprint arXiv: 09545, 2018.
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, 2020.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN, 2017.
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is neuron? arXiv preprint arXiv: 12233, 2018
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry blame. Neural Information Processing Systems, 2019.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 2016.
Damien Garreau and Dina Mardaoui. What does lime really see in images? arXiv:2102.06307, 2021.
arXiv preprint
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
//www. www.:ddelleee. ea rigooooook org.
ngbook.
Seyyed Hossein Hasanpour, Mohammad Rouhani, Mohsen Fayyaz, and Mohammad Sabokrou, Lets keep simple, using simple architectures outperform deeper and more complex architectures. arXiv preprint arXiv:
2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE conference on computer vision and pattern recognition, 2016.
Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adver- sarial model manipulation. Neural Information Processing Systems, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint arXiv: 1905.02 2019.
Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang. Xiaokang Wu, Fangbing Yan, et al. Identifying medical diag- noses and treatable diseases by image-based deep learning. Cell, 172(55)1222-1133 2018.
Beomsu Kim, Junghoon Seo, and Taegyun Jeon. Bridging adversarial robustness and gradient inter- pretability. arXiv preprint 1rXivv1903.. 1626, 2019.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dühne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. 2019.
11

---

Under review as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.698)/ 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
arXiv:1312.6114 2013.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEFE, 1998.
Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, and Linyi Li. Influence-directed explana- tions for deep convolutional networks. In 2018 IEEE International Test Conference, 2018.
Dohun Lim, Hyeonseok Lee, and Sungchan Kim. Building reliable explanations of unreliable neu- ral networks: Locally smoothing perspective of model interpretation. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Juan Eduardo Luján- Cornelio Yáfiez-Márquez Yenny Villuendas- -eyu, and Oscar Camacho- Nieto. A transfer learning method for pneumonia classification and visualization. Applied Sci- ences, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0660 2017.
Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial intel- ligence, 2019.
Menaka Narayanan, Emily Chen, Jeffrey He, Been Kim, Sam Gershman, and Finale Doshi-Veled How do humans understand explanations from machine learning systems? an evaluation of the human- iitmaniiterrretabblityy of explanation. arXiv preprint arXiv:1802.00682. 2018.
Sivaramakrishnan Rajaraman, Sema Candemir, George Thoma, and Sameer Antani. Visualizing and explaining deep learning predictions for pneumonia detection in pediatric chest radiographs. In Medical Imaging 2019: Computer-Aided Diagnosis, 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 2019.
Wojeiech Samck, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 2016.
Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher Anders, and Klaus- Robert Müller. Explaining deep neural networks and beyond: A review of methods and applica- tions. Proceedings of the IEEE, 2021.
Rory Sayres, Ankur Taly, Ehsan Rahimy, Katy Blumer, David Coz, Naama Hammel, Jonathan Krause, Arunachalam Narayanaswamy, Zahra Rastegar, Derek Wu, et al. Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy. Oph- thalmology, 2019.
Patrick Schwab and Walter Karlen. Cxplain: Causal explanations for model interpretation under uncertainty. Neural Information Processing Systems, 2019.
12

---

Under review as a conference paper at ICLR 2022
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based local- ization. In IEEE International Conference Computer Vision, 2017.
Harshay Shah, Prateek Jain, and Praneeth Netrapalli. Do input gradients highlight discriminative features? arXiv preprint arXiv:2102.1 222220022....... 12781, 2021.
Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deep generative models. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference Machine Learning, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In In Workshop at International Con- ference on Learning Representations, 2014.
Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by pro-
gressive exaggeration. In International Conference on Learning Representations, 2020.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint wxxvv:706.03825. 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, 2017.
Natasa Tagasovska, Damien Ackerer, and Thibault Vatter. Copulas as high-dimensional generative models: Vine copula autoencoders. arXiv preprint arXiv: 2019.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint 1rXiv:1805.121:. 2018.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03888, 2020.
Toon Van Craenendonck, Bart Elen, Nele Gerrits, and Patrick De Boever. Systematic comparison of heatmapping techniques in deep learning in the context of diabetic retinopathy lesion detection. Translational vision science & technology, 2020.
Kilian Q Weinberger and Lawrence K Saul. Unsupervised learning of image manifolds by semidef-
inite programming. International journal of computer vision, 2006.
Yuan Yang and Le Song. Learn explain efficiently via neural logic inductive learning. In Interna- tional Conference on Learning Representations, 2020.
Chib-Kuaa Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-awe concept-based explanations in deep neural networks. Neural Information Processing Systems, 2020.
13

---

Under review as a conference paper at ICLR 2022
A MODEL ARCHITECTURES AND TRAINING DETAILS
A.I MNIST32
We first describe the creation of the MNIST32 dataset. We autoencoded the original MNIST dataset with a 3-TCVAE (Chen et al.. 2018b) and the same architecture Burgess et al. (2018). The hyperparameters were
We use pettps:/ggitaee com/ YannDubn/DDDDDD/ d diseannggl wenlllng-vvee vae. On the reconstructed images, we trained SimpleNet-V1 to replicate the original labels (Hasanpour et al., 2016). Training with Adam and learning rate of 1e allowed to replicated the labels of the test images with accuracy of 96%. To increase the quality of the generated images, we additionally applied rejection sampling based on the the softmax score of the class predicted by the SimpleNet. Every sample from the autoencoder was accepted with probability pa Psoftmax Random samples from the MÑIST32 dataset are depicted the left part of Figure 7.
? 1,
On the MNIST32 dataset, trained the default model architecture from https: : / gitttub. com/ com/pytorch exampless espp/es////e reee master r /m i st. We trained for 50 epochs with Adam, an initial learning rate of 1e 4 and learning rate decay of le 1 after 10 epochs each.
Adversarially robust training on MNIST32 was perfomed as follows. We trained the same model ar- chitecture against an lz-adversary with projected gradient descent (PGD). For each gradient step, the size of the adversarial perturbation was randomly chosen from [1, 4, 8] and we took 100 iterations with step size of 2 /100 each (Madry et al., 2017).
To overfit the MNIST32 dataset with random labels, we disabled the dropout layers of the neural network. We then trained for 3000 epochs with Adam, and intial learning rate of 1e 4 and learning rate decay of le- - 1 after 1000 epochs each.
A.2 MNIST256
To create the MNIST256 dataset, we appended a bilinear upsampling layer to the decoder that was used to generate the MNIST32 dataset. Note that bilinear upsampling is differentiable, which required to compute the tangent spaces. Random samples from the MNIST256 dataset are depicted in the right part Figure 7.
On the MNIST256 dataset, we trained a ResNet18 for 50 epochs with Adam, an initial learning rate of le - 2 and learning rate decay of after 10 epochs each.
A.3 EMNISTI28
The EMNIST dataset is a set of handwritten character digits derived from the NIST Special Database 19 and converted to 28 x 28 pixel image format and dataset structure that directly matches the MNIST dataset. We used the dataset as available from PyTorch https: : htpp//ttorcci worgg.orh.ooooooooooooo..o.ooo.r vis aassion/taalll /attaaatsssseteeeee hhtml emnist. The images were resized to 128 128 make it a high-dimensional problem and we used a subset consisting of 60 classes (in contrast to other experiments where number of classes are typically low). We trained autoencoder using the recon- struction approach and encoder-decoder architecture as shown in Section D using Adam optimizer with learning rate set to le-4, decayed over 200 epochs using cosine annealing. We then train VGG network to perform the classification, in a similar manner.
A.4 CIFAR10
The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. We use the dataset available directly from the PyTorch dataloaders as described here https: httpttttt.orrg...o v a i on stabl e/ datasets html fcifar. To learn the manifold, we use the reconstruction approach using the autoencoder shown in Section D with the latent dimension set to be 144 with /k/d = 0.20. We use the Adam optimizer with learning rate set to lc-4 decayed using cosine annealing over 200 epochs to learn the autoencoder. We then use the VGG16 to learn a classifier again trained using Adam optimizer with learning rate set to le-4 decayed using cosine annealing.
14

---

Under review as a conference paper at ICLR 2022
A.5 PNEUMONIA DETECTION
The original dataset at https://www.
contains high-resolution chest X-ray images with classes: Normal and Pneumonia (with pneumonia being of two types viral and bacterial, but within class distinction of pneumonia is not considered in this problem). The problem posed as a binary classification problem to decide between the normal and an abnormal class (pneumonia). The images were resized to 256 224 (i.e., 57344 dimensional data) and the autoencoder shown in Section D is used to learn the manifold of the images where the latent dimension is reduced to 8 X 28 x 32 (i.e., 7168) with v Vk/d 0.20, we then fine tune a Resnet18 model (previously trained on ImageNet) to perform the classification using learning rate of lc-4, decayed with cosine annealing over 200 epochs and using Adam optimizer.
A.6 DIABETIC RETINOPATHY DETECTION
The original 3 channel (RGB) fundus image dataset at https: /www. kaggle com / c/ diabetic-ret
ion contains 5 classes with varying degrees of diabetic retinopathy. We posed the problem as binary classification problem to decide between the normal and an abnormal class. The images were resized to 3 x 224 x 224 (i.e., 150528 dimensional data) and the autoencoder shown in Section D is used to learn the manifold of the images where the latent dimension is reduced to 8 28 x 28 (i.e., 6272) with vk/d 0.20, we then fine tune 1 model (previously trained on ImageNet) to perform the classification using learning rate of 2e-4, decayed with cosine annealing over 150 epochs and using Adam optimizer.
B PROOF OF THEOREM
Proof. Let d > 1. We begin by defining the manifold. Let
44dd [4(d-2) 2) if d is even
mmxx
(3)
44d if d is odd.
For E (mxx). consider the continuous curve f(t) that walks along the edges of the shifted hypercube, alternating between the first and other dimensions
(-1/2 + (t L(),,
[4]), I....,, 0.....0) )
[4] mod 4 = 0
[4/2] d-1-10/21
(1/2, 1, 1, [t], 0 0....,0
if [t] mod 4 = 1
[t/2] d-2 [4/2]
f(t)
(4)
(1/2 (t- [t]), ب.ايي ye.r.
if [t] mod 4 =
[t/2] d-1-le/2j
(-1/2, I,...,,, 1,, [t], 0 0.....0 )r
if [4] mod 4 =
[1/2]
d-2- [1/2]
In all dimensions, ft) starts at (-1/2, 0, 0).. If d is even, f(t) ends (-1/2, 1,.... 1, 1 If d is odd, f(t) ends at (-1/2, 1, 1)a. In even dimensions, connect the end- point (-1/2 1,0) to the starting point via straight lines to the corner points (Vd- - 1/2, 1,...,1,07) (vd 1/2,1, 1..., 1) and 1-1/2,1,.... 1)
odd dimensions, connect the endpoint - 1/2,1 1)a to the starting point via straight lines to the comer points 1/2,
2/3)5, (-1/2 + vd - 1/2,, 1/3),, (-1/2,1/3, 1/3)a. The whole point of this construction is to obtain closed connected curve that does not lie in any proper subspace and that walks exactly half of the time along the first coor- dinate, and the rest of the time orthogonal to it. By smoothing the corners of this connected curve, obtain smooth connected manifold M.
(-1/2,2/3, 2/3).
15

---

Under review as a conference paper at ICLR 2022
Let Uet be the uniform distribution on M.. Let Dr be given by (A) ((44)=UM(A/MM))) UUM(A/MMMMMUUM(((( where M = (ir E M [21| < e}. Let the label given by = sign 21. The sep-
arating hyperplane with maximum margin is 21 0. We claim that V- fò0z + to0;,
0
V2' V2(1+107:00 O,....... O, (2(1+62)
01 = V2 ( 0, 0, v2(1+62)
is max-
imizer of
arg max min V f(u, x). vEP(S (x.j)ED
(S)
By Proposition 12 in (Chizat & Bach, 2020), we have to show that there exists a measure p' on M (the support vectors) such that
Support(v") E
arg max ((x.)))) (v wwll a, x) + 6)) (w.0,,))844)
(6)
and
Support(p") E arg min (m,,a,, (y wp((a, x)) + b)).
(7)
(u.v)ED
We claim that p* is given by
1 D" 2 -0 +50000....0.
(8)
We first show (6). It holds that
w (6(a 100006((aa0 6(-aa(e+000
B(z.g)>p' (y . wol( a, x) + b))
We differentiate two cases. Note that 0h achieves an objective larger than zero, hence d1 + 0.
Case 1, 0. aj O, then o(a b) o(-afc + b). This implies b 2 0 and b S ajc. The maximization problem can then be written as
2 01E + =
max ara b
subject to
w² a b² =1
< b < are
a>>0 0.
For small enough, the unique solution is given by b 01 w 0g The objective +2).
V2 and a1 V2(1+o)) i.e. by
Case 2, < O. a < 0, then Φ(are + < 06-016 + b). This implies h 2 0 and b 5 -016. The maximization problem can now written as
2 ai + b
w, b subject to
2 + a + 6² =1
0 5 b < -afe
a1 < 0.
For small enough, the unique solution is given by 6, w
J/2
and 01
i.e. by
v 2
07. The objective again (1/21+++), This shows (6).
We now show (7). Explicit computation shows
a (y we(( a, x)) + b))
22" Φ
I1 + 2(1 + +2)
y ① v2"
Ii
/2(1 + (2)
For y 1, 2 and the second term vanishes. The minimum is then attained iff 21 For
y = - 1, x E and the first term vanishes. The minimum is then attained iff x = -c. This proves
(7).
16

---

Under review as a conference paper at ICLR 2022
We now compute the gradient of We have
( -01 +e
1
21 +
1 ///
[(()
(9)
v2
2 + (2)
V2 2 + c2)
Thus, for 1,
0f(v*, 0.
(10)
dz,
For = 1,
(f xx) 1
(11)
dx 2 2
Thus, the gradient of f is constant and multiple of (1, .1.0.....)) 0)
Except at the corners, the tangent space is either given by span ( (1, ,..,0....00.............. 0)*) or orthogonal to
(1,0,..., 0)* 0.5, that we can smooth the corners regions of arbitrarily small measure, and by choosing arbitrarily small.
The proof is completed by noting that it is orthogonal to (1, 11. 0)* with probability
C PLOTS AND FIGURES
C.I FIGURE CREATION DETAILS
Figures and were created by normalizing both vectors to unit norm and trimming the pixels with the largest 0.005% absolute values. Images within the same column lie within the same color space, iec equal colors imply equal values along the respective coordinates.
To create Figure 3 use the following function which overlays the explanation maps on top of the images.
def overlay_saliency
rec, smap, alpha=0.5, thr=0. 75, sigma=10,
wt__owwigh=00211
smap (smap-np. min (smap)) ) (np. max (smap) mp.ming (smap)) smap = mp.mean(smap (smap, axis-2)
mask smap > thr*np.max (smap)
smap high mask*smap
smap bg (1-mask) smma
#*
imap_high coussian_fitted (smap_high, sigma)
mmap_high (smap_high-np. min (smap_high) Inp.max (smap_high) np min
(smap_high))
10
smap_bg (smap_bg paaap_g--np... min
min (smap_bg))/ (ts.n.nnnnnmmm.n (np. max (smap_bg) ippminn (snap_bg
)
smap = wt www.ighssmappppp.o....pp....p bg (1-wt_lowhigh) +smap_high
smap (smap-np. min (smap) (np. (smap) np np.min (smap))
13
smap np. up.uint8 (smap+255)
14
#
15
smap - cv2. u22.applyCollrtte (smap, col wwt.apppcv.c....MA...... (smap, C01.CCOOO_BGGH2GGG smap cv2. (smap,,,
17
18
rec (11_
19
smap = cv2. wwwWeighttdeddeoooooood..i.dd... (il rec, alpha, smap, 1-alpha, 0)
return smap
Listing 1: Function used to create Figure 3.
C.2 ADDITIONAL PLOTS AND FIGURES
17

---

Under review as a conference paper at ICLR 2022
Omn
Daw
Oen
CM
8
3
7
2
Gwe
Con
Omn
Gme
7
2
2
4
)
6
3
6
6
3
4
a
7
7
2
8
S
5
6
4
3
0
7
7
2
9
9
8
8
5
3
0
5
9
Figure 7: Random samples from the generated datasets. Left: MNIST32. Right: MNIST256.
a
3
4
5
6
7
6
9
Figure 8: Images from the test of the synthetic MNIST256 task. The second depicts the part of the gradient that lies in tangent space. The third row depicts the part of the gradient that is orthogonal the tangent space. Blue corresponds negative attribution, red positive attribution.
D CODE LISTINGS
def compute_tangent__ee (NetAE, device=/ cuda');
=** compute the tangent space of generative model
NetAE: The decoder. A pytorch medule that implements decode (2) 2: pytorch tensor (latent vector)
vectors that span the tangent space (tangent space dim, model output dim). These vectors correspond 1:1 to the latent dimensions of 2.
Returns:
10
batch dimension in 2 is not supported. has to be 1-dimensional vector" NetAE.to (device)
assert len (en(z.shape
1,
11
z.to (device)
latent_dis
z.shape [0]
14
2. wereuiire__grae True
15
out RecAdd (2)
16
out = out squeeze ()
reneve singleton batch dimension
put_appp out.shape out cut reshape (-1)
store original output shape
18
and transform the output into vector
18

---

Under review as a conference paper at ICLR 2022
tangent__span space torch. zeros (latent_ out. shape )0]))
20
for range (out sout.shape [01):
21
out backkaack (raaaaigrrtttnnnnaa---t-
22
tangent_spacel [;, i] = 2.grad
v.grad.zero_!.
tangent_space tangent_spacee reshape (-1, <output_shape)) space in model output shape
tangent
25
return tangent_space
Listing 2: Code used compute the tangent space of generative model.
class MKIST32Model (nn. Monuuee
def _niit_
super (MNIST32Mode,, sclf) init () sclf.convl nn. Cnnvon (1, 32, 3, 1)
self.conv2 nn.Conv2d (32, 64, 3, 1)
self.dropout nn.Dropout (0.25)
self.dropaut2 nn.Dropout (0.5)
seli.icl Inneir (12544, 128)
seli.ic2 nn. Linear (128, 10)
10
11
def forward (self, x):
(x)
13
r.relu(x)
14
X self.com conv.ccov20 (x)
15
x = F.relu
F.max_pon (sax__pol2d 2)
sclf. arofoutopouti (x)
18
X torch. flatten (x, 1)
19
X self.fel
20
X F.relu(x)
(x)
self. fel
23
return
Listing 3: The model trained on MNIST32.
class ResBlock (nn (nn.Module):
def init__ in_channel, channel)
super () init_
self.conv nn. Sn. nn. ReLU nn. Co.o22ddiin chaanneehnn, channel, 3, padding=1),
10
(channee, in in_channel, 1),
11
12
def (self, input): : out self.conv (input)
14
15
out +- input
16
return out
Listing 4: Convolutional residual block.
class Enceder (nn, Module)
def init_ (self, in_channel, channel, n_res_block, n_res_channel stride):
super init ()
elif stride -- 8:
blacks
19

---

Under review as a conference paper at ICLR 2022
nn.Convd (in__canne____ channel//2, 4, stride=2, padding=1)
nn (i..eclU((n))llc--Trr)) mm.Conv2d(channc//// channel//2, 4, stride=2, padding=1)
nn. (m.Reell((nplllace1Trre)))
11
nn, Cm.Conv2 channel, 4, stride-2, padding=i),
12
nn. (m.Reel
nn. Cono22 channei, 3, padding=1),
15
16
elif stride 4:
17
blocks - nn. Cn.von (in_chanei channel//2, 4, stride=2, padding=1)
19
nn Rm. (ingleeri)n)ll
20
nn ( channel, 4, stride=2, padding=1),
21
nn ( cmnoo2 channel, 3, padding-1),
24
25
for i in range in_res_bloccee cck)) blocks. (ResBlock (channei, rere_ccannell)
26
28
blocks... plocks.append (nn. ReLU (inplace=True))
29
30
self.blocks Sequennntial (4blocks)
def (sewf, input)
33
return self.blosh (input)
Listing 5: Convolutional encoder.
class Deceder (nn, (nn, Module)
def _init_((
self, in_channel, out_channel, channel, n_res_ neres aares_channee_ stride ):
super () _init_
blocks [nn.Conv2d] (in_channel, channel, 3, padding-1) ]
for i range (n. res res__r_________ block) res_block))
10
blocks. append (ResBlock (channel, n_res_channell)
11
blocks. append Ren. (Lnpaaco-rr))))
13
14
elif stride 8: blocks.extend
15
acCoorrraaasccee2tcchmeeeelll channel//2, 4, stride=2,
padding=1),
18
nn (i.eeeuiinp)))ce-rrree nn. ww..Coovirannppsse2lccccmmmeee// channel//2, 4, stride
19
-2, padding-1),
20
nn Rei.linnllle--rrre)
21
nn ConVTraagscccee2d
22
channel//2. out_channel,, 4, stride=2, padding=1
25
26
27
elif stride == 4: Blacks.extend
20

---

Under review as a conference paper at ICLR 2022
mcCCovTraaaspceeecccaane.. channel//2, 4, stride=2,
padding=1),
31
nn ReLJ (Inpaccccrrr
32
nn. won.Transscsss22l
channel//2, out_channel, 4, stride-2, padding=1.
35
36
37
self blockk Sequentiia (+blocks)
40
def (self,, input):
41
return self.block (input)
Listing 6: Convolutional decoder.
class AutcEncoder (nn.Module))
def _inii_it____ in oh_channel, eut_channel, channel, n_res_bloos n_res_channel, stride): super ()ntt)tt_i_n__t
self.E Encoder (in_channel, channel, n_res_block, n_res_channel, stride)
self.0 Decoder (channel, cut_channe), channel, n_res_block, n_res_channel, stride)
def (sew, x)
(x) X_rec self.D(2)
10
11
return x_ree,
12
def encode (self, x): Self.E
15
16
def decode (self, 2):
17
return Sel
Listing Convolutional autoencoder used estimate data manifolds.
21


Reviews:
[{"review_id": "jZOV5Zn9c9V", "reviewer": "Reviewer_993V", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* The main question in the paper is interesting: Does explainability has something to do with alignment of the explanation map with data manifold.\n\n* The paper's central claim about meaningfulness is not quantifiable (at least with experiments in this paper) and, as a result, not falsifiable. The figures shown in the paper can only show that the gradient-based explanation is more aligned with data manifold than random, but the main argument about the meaningfulness of a method vs another method is very subjective. \n\n* Also, there is no clear trend between different gradient-based explanation methods. Perhaps the only trend is that explanation methods are more aligned with data manifold than the gradient, which is interesting but hardly conclusive about the paper's central claim. \n\n* The authors observe the alignment increases and decreases with epochs. This phenomenon can happen for many reasons, and I am not sure how it has anything to do with explainability. \n\n* Perhaps, the most exciting part of the paper is the negative theoretical results at the end: alignment with data manifold has nothing to do with generalizability. However, this theory has not much to do with the central claim.", "summary_of_the_paper": "The paper argues that the main reason (or a good reason) for the \"meaningfulness\" of a gradient with data manifold. The authors perform a set of controlled experiments with different feature attribution methods. Finally, they theoretically show that alignment of the gradient with data manifold has nothing to do with generalizability. ", "summary_of_the_review": "* The paper reads a set of subjective observations about the meaningfulness of explanation and relationship with data manifold + tangential theory. The paper does not have a coherent story. However, the central question is interesting. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "e4WxInWK_yu", "reviewer": "Reviewer_AMGx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the experimental design is explained in detail. Much of the evaluation relies on informal observation of gradients, and the examples appear to be carefully picked. In Figure 2, many of the digits are slightly ambiguous and exhibit artifacts that highlight the explanation. The qualitative difference between on-manifold and orthogonal components appears consistent and convincing. In Figure 3 it is difficult to say if the measured fraction of on-manifold component correlates with quality of explanation, except maybe for the top rows with CIFAR10. Judging if the lower rows make sense would require expertise in diagnosing pneumonia or retinopathy, which I believe to be rare in the reviewer pool.\n\nI am somewhat concerned by how the relative ordering of various explanation methods changes between different variants of MNIST. How certain can we be that this ordering is not due to random chance? I would expect that training the same VAE multiple times with different random initializations could result in quite different latent spaces, and these might favor different explanation methods. In other words, how robust are the measured distributions such as those shown in Figure 2 and 3, and the related explanation figures, to the inherent randomness in training? This should be measured in order to assure the reader that the differences are real and consistent.\n\nFormula 1 calculates the cosine of the angle between the vectors v1 and grad. I don’t think it is appropriate to call this a “fraction”, because a similar computation between v2 and grad (corresponding to the sine of the angle between v1 and grad) and this fraction do not sum to 1. Squaring the formula would correspond to the length of projection of v1 onto grad relative to length of grad, and would seem like a perhaps more appropriate choice. \n\nThe adversarial training test in Section 4.1 is very interesting and convincing. I have no opinion of the usefulness of the theoretical result in Section 5.\n\nSection 3.1 heading has a typo \"graysacle\".\n", "summary_of_the_paper": "The paper constructs a synthetic classification task with a known manifold structure by training the classifier with data from a variational autoencoder with a low-dimensional latent space. The paper argues that the components of image gradients that lie in the tangent space of the data manifold are semantically meaningful, whereas the part orthogonal to the image manifold is nonsensical. The experiments in the paper support this hypothesis to an extent. This is an interesting, although not unexpected, conclusion.", "summary_of_the_review": "The paper constructs a clever setup to test its central hypothesis and provides some convincing results that the hypothesis holds true. However, there is no analysis of stochastic variation in the quantitative results, so they may not hold water as well as the central tenet that the gradients in line with the tangent space of the data manifold are qualitatively different from gradients orthogonal to it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "U1k6YQTu9R", "reviewer": "Reviewer_oTn6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- The paper is well-written; the empirical and theoretical results are easy-to-follow. The evaluation metric and data manifold construction are clearly explained. The hypothesis is well-posed and relevant to the high-level problem of evaluation metrics for instance-specific explanation methods. \n\n- The generative approach to create datasets with a completely known manifold structure is interesting. By training a new model on this dataset (with known manifold), this approach sidesteps the possible mismatch between the \"true\" manifold and the estimated manifold. It would be great to have some discussion on Algorithm 1 (tangent space computation with this approach). The discussion of why a reconstructive approach is needed for high-dimensional datasets (k/d argument) is insightful as well. \n\n- The experiments on real-world datasets are quite thorough. Section 3 evaluates four different gradient-based methods (and the random vector baseline) on five real-world datasets, using multiple architectures. The results consistently show that raw gradients are worse than gradient-based methods such as smoothgrad. Also, figure 2 clearly shows how the out-of-manifold component of explanations looks  less \"meaningful\" than the on-manifold component of explanations.\n\n- The experiment on the fraction of gradient in tangent over the course of training is novel and interesting. The observation that the fraction of gradient in tangent space increases rapidly and then slightly deteriorates is quite surprising. However, It would be good to sanity-check whether this phenomenon holds on larger non-MNIST datasets. \n\n**Weaknesses (in order of significance)**\n\n- Insufficient evaluation of explanation meaningfulness/correctness/quality. To test the proposed hypothesis, it is necessary to test whether explanations that are better aligned with the data manifold are more \"meaningful\". While there are multiple experiments to test data manifold alignment, the paper uses qualitative visual inspection to evaluate explanation meaningfulness/correctness/quality. It is now well known that qualitative visual inspection is subjective and misleading. Explanation methods that are known to output visually sharp saliency maps often fail basic sanity checks [R1]. Instead of visual assessment, evaluation metrics such as ROAR [R2] (not cited) and DiffROAR [R3] can be used to quantitatively test and compare the correctness / quality of explanation methods. The current evaluation vis-a-vis explanation quality is insufficient to reliably test the hypothesis. \n\n- Limited novelty: First, [R4] (not cited) and [R5] (cited but not in this context) use generative models such as VAE-GANs to obtain a \"learned\" data manifold in order to evaluate whether gradient-based adversarial perturbations and raw gradients (resp.) are close to the data manifold / tangent space. Second, similar to results in Section 5, [R5] show that adversarially robust model's raw gradients are better aligned with the data manifold. [R3] show that robust model's raw gradients have better explanation quality. \n\n- Section 4 (when and why are gradients aligned with the data manifold) shows that (i) adversarial training improves alignment between explanations and data manifold and (ii) evaluates the effect of training with random labels. However, the section title is misleading because it does not study the \"why\" aspect. For example, there is no discussion on why adversarial training improves alignment. \n\n- Section 5 (generalization does not imply alignment) does not justify the choice of the dataset or 1-dimensional manifold design that is used in the theoretical analysis. What is the design principle behind this synthetic dataset? Is it representative (to some extent) of natural data distributions / real datasets considered in previous sections? \n\n**Clarifications and questions**\n\n- Why is the hypothesis restricted to gradient-based explanations? Can explanations not based on gradients (e.g. occlusion-based saliency maps) be meaningful if they are orthogonal to the data manifold? \n\n- The results in Section 4 suggest that integrated gradients and input times gradient are better than raw gradients, as they are better aligned with the tangent space. This seems to possibly contradict previous findings [R1,R6] that show that unlike raw gradients, integrated gradients and input x gradients fail basic sanity checks. \n\n- \"If a gradient-based explanation approximately lies in tangent space...contribute to prediction\" (section 1): This statement is a bit unclear. Based on how I understood it, I am not sure that it is fully correct. If an explanation approximately lies in the tangent space, it may still lack fidelity w.r.t. the prediction rules learned by the model. For example, it is possible that an explanation that lies in the tangent space can highlight some component (e.g. texture of object in image) that is different from the components (e.g. shape and location of object) of the image that the model employs for its predictions.\n\n--- \n\n[R1] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M. and Kim, B., 2018. Sanity checks for saliency maps. arXiv preprint arXiv:1810.03292.\n\n[R2] Hooker, S., Erhan, D., Kindermans, P.J. and Kim, B., 2018. A benchmark for interpretability methods in deep neural networks. arXiv preprint arXiv:1806.10758.\n\n[R3] Shah, H., Jain, P. and Netrapalli, P., 2021. Do Input Gradients Highlight Discriminative Features?. arXiv preprint arXiv:2102.12781.\n\n[R4] Stutz, D., Hein, M. and Schiele, B., 2019. Disentangling adversarial robustness and generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6976-6987).\n\n[R5] Kim, B., Seo, J. and Jeon, T., 2019. Bridging adversarial robustness and gradient interpretability. arXiv preprint arXiv:1903.11626.\n\n[R6] Yang, M. and Kim, B., 2019. Benchmarking attribution methods with relative feature importance. arXiv preprint arXiv:1907.09701.", "summary_of_the_paper": "This paper studies the following hypothesis: \"Gradient-based explanations are more meaningful the more they are aligned with the tangent space of the data manifold\". The work has three contributions: (i) an autoencoder-based approach to estimate data manifolds of five datasets in order to evaluate the alignment between explanations and tangent space of the data; (ii) analysis of the alignment between explanations and tangent space during training; (iii) theoretical analysis to show that generalization does not imply alignment with data manifold. ", "summary_of_the_review": "Overall, the weaknesses of the paper outweigh its strengths. While the hypothesis is well-posed and the experiments are thorough, some major weaknesses are: (i) insufficient/misleading evaluation of explanation correctness/quality,  (ii) limited novelty vis-a-vis methodology and results on robust models, (iii) missing discussion on why robustness improves alignment,  justification of synthetic dataset and theory, and connection to previous work on sanity checks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AQRQaNtWcda", "reviewer": "Reviewer_gHy3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) **No definition of \"meaningful\" explanation makes core hypothesis unverifiable**: The main drawback of this paper is that it fails to quantify or define what it means for an explanation to be \"meaningful\", which is central to the hypothesis presented in the paper. Without such a definition, it is impossible to verify the correctness of the hypothesis. For example, is an explanation more meaningful if is more structured? If so then it is plausible that for highly structured domains such as images, the tangent space is also similarly structured and hence \"meaningful\". However, when the underlying domain itself is unstructured, what constitutes a meaningful explanation? Note that it is perfectly fine if \"meaningful\" explanations exist only for highly structured domains, and it is important nonetheless to define these terms precisely to be able to verify their correctness. It is also unclear if this \"meaningfullness\" is distinct from the so-called \"faithfullness\" condition, where the explanation method must accurately reflect model behaviour? For instance, an explanation can be highly structured but be unrelated to the underlying model. How do we guard against these issues in a coherent definition of \"meaningful\" explanations ?\n\n2) **No quantitative metrics to measure saliency map quality**: Similar to point 1, the paper does not compute any quantitative metrics regarding the quality of saliency maps, besides the alignment of saliency map with the tangent space. The hypothesis in the paper is that tangent-space-aligned saliency maps are higher in quality, and the experiments in the paper demonstrate simply that some saliency maps are more aligned to the tangent space than others, but the question remains - are they of higher quality according to some well-defined metric? Unfortunately the experiments do not answer this question and this is yet another major drawback of the paper. \n\n3) **Clarification regarding gradient direction within the subspace**: In the setting proposed in the paper, the tangent space is a k-dimensional subspace of R^d. However the gradient corresponds to a single direction within this subspace. It is unclear to me whether any specific direction within this subspace must be preferred or any direction is equally good? Some discussion on this would be illuminating.\n\n4) **Missing highly related reference**: The paper misses reference to a highly related work - Srinivas & Fleuret, \"Rethinking the role of gradient-based attribution methods for model interpretability\", ICLR 2021. Both papers are similar in that they hypothesize that discriminative models seem to have a generative modelling / manifold learning component which ensures that the gradients are related to the underlying data distribution. However they also present different hypotheses in the sense that Srinivas&Fleuret state that model gradients are \"interpretable\" if they are aligned to the gradients of the data distribution, whereas this paper posits that gradients are aligned to the tangent space of the data manifold. The above paper also shows that pre-softmax gradients can be arbitrarily structured, which seems related to section 5 of the current paper \"generalization does not imply alignment with the manifold\". Overall I think this is an important point to discuss and compare the two hypotheses presented in both papers. On a related note it would be nice to present visualizations like Figure2 where the gradient components are presented in the normal space and tangent space, but for image datasets such as CIFAR10/100.\n\n5) **Nice experimental approach**: On a more positive note, I like the approach taken by this paper to verify its hypothesis - by explicitly generating data that lies on a manifold. I also like Figure 4, which shows how such alignment changes during training. This seems to point to some form of approximate manifold learning being performed by the model implicitly. Figure 5 is also very interesting, as it shows the dramatic shifts that adversarial training can produce. I'm wondering whether similar observation holds true for simple gradient norm regularization, which is also shown to boost robustness?", "summary_of_the_paper": "This paper makes the hypothesis that gradient-based explanations are \"meaningful\" if they are aligned with the tangent space of the underlying data manifold. Experiments in the paper compute this alignment for various explanation methods and adversarially trained models, and conclude that smoothgrad, integrated gradients and adversarially trained models generally produce gradients that are more aligned with the tangent space.", "summary_of_the_review": "Overall, while I certainly think that this paper makes a hypothesis that is interesting and is at least partly true, it does not make its definitions (see point 1) and experiments (point 2) precise, which makes it impossible to prove or disprove the hypothesis. I would be willing to accept this work only when the paper makes a more clearly stated hypothesis, and designs similarly clear experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the following hypothesis that gradient-based explanations are more meaningful the more they are aligned with the tangent space of the data manifold. The reviews are negative overall. The general feeling is that the paper reads like a set of subjective observations about the meaningfulness of explanation and relationship with data manifold + tangential theory. There isn’t a coherent story.
Decision: Reject