The Manifold Hypothesis for Gradient-Based Explanations

Abstract: When are gradient-based explanations meaningful? We propose a necessary criterion: explanations need to be aligned with the tangent space of the data manifold. To test this hypothesis, we employ autoencoders to estimate and generate data manifolds. Across a range of different datasets -- MNIST, EMNIST, CIFAR10, X-ray pneumonia and Diabetic Retinopathy detection -- we demonstrate empirically that the more an explanation is aligned with the tangent space of the data, the more interpretable it tends to be. In particular, popular post-hoc explanation methods such as Integrated Gradients and SmoothGrad tend to align their results with the data manifold. The same is true for the outcome of adversarial training, which has been claimed to lead to more interpretable explanations. Empirically, alignment with the data manifold happens early during training, and to some degree even when training with random labels. However, we theoretically prove that good generalization of neural networks does not imply good or bad alignment of model gradients with the data manifold. This leads to a number of interesting follow-up questions regarding gradient-based explanations.



Reviews and author responses:

 We thank the reviewers for providing detailed reviews and comments on our paper.  

The reviews note that we do not quantitatively evaluate the "meaningfulness” of explanations to humans. While we believe that our paper contains empirical evidence in support of our hypothesis, we recognize that quantification of the meaningfulness of explanations would be desirable. In addition to comparing the fraction of the gradient in tangent space to other notions and sanity checks like ROAR, as suggested in the reviews, we have decided to conduct human-centric evaluations of the meaningfulness of explanations. We will however not be able to provide these evaluations until the paper revision deadline. 

We agree with Reviewer gHy3 that an explanation might be highly structured and thus appear “meaningful” to humans while not being faithful to the model. This is likely the case for many directions in the tangent space of the data manifold. For this reason we decided to formulate our hypothesis only as a necessary criterion, which allows us to remain relatively agnostic about the relation to other criteria. We do not have an opinion as to which regions in the tangent space should be preferred – this might almost amount to a necessary criterion.  

We thank Reviewer AMGx for pointing out a number of robustness checks regarding the relative ordering of various explanation methods. They will be included in a future version of the paper. 

Regarding the relation between our experiments and theory section, a main motivation for our paper is to come up with a criterion for explanations that is amenable to rigorous theoretical analysis. Broadly speaking, we are interested in analyzing the statistical and computational complexity of “explaining”. We think that analyzing whether model gradients are aligned with the tangent space of the data manifold might provide a first way to tackle such questions.  

We also thank the reviewers, especially Reviewer oTn6, for pointing out a number of related references. We were aware of Srinivas & Fleuret (2021) before the submission, however working our the relation between their approach and ours is not straightforward. We will, however, relate to this work in a future version of the paper.


This paper makes the hypothesis that gradient-based explanations are "meaningful" if they are aligned with the tangent space of the underlying data manifold. Experiments in the paper compute this alignment for various explanation methods and adversarially trained models, and conclude that smoothgrad, integrated gradients and adversarially trained models generally produce gradients that are more aligned with the tangent space. 1) **No definition of "meaningful" explanation makes core hypothesis unverifiable**: The main drawback of this paper is that it fails to quantify or define what it means for an explanation to be "meaningful", which is central to the hypothesis presented in the paper. Without such a definition, it is impossible to verify the correctness of the hypothesis. For example, is an explanation more meaningful if is more structured? If so then it is plausible that for highly structured domains such as images, the tangent space is also similarly structured and hence "meaningful". However, when the underlying domain itself is unstructured, what constitutes a meaningful explanation? Note that it is perfectly fine if "meaningful" explanations exist only for highly structured domains, and it is important nonetheless to define these terms precisely to be able to verify their correctness. It is also unclear if this "meaningfullness" is distinct from the so-called "faithfullness" condition, where the explanation method must accurately reflect model behaviour? For instance, an explanation can be highly structured but be unrelated to the underlying model. How do we guard against these issues in a coherent definition of "meaningful" explanations ?

2) **No quantitative metrics to measure saliency map quality**: Similar to point 1, the paper does not compute any quantitative metrics regarding the quality of saliency maps, besides the alignment of saliency map with the tangent space. The hypothesis in the paper is that tangent-space-aligned saliency maps are higher in quality, and the experiments in the paper demonstrate simply that some saliency maps are more aligned to the tangent space than others, but the question remains - are they of higher quality according to some well-defined metric? Unfortunately the experiments do not answer this question and this is yet another major drawback of the paper. 

3) **Clarification regarding gradient direction within the subspace**: In the setting proposed in the paper, the tangent space is a k-dimensional subspace of R^d. However the gradient corresponds to a single direction within this subspace. It is unclear to me whether any specific direction within this subspace must be preferred or any direction is equally good? Some discussion on this would be illuminating.

4) **Missing highly related reference**: The paper misses reference to a highly related work - Srinivas & Fleuret, "Rethinking the role of gradient-based attribution methods for model interpretability", ICLR 2021. Both papers are similar in that they hypothesize that discriminative models seem to have a generative modelling / manifold learning component which ensures that the gradients are related to the underlying data distribution. However they also present different hypotheses in the sense that Srinivas&Fleuret state that model gradients are "interpretable" if they are aligned to the gradients of the data distribution, whereas this paper posits that gradients are aligned to the tangent space of the data manifold. The above paper also shows that pre-softmax gradients can be arbitrarily structured, which seems related to section 5 of the current paper "generalization does not imply alignment with the manifold". Overall I think this is an important point to discuss and compare the two hypotheses presented in both papers. On a related note it would be nice to present visualizations like Figure2 where the gradient components are presented in the normal space and tangent space, but for image datasets such as CIFAR10/100.

5) **Nice experimental approach**: On a more positive note, I like the approach taken by this paper to verify its hypothesis - by explicitly generating data that lies on a manifold. I also like Figure 4, which shows how such alignment changes during training. This seems to point to some form of approximate manifold learning being performed by the model implicitly. Figure 5 is also very interesting, as it shows the dramatic shifts that adversarial training can produce. I'm wondering whether similar observation holds true for simple gradient norm regularization, which is also shown to boost robustness? Overall, while I certainly think that this paper makes a hypothesis that is interesting and is at least partly true, it does not make its definitions (see point 1) and experiments (point 2) precise, which makes it impossible to prove or disprove the hypothesis. I would be willing to accept this work only when the paper makes a more clearly stated hypothesis, and designs similarly clear experiments. 

The paper argues that the main reason (or a good reason) for the "meaningfulness" of a gradient with data manifold. The authors perform a set of controlled experiments with different feature attribution methods. Finally, they theoretically show that alignment of the gradient with data manifold has nothing to do with generalizability.  * The main question in the paper is interesting: Does explainability has something to do with alignment of the explanation map with data manifold.

* The paper's central claim about meaningfulness is not quantifiable (at least with experiments in this paper) and, as a result, not falsifiable. The figures shown in the paper can only show that the gradient-based explanation is more aligned with data manifold than random, but the main argument about the meaningfulness of a method vs another method is very subjective. 

* Also, there is no clear trend between different gradient-based explanation methods. Perhaps the only trend is that explanation methods are more aligned with data manifold than the gradient, which is interesting but hardly conclusive about the paper's central claim. 

* The authors observe the alignment increases and decreases with epochs. This phenomenon can happen for many reasons, and I am not sure how it has anything to do with explainability. 

* Perhaps, the most exciting part of the paper is the negative theoretical results at the end: alignment with data manifold has nothing to do with generalizability. However, this theory has not much to do with the central claim. * The paper reads a set of subjective observations about the meaningfulness of explanation and relationship with data manifold + tangential theory. The paper does not have a coherent story. However, the central question is interesting. 

This paper studies the following hypothesis: "Gradient-based explanations are more meaningful the more they are aligned with the tangent space of the data manifold". The work has three contributions: (i) an autoencoder-based approach to estimate data manifolds of five datasets in order to evaluate the alignment between explanations and tangent space of the data; (ii) analysis of the alignment between explanations and tangent space during training; (iii) theoretical analysis to show that generalization does not imply alignment with data manifold.  **Strengths**

- The paper is well-written; the empirical and theoretical results are easy-to-follow. The evaluation metric and data manifold construction are clearly explained. The hypothesis is well-posed and relevant to the high-level problem of evaluation metrics for instance-specific explanation methods. 

- The generative approach to create datasets with a completely known manifold structure is interesting. By training a new model on this dataset (with known manifold), this approach sidesteps the possible mismatch between the "true" manifold and the estimated manifold. It would be great to have some discussion on Algorithm 1 (tangent space computation with this approach). The discussion of why a reconstructive approach is needed for high-dimensional datasets (k/d argument) is insightful as well. 

- The experiments on real-world datasets are quite thorough. Section 3 evaluates four different gradient-based methods (and the random vector baseline) on five real-world datasets, using multiple architectures. The results consistently show that raw gradients are worse than gradient-based methods such as smoothgrad. Also, figure 2 clearly shows how the out-of-manifold component of explanations looks  less "meaningful" than the on-manifold component of explanations.

- The experiment on the fraction of gradient in tangent over the course of training is novel and interesting. The observation that the fraction of gradient in tangent space increases rapidly and then slightly deteriorates is quite surprising. However, It would be good to sanity-check whether this phenomenon holds on larger non-MNIST datasets. 

**Weaknesses (in order of significance)**

- Insufficient evaluation of explanation meaningfulness/correctness/quality. To test the proposed hypothesis, it is necessary to test whether explanations that are better aligned with the data manifold are more "meaningful". While there are multiple experiments to test data manifold alignment, the paper uses qualitative visual inspection to evaluate explanation meaningfulness/correctness/quality. It is now well known that qualitative visual inspection is subjective and misleading. Explanation methods that are known to output visually sharp saliency maps often fail basic sanity checks [R1]. Instead of visual assessment, evaluation metrics such as ROAR [R2] (not cited) and DiffROAR [R3] can be used to quantitatively test and compare the correctness / quality of explanation methods. The current evaluation vis-a-vis explanation quality is insufficient to reliably test the hypothesis. 

- Limited novelty: First, [R4] (not cited) and [R5] (cited but not in this context) use generative models such as VAE-GANs to obtain a "learned" data manifold in order to evaluate whether gradient-based adversarial perturbations and raw gradients (resp.) are close to the data manifold / tangent space. Second, similar to results in Section 5, [R5] show that adversarially robust model's raw gradients are better aligned with the data manifold. [R3] show that robust model's raw gradients have better explanation quality. 

- Section 4 (when and why are gradients aligned with the data manifold) shows that (i) adversarial training improves alignment between explanations and data manifold and (ii) evaluates the effect of training with random labels. However, the section title is misleading because it does not study the "why" aspect. For example, there is no discussion on why adversarial training improves alignment. 

- Section 5 (generalization does not imply alignment) does not justify the choice of the dataset or 1-dimensional manifold design that is used in the theoretical analysis. What is the design principle behind this synthetic dataset? Is it representative (to some extent) of natural data distributions / real datasets considered in previous sections? 

**Clarifications and questions**

- Why is the hypothesis restricted to gradient-based explanations? Can explanations not based on gradients (e.g. occlusion-based saliency maps) be meaningful if they are orthogonal to the data manifold? 

- The results in Section 4 suggest that integrated gradients and input times gradient are better than raw gradients, as they are better aligned with the tangent space. This seems to possibly contradict previous findings [R1,R6] that show that unlike raw gradients, integrated gradients and input x gradients fail basic sanity checks. 

- "If a gradient-based explanation approximately lies in tangent space...contribute to prediction" (section 1): This statement is a bit unclear. Based on how I understood it, I am not sure that it is fully correct. If an explanation approximately lies in the tangent space, it may still lack fidelity w.r.t. the prediction rules learned by the model. For example, it is possible that an explanation that lies in the tangent space can highlight some component (e.g. texture of object in image) that is different from the components (e.g. shape and location of object) of the image that the model employs for its predictions.

--- 

[R1] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M. and Kim, B., 2018. Sanity checks for saliency maps. arXiv preprint arXiv:1810.03292.

[R2] Hooker, S., Erhan, D., Kindermans, P.J. and Kim, B., 2018. A benchmark for interpretability methods in deep neural networks. arXiv preprint arXiv:1806.10758.

[R3] Shah, H., Jain, P. and Netrapalli, P., 2021. Do Input Gradients Highlight Discriminative Features?. arXiv preprint arXiv:2102.12781.

[R4] Stutz, D., Hein, M. and Schiele, B., 2019. Disentangling adversarial robustness and generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6976-6987).

[R5] Kim, B., Seo, J. and Jeon, T., 2019. Bridging adversarial robustness and gradient interpretability. arXiv preprint arXiv:1903.11626.

[R6] Yang, M. and Kim, B., 2019. Benchmarking attribution methods with relative feature importance. arXiv preprint arXiv:1907.09701. Overall, the weaknesses of the paper outweigh its strengths. While the hypothesis is well-posed and the experiments are thorough, some major weaknesses are: (i) insufficient/misleading evaluation of explanation correctness/quality,  (ii) limited novelty vis-a-vis methodology and results on robust models, (iii) missing discussion on why robustness improves alignment,  justification of synthetic dataset and theory, and connection to previous work on sanity checks.

The paper constructs a synthetic classification task with a known manifold structure by training the classifier with data from a variational autoencoder with a low-dimensional latent space. The paper argues that the components of image gradients that lie in the tangent space of the data manifold are semantically meaningful, whereas the part orthogonal to the image manifold is nonsensical. The experiments in the paper support this hypothesis to an extent. This is an interesting, although not unexpected, conclusion. The paper is well written and the experimental design is explained in detail. Much of the evaluation relies on informal observation of gradients, and the examples appear to be carefully picked. In Figure 2, many of the digits are slightly ambiguous and exhibit artifacts that highlight the explanation. The qualitative difference between on-manifold and orthogonal components appears consistent and convincing. In Figure 3 it is difficult to say if the measured fraction of on-manifold component correlates with quality of explanation, except maybe for the top rows with CIFAR10. Judging if the lower rows make sense would require expertise in diagnosing pneumonia or retinopathy, which I believe to be rare in the reviewer pool.

I am somewhat concerned by how the relative ordering of various explanation methods changes between different variants of MNIST. How certain can we be that this ordering is not due to random chance? I would expect that training the same VAE multiple times with different random initializations could result in quite different latent spaces, and these might favor different explanation methods. In other words, how robust are the measured distributions such as those shown in Figure 2 and 3, and the related explanation figures, to the inherent randomness in training? This should be measured in order to assure the reader that the differences are real and consistent.

Formula 1 calculates the cosine of the angle between the vectors v1 and grad. I don’t think it is appropriate to call this a “fraction”, because a similar computation between v2 and grad (corresponding to the sine of the angle between v1 and grad) and this fraction do not sum to 1. Squaring the formula would correspond to the length of projection of v1 onto grad relative to length of grad, and would seem like a perhaps more appropriate choice. 

The adversarial training test in Section 4.1 is very interesting and convincing. I have no opinion of the usefulness of the theoretical result in Section 5.

Section 3.1 heading has a typo "graysacle".
 The paper constructs a clever setup to test its central hypothesis and provides some convincing results that the hypothesis holds true. However, there is no analysis of stochastic variation in the quantitative results, so they may not hold water as well as the central tenet that the gradients in line with the tangent space of the data manifold are qualitatively different from gradients orthogonal to it.

Meta-review: This paper studies the following hypothesis that gradient-based explanations are more meaningful the more they are aligned with the tangent space of the data manifold. The reviews are negative overall. The general feeling is that the paper reads like a set of subjective observations about the meaningfulness of explanation and relationship with data manifold + tangential theory. There isn’t a coherent story.

Decision: Reject