The Manifold Hypothesis for Gradient-Based Explanations

1 INTRODUCTION . A large number of algorithms aim to provide post-hoc explanations for the output of neural networks ( Simonyan et al. , 2014 ; Bach et al. , 2015 ; Shrikumar et al. , 2017 ; Ancona et al. , 2018 ; Lim et al. , 2021 ) . Many of them are , directly or indirectly , based on the gradient with respect to the input ( Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ; Garreau & Mardaoui , 2021 ; Agarwal et al. , 2021 ) . A particularly interesting application for gradient-based input attribution methods is natural image classification . Despite recent attempts to provide a priori interpretable image classifiers , neural networks remain exceptionally successful at classification ( Chen et al. , 2018a ) . However , recent work has demonstrated that post-hoc explanation methods fail various sanity checks ( Adebayo et al. , 2018b ; a ; Kindermans et al. , 2019 ; Arun et al. , 2020 ) , and some have even suggested that they should not be used at all ( Rudin , 2019 ) . In this paper , we try to understand when and why gradient-based input attribution methods can be meaningful . To this end , we propose the following hypothesis : Gradient-based explanations are more meaningful the more they are aligned with the tangent space of the data manifold . Consider image classification . It is widely believed that natural image data concentrates around a low-dimensional image manifold ( Goodfellow et al. , 2016 , Section 5.11.3 ) . This image manifold captures the geometric structure of the data . In particular , the tangent space captures all components of an image that can be changed while still staying within the realm of natural images . If a gradientbased explanation approximately lies in the tangent space , this means that it highlights a meaningful way in which the different components of an image contribute to the prediction . If a gradient based explanation lies orthogonal to the tangent space , this means that it points in some direction that would not lead to realistic images , and a human would have a hard time to understand its meaning . Our motivation for proposing this hypothesis is twofold . First , we believe that it is intuitive , and we provide empirical evidence in its support . Second , we hope that the hypothesis can provide a perspective on why obtaining gradient-based explanations might be more difficult than classification . To evaluate the hypothesis empirically , we employ autoencoders to estimate the data manifolds of five different datasets - MNIST , EMNIST , CIFAR10 , X-ray pneumonia and diabetic retinopathy detection . As depicted in Figure 1 , we also use variational autoencoders as generative models . This allows us to generate datasets with completely known manifold structure . With this approach , we provide qualitative and quantitative evidence that explanations that are more aligned with the tangent space of the data are more interpretable . To study when and why model gradients are aligned with the tangent space of the data , we first show that the gradients of neural networks at initialization are unrelated to the structure of the data manifold . This means that the learning algorithm picks up some aspects of the structure of the data manifold during training . We show that this happens early during training , and to some extent even when training with random labels . Moreover , under standard training procedures , the alignment between model gradients and the data manifold deteriorates as the model increasingly fits the labels . This is avoided by l2 adversarial training , which significantly aligns the model ’ s gradients with the tangent space of the data . Is it always the case that a neural network that generalizes necessarily adapts its gradients to the data manifold , at least to some degree ? The answer is no , as we show theoretically . Without further assumptions , the relation between data manifold and model gradients is ambiguous : The alignment between the two quantities can be arbitrarily good or bad . The organization of the paper is as follows . Sec.2 formally introduces the manifold hypothesis and outlines our conceptual approach . Sec.3 evaluates the hypothesis on five datasets . Sec.4 discusses the effects of adversarial training and the evolution of model gradients over the course of training . Sec.5 contains a formal proof that generalization does not imply alignment with the data manifold , Sec.6 discusses the related work and Sec.7 discusses the implications of our results . 2 THE MANIFOLD HYPOTHESIS . Our goal is to evaluate the following hypothesis : A gradient-based explanation E ∈ Rd at a point x ∈ M is more meaningful the more it is aligned with the tangent space of the data manifold at x . Below we first give a background on data manifolds , tangent spaces and model gradients ; then we detail our evaluation approach . 2.1 BACKGROUND . Data manifolds and tangent spaces . A k-dimensional differentiable manifold M ⊂ Rd is a subset of a d-dimensional space that locally resembles Rk . At every point x ∈M , the tangent space Tx is a k-dimensional subspace of Rd . The tangent space Tx consists of all directions v such that x+ v , for ‖v‖ small , is again close to the manifold . Manifolds and tangent spaces are the subject of differential geometry , to which we refer for a comprehensive introduction . The long-standing hypothesis that natural image data concentrates around a low-dimensional image manifold is supported by a number of empirical studies ( Weinberger & Saul , 2006 ; Fefferman et al. , 2016 ) . However , accurately learning the data manifolds of natural image datasets – manifold learning – is difficult and the exact properties of these manifolds remain unknown ( Cayton , 2005 ; Aamari & Levrard , 2019 ) . Shao et al . ( 2018 ) investigate the properties of manifolds generated by deep generative models and find that they have mostly low curvature . Model gradients and the fraction of the gradient in tangent space . We consider neural networks that learn differentiable functions f : Rd → RC . Here C is the number of classes and the model prediction is given by argmaxi f ( x ) i . The gradient of class i with respect to the input is given by gradi ( x ) = ∂ ( f ( x ) i ) ∂x . Unless mentioned otherwise , we always consider the model gradient with respect to the predicted class and before the softmax . At every point x that lies on the data manifoldM , we can decompose the gradient into a part that lies in tangent space and a part that is orthogonal to it . Formally , we have gradi ( x ) = v1 + v2 with v1 ∈ Tx , v2 ∈ T ⊥x and v1 ⊥ v2 . Here v1 is the part of the gradient that lies in the tangent space , and v2 is the part of the gradient that is orthogonal to the tangent space . If the gradient completely lies in the tangent space , we have v2 = 0 . If the gradient is completely orthogonal to the tangent space , we have v1 = 0 . In practice , some part of the gradient will lie in the tangent space and another part be orthogonal to it , that is we have v1 6= 0 and v2 6= 0 . To quantitatively measure how well the gradient is aligned with the tangent space , we compute the Fraction of the Gradient in Tangent Space = ‖v1‖ ‖gradi ( x ) ‖ ∈ [ 0 , 1 ] . ( 1 ) 2.2 HOW DO WE KNOW THE DATA MANIFOLD ? . To estimate whether an explanation is aligned with the tangent space we make use of autoencoders . The various variants of variational autoencoders ( Kingma & Welling , 2013 ; Higgins et al. , 2017 ) , allow to estimate the data manifolds of existing datasets . Importantly , they also allow to generate datasets with completely known manifold structure ( Algorithm 1 ) . We make use of two related approaches that we term the generative approach and the reconstructive approach . In both approaches , we first train an autoencoder on the orignal dataset . The generative approach to create datasets with a completely known manifold structure . To generate a dataset with completely known manifold structure , we have to train a variational- or another generative autoencoder ( Tagasovska et al. , 2019 ) . After training , we pass the original dataset through the autoencoder . Then we train an additional classifier that reproduces the original labels from latent codes and reconstructed images . Equipped with this labeling function , we sample from the prior and use decoder and labeling function to generate a dataset . If the decoder is differentiable , we can compute the tangent space at each datapoint x ( Shao et al. , 2018 ; Anders et al. , 2020 ) . The reconstructive approach to create datasets with an estimated manifold structure . The main limitation of the generative approach is that we might not be able to obtain high-quality samples with reasonably small latent spaces . While there have been great advances in generative modeling , stateof-the-art models like hierarchical variational autoencoders ( Vahdat & Kautz , 2020 ) require very large latent spaces , i.e . k ≈ d. For our analysis , it is however critical that √ k/d is small – with k = d , the fraction of the gradient in tangent space is always 1 . To evaluate our hypothesis on realworld high-dimensional image data where it is difficult to obtain realistic samples with not-too-large latent spaces , we rely on estimating the tangent space . That is we simply pass the original dataset through the autoencoder and take the reconstucted images with the original labels as our new dataset . 3 PUTTING THE HYPOTHESIS TO THE TEST . Explanation algorithms . We consider four gradient-based input attribution methods : The gradient ( Simonyan et al. , 2014 ) , Integrated Gradients ( Sundararajan et al. , 2017 ) , Input × Gradient ( Ancona et al. , 2018 ) , and SmoothGrad ( Smilkov et al. , 2017 ) . The motivation behind Integrated Gradients is axiomatic . The motivation behind SmoothGrad is to reduce noise in the gradient . All four methods provide explanations as vectors in Rd . We can evaluate how each method is aligned with the tangent space of the data manifold by computing the fraction of the explanation method in tangent space . While other methods also provide explanations as vectors in Rd , we restrict ourselves to these four methods because they are directly related to the gradient with respect to the input , which is our main object of investigation . Experimental setting . Given a dataset , obtained either with the generative or the reconstructive approach , we train a neural network to minimize the test error . For this network , we then evaluate how gradients and other gradient-based explanation methods relate to the data manifold . To evaluate whether an explanation is meaningful , we use qualitative evaluations as demonstrated in ( Simonyan et al. , 2014 ; Bach et al. , 2015 ; Sundararajan et al. , 2017 ; Smilkov et al. , 2017 ) . We also rely on the literature that demonstrates the utility of Integrated Gradients and SmoothGrad for diabetic retinopathy detection ( Sayres et al. , 2019 ; Van Craenendonck et al. , 2020 ) . When we quantitatively evaluate the fraction of an explanation in tangent space ( 1 ) , we need to account for the fact that even a random vector has a non-zero fraction in tangent space . A random vector is by definition completely unrelated to the structure of the data manifold . The expected fraction of a random vector that lies in any k-dimensional subspace is √ k/d . In our MNIST32 task , for example , d = 1024 , k = 10 and √ 10/1024 ≈ 0.1 . Thus , we could only say that a gradientbased explanation is systematically related to the tangent space of the data manifold if , on average , the fraction of the explanation in tangent space is significantly larger than 0.1 . Datasets . We evaluate our hypothesis several datasets . This includes ( i ) MNIST32 and MNIST256 , two variants of the MNIST dataset ( LeCun et al. , 1998 ) with 10 classes and 60000 grayscale training images and 10000 grayscale test images of size 32 × 32 and 256 × 256 , respectively . ( ii ) EMNIST128 , a variant of the EMNIST dataset ( Cohen et al. , 2017 ) that extends MNIST with handwritten letters and has over 60 classes , ( iii ) the CIFAR10 dataset ( Krizhevsky et al. , 2009 ) . We also evaluate our hypothesis on two real world high dimensional image datasets : X-ray Pneumonia ( Kermany et al. , 2018 ) and Diabetic Retinopathy Detection 1 . Both tasks have been used before to study the properties of post-hoc explanation methods ( Rajaraman et al. , 2019 ; Luján-Garcı́a et al. , 2020 ; Amyar et al. , 2020 ; Arun et al. , 2020 ; Chetoui & Akhloufi , 2020 ; Van Craenendonck et al. , 2020 ) . All further details on the datasets are provided in appendix A .

Summary: The paper argues that the main reason (or a good reason) for the "meaningfulness" of a gradient with data manifold. The authors perform a set of controlled experiments with different feature attribution methods. Finally, they theoretically show that alignment of the gradient with data manifold has nothing to do with generalizability.