The Manifold Hypothesis for Gradient-Based Explanations

1 INTRODUCTION . A large number of algorithms aim to provide post-hoc explanations for the output of neural networks ( Simonyan et al. , 2014 ; Bach et al. , 2015 ; Shrikumar et al. , 2017 ; Ancona et al. , 2018 ; Lim et al. , 2021 ) . Many of them are , directly or indirectly , based on the gradient with respect to the input ( Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ; Garreau & Mardaoui , 2021 ; Agarwal et al. , 2021 ) . A particularly interesting application for gradient-based input attribution methods is natural image classification . Despite recent attempts to provide a priori interpretable image classifiers , neural networks remain exceptionally successful at classification ( Chen et al. , 2018a ) . However , recent work has demonstrated that post-hoc explanation methods fail various sanity checks ( Adebayo et al. , 2018b ; a ; Kindermans et al. , 2019 ; Arun et al. , 2020 ) , and some have even suggested that they should not be used at all ( Rudin , 2019 ) . In this paper , we try to understand when and why gradient-based input attribution methods can be meaningful . To this end , we propose the following hypothesis : Gradient-based explanations are more meaningful the more they are aligned with the tangent space of the data manifold . Consider image classification . It is widely believed that natural image data concentrates around a low-dimensional image manifold ( Goodfellow et al. , 2016 , Section 5.11.3 ) . This image manifold captures the geometric structure of the data . In particular , the tangent space captures all components of an image that can be changed while still staying within the realm of natural images . If a gradientbased explanation approximately lies in the tangent space , this means that it highlights a meaningful way in which the different components of an image contribute to the prediction . If a gradient based explanation lies orthogonal to the tangent space , this means that it points in some direction that would not lead to realistic images , and a human would have a hard time to understand its meaning . Our motivation for proposing this hypothesis is twofold . First , we believe that it is intuitive , and we provide empirical evidence in its support . Second , we hope that the hypothesis can provide a perspective on why obtaining gradient-based explanations might be more difficult than classification . To evaluate the hypothesis empirically , we employ autoencoders to estimate the data manifolds of five different datasets - MNIST , EMNIST , CIFAR10 , X-ray pneumonia and diabetic retinopathy detection . As depicted in Figure 1 , we also use variational autoencoders as generative models . This allows us to generate datasets with completely known manifold structure . With this approach , we provide qualitative and quantitative evidence that explanations that are more aligned with the tangent space of the data are more interpretable . To study when and why model gradients are aligned with the tangent space of the data , we first show that the gradients of neural networks at initialization are unrelated to the structure of the data manifold . This means that the learning algorithm picks up some aspects of the structure of the data manifold during training . We show that this happens early during training , and to some extent even when training with random labels . Moreover , under standard training procedures , the alignment between model gradients and the data manifold deteriorates as the model increasingly fits the labels . This is avoided by l2 adversarial training , which significantly aligns the model ’ s gradients with the tangent space of the data . Is it always the case that a neural network that generalizes necessarily adapts its gradients to the data manifold , at least to some degree ? The answer is no , as we show theoretically . Without further assumptions , the relation between data manifold and model gradients is ambiguous : The alignment between the two quantities can be arbitrarily good or bad . The organization of the paper is as follows . Sec.2 formally introduces the manifold hypothesis and outlines our conceptual approach . Sec.3 evaluates the hypothesis on five datasets . Sec.4 discusses the effects of adversarial training and the evolution of model gradients over the course of training . Sec.5 contains a formal proof that generalization does not imply alignment with the data manifold , Sec.6 discusses the related work and Sec.7 discusses the implications of our results . 2 THE MANIFOLD HYPOTHESIS . Our goal is to evaluate the following hypothesis : A gradient-based explanation E ∈ Rd at a point x ∈ M is more meaningful the more it is aligned with the tangent space of the data manifold at x . Below we first give a background on data manifolds , tangent spaces and model gradients ; then we detail our evaluation approach . 2.1 BACKGROUND . Data manifolds and tangent spaces . A k-dimensional differentiable manifold M ⊂ Rd is a subset of a d-dimensional space that locally resembles Rk . At every point x ∈M , the tangent space Tx is a k-dimensional subspace of Rd . The tangent space Tx consists of all directions v such that x+ v , for ‖v‖ small , is again close to the manifold . Manifolds and tangent spaces are the subject of differential geometry , to which we refer for a comprehensive introduction . The long-standing hypothesis that natural image data concentrates around a low-dimensional image manifold is supported by a number of empirical studies ( Weinberger & Saul , 2006 ; Fefferman et al. , 2016 ) . However , accurately learning the data manifolds of natural image datasets – manifold learning – is difficult and the exact properties of these manifolds remain unknown ( Cayton , 2005 ; Aamari & Levrard , 2019 ) . Shao et al . ( 2018 ) investigate the properties of manifolds generated by deep generative models and find that they have mostly low curvature . Model gradients and the fraction of the gradient in tangent space . We consider neural networks that learn differentiable functions f : Rd → RC . Here C is the number of classes and the model prediction is given by argmaxi f ( x ) i . The gradient of class i with respect to the input is given by gradi ( x ) = ∂ ( f ( x ) i ) ∂x . Unless mentioned otherwise , we always consider the model gradient with respect to the predicted class and before the softmax . At every point x that lies on the data manifoldM , we can decompose the gradient into a part that lies in tangent space and a part that is orthogonal to it . Formally , we have gradi ( x ) = v1 + v2 with v1 ∈ Tx , v2 ∈ T ⊥x and v1 ⊥ v2 . Here v1 is the part of the gradient that lies in the tangent space , and v2 is the part of the gradient that is orthogonal to the tangent space . If the gradient completely lies in the tangent space , we have v2 = 0 . If the gradient is completely orthogonal to the tangent space , we have v1 = 0 . In practice , some part of the gradient will lie in the tangent space and another part be orthogonal to it , that is we have v1 6= 0 and v2 6= 0 . To quantitatively measure how well the gradient is aligned with the tangent space , we compute the Fraction of the Gradient in Tangent Space = ‖v1‖ ‖gradi ( x ) ‖ ∈ [ 0 , 1 ] . ( 1 ) 2.2 HOW DO WE KNOW THE DATA MANIFOLD ? . To estimate whether an explanation is aligned with the tangent space we make use of autoencoders . The various variants of variational autoencoders ( Kingma & Welling , 2013 ; Higgins et al. , 2017 ) , allow to estimate the data manifolds of existing datasets . Importantly , they also allow to generate datasets with completely known manifold structure ( Algorithm 1 ) . We make use of two related approaches that we term the generative approach and the reconstructive approach . In both approaches , we first train an autoencoder on the orignal dataset . The generative approach to create datasets with a completely known manifold structure . To generate a dataset with completely known manifold structure , we have to train a variational- or another generative autoencoder ( Tagasovska et al. , 2019 ) . After training , we pass the original dataset through the autoencoder . Then we train an additional classifier that reproduces the original labels from latent codes and reconstructed images . Equipped with this labeling function , we sample from the prior and use decoder and labeling function to generate a dataset . If the decoder is differentiable , we can compute the tangent space at each datapoint x ( Shao et al. , 2018 ; Anders et al. , 2020 ) . The reconstructive approach to create datasets with an estimated manifold structure . The main limitation of the generative approach is that we might not be able to obtain high-quality samples with reasonably small latent spaces . While there have been great advances in generative modeling , stateof-the-art models like hierarchical variational autoencoders ( Vahdat & Kautz , 2020 ) require very large latent spaces , i.e . k ≈ d. For our analysis , it is however critical that √ k/d is small – with k = d , the fraction of the gradient in tangent space is always 1 . To evaluate our hypothesis on realworld high-dimensional image data where it is difficult to obtain realistic samples with not-too-large latent spaces , we rely on estimating the tangent space . That is we simply pass the original dataset through the autoencoder and take the reconstucted images with the original labels as our new dataset . 3 PUTTING THE HYPOTHESIS TO THE TEST . Explanation algorithms . We consider four gradient-based input attribution methods : The gradient ( Simonyan et al. , 2014 ) , Integrated Gradients ( Sundararajan et al. , 2017 ) , Input × Gradient ( Ancona et al. , 2018 ) , and SmoothGrad ( Smilkov et al. , 2017 ) . The motivation behind Integrated Gradients is axiomatic . The motivation behind SmoothGrad is to reduce noise in the gradient . All four methods provide explanations as vectors in Rd . We can evaluate how each method is aligned with the tangent space of the data manifold by computing the fraction of the explanation method in tangent space . While other methods also provide explanations as vectors in Rd , we restrict ourselves to these four methods because they are directly related to the gradient with respect to the input , which is our main object of investigation . Experimental setting . Given a dataset , obtained either with the generative or the reconstructive approach , we train a neural network to minimize the test error . For this network , we then evaluate how gradients and other gradient-based explanation methods relate to the data manifold . To evaluate whether an explanation is meaningful , we use qualitative evaluations as demonstrated in ( Simonyan et al. , 2014 ; Bach et al. , 2015 ; Sundararajan et al. , 2017 ; Smilkov et al. , 2017 ) . We also rely on the literature that demonstrates the utility of Integrated Gradients and SmoothGrad for diabetic retinopathy detection ( Sayres et al. , 2019 ; Van Craenendonck et al. , 2020 ) . When we quantitatively evaluate the fraction of an explanation in tangent space ( 1 ) , we need to account for the fact that even a random vector has a non-zero fraction in tangent space . A random vector is by definition completely unrelated to the structure of the data manifold . The expected fraction of a random vector that lies in any k-dimensional subspace is √ k/d . In our MNIST32 task , for example , d = 1024 , k = 10 and √ 10/1024 ≈ 0.1 . Thus , we could only say that a gradientbased explanation is systematically related to the tangent space of the data manifold if , on average , the fraction of the explanation in tangent space is significantly larger than 0.1 . Datasets . We evaluate our hypothesis several datasets . This includes ( i ) MNIST32 and MNIST256 , two variants of the MNIST dataset ( LeCun et al. , 1998 ) with 10 classes and 60000 grayscale training images and 10000 grayscale test images of size 32 × 32 and 256 × 256 , respectively . ( ii ) EMNIST128 , a variant of the EMNIST dataset ( Cohen et al. , 2017 ) that extends MNIST with handwritten letters and has over 60 classes , ( iii ) the CIFAR10 dataset ( Krizhevsky et al. , 2009 ) . We also evaluate our hypothesis on two real world high dimensional image datasets : X-ray Pneumonia ( Kermany et al. , 2018 ) and Diabetic Retinopathy Detection 1 . Both tasks have been used before to study the properties of post-hoc explanation methods ( Rajaraman et al. , 2019 ; Luján-Garcı́a et al. , 2020 ; Amyar et al. , 2020 ; Arun et al. , 2020 ; Chetoui & Akhloufi , 2020 ; Van Craenendonck et al. , 2020 ) . All further details on the datasets are provided in appendix A .

Summary: The paper constructs a synthetic classification task with a known manifold structure by training the classifier with data from a variational autoencoder with a low-dimensional latent space. The paper argues that the components of image gradients that lie in the tangent space of the data manifold are semantically meaningful, whereas the part orthogonal to the image manifold is nonsensical. The experiments in the paper support this hypothesis to an extent. This is an interesting, although not unexpected, conclusion.