Interpreting Neural Networks Through the Lens of Heat Flow
Authors: Rui Xia
Fan Yang
Liang Sun
Venue: Submitted to ICLR 2023 (2023)
Keywords: interpretable
explanation
heat
Laplacian
attribution
geometry
flow
PDE

Abstract: Machine learning models are often developed in a way that prioritizes task-specific performance but defers the understanding of how they actually work. This is especially true nowadays for deep neural networks. In this paper, we step back and consider the basic problem of understanding a learned model represented as a smooth scalar-valued function. We introduce HeatFlow, a framework based upon the heat diffusion process for interpreting the multi-scale behavior of the model around a test point. At its core, our approach looks into the heat flow initialized at the function of interest, which generates a family of functions with increasing smoothness. By applying differential operators to these smoothed functions, summary statistics (i.e., explanations) characterizing the original model on different scales can be drawn. We place an emphasis on studying the heat flow on data manifold, where the model is trained and expected to be well behaved. Numeric approximation procedures for implementing the proposed method in practice are discussed and demonstrated on image recognition tasks.

Under review as a conference paper at ICLR 2023
INTERPRETING NEURAL NETWORKS THROUGH THE LENS OF HEAT FLOW
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning models are often developed in way that prioritizes task-specific performance but defers the understanding of how they actually work. This is especially true nowadays for deep neural networks. In this paper, we step back and consider the basic problem of understanding a learned model represented as a smooth scalar-valued function. We introduce HeatFlow, a framework based upon the heat diffusion process for interpreting the multi-scale behavior of the model around test point. At its core, approach looks into the heat flow initialized at the function of interest, which generates a family of functions with increasing smoothness. By applying differential operators these smoothed functions, summary statisties (i.e., explanations) characterizing the original model different scales be drawn. We place emphasis on studying the heat flow on data manifold, where the model is trained and expected to be well behaved. Numeric approximation procedures for implementing the proposed method in practice are discussed and demonstrated on image recognition tasks.
INTRODUCTION
In recent years, thanks to the growing availability of computation power and data, together with the rapid advancement of methodology, the machine learning community witnessing the success of creating models with increasingly higher capacity and performance. However, downside of scaling the model complexity that it complicates the understanding of how the learned models work and why sometimes they fail. Such requirements for interpretability arise from both scientífic research and engineering practices. Carefully interpreting the working mechanism a predictive model may help uncover its weakness in robustness, informing further improvements should to be made before deployment in high-stakes decision making..
In this paper, consider the interpretation of scalar-valued smooth functions, basic hypothesis class in machine learning. Models of this type arise naturally in regression and binary classification tasks that deal with continuous input features. Multi-output models, e.g., neural networks for multi- class classification, can be treated as such functions by investigating each output separately. While 1D and 21) such functions can be understood intuitively through graphical visualization, there is no straightforward way to visualize or even imagine general dimens-aimensionall functions. Fortunately, mathematicians developed the derivative to interpret functions in pointwise manner. The directional derivative at a point measures the instantancous rate of change of the function along given direction, and the gradient gives the direction of steepest ascent. This forms the basis of popular gradient-based explanation methods for neural networks (Simonyan et al., 2014; Selvaraju et al., 2017; Sundararajan al., 2017: Smilkov et al., 2017; Ancona et al., 2018; Erion al., 2021; Xu et al., 2020; Hesse et al., 2021; Srinivas & Fleuret, 2021; Kapishnikov et al., 2021).
To interpret the outcome of a learned high-dimensional function at a test point, the gradient there is only part of the story because it just characterizes the first-order behavior of the function in an infinitesimal range. Such extreme localness is to blame for several known pitfalls of vanilla gradient- based interpretation. For example, if the point falls into locally constant region, the gradient will be zero (Shrikumar et al., 2017). On the other hand, the gradient may change dramatically even for nearby points, leading noisy and non-robust explanations in practice (Dombrowski et al., 2019; Wang et al., 2020). Moreover, the gradient will also be zero at different classes of critical points, suggesting the need for higher-order derivatives.

---

Under review as a conference paper at ICLR 2023
BAXI
(a) Heat flow and the dccomposition of Laplacian in two directions.
(b) Interaction strength.
Figure 1: A toy example in I2"
To this end, we introduce HeatFlow, an interpretation framework that enables summarizing the behavior of a Iearned model at different scales from the point of view of test input. Our approach motivated by a natural question to ask about a function f: How much does the value of f at a point x deviate from the average value of f in neighborhood of x? In our opinion, such deviation from local average is more comprehensible than instantaneous rate of change for non-mathematical audience. If the neighborhood is taken be small open ball centered at x, then the answer is related to Ax where A is the Laplace operator, a fundamental second-order differential operator. To consider increasingly larger neighborhoods in a multi- scale manner, we propose solving heat equation, a fundamental partial differential equations (PDE) studied in mathematics and physics.
We show that detailed interpretation for a function of interest may be achieved by extracting a rich set of principled summary statistics from the solution of the heat equation initialized at it. Furthermore, because learned function is expected to be vell-behaved only on the data manifold embedded in Euclidean space, it is possible to restrict the function on the manifold and solve the corresponding heat equation. doing so, the interpretation problem treated theory of differential geometry. Briefly, HeatFlow satisfies the following desiderata: (i) It provides a multi-scale analysis of feature importance in the formation of model predictions. (ii) is stable and informative because, implicitly, the neighborhood of test point is exhaustively explored by Brownian motion. (iii) It offers practitioners the flexibility to restrict their analysis on a manifold chosen from among Euclidean space, (learned) data manifold, and other interested submanifolds.
principled way based on the
A Toy Example. A toy example of understanding the of two Gaussian functions in 2D Euclidean space is illustrated in Figure la. First row demonstrates the initial function and its heat flow. The heat flow generates sequence of functions that are increasingly smoother than the initial one. Later we will see one possible representation of these functions as B[S((x)) which f is the initial function and {X,]e>o a path of Brownian motion. Intuitively. the values of these smoothed functions at x are local average values of the initial function in increasingly larger neighborhoods centered at x. The first subplot in the second row shows the deviation between the initial function and smoothed functions at three points, while the remaining subplots show Laplacian of smoothed functions along with their gradient fields. The deviation and the Laplacian are further decomposed into two directions, I and Z2, presented in the third and forth rows, respectively. Intuitively, our core idea is to distribute the deviation caused by the heat flow to each input feature by decomposing the heat flow as the sum of "sub-flows" in corresponding directions. We show that the proposed method further enables the detection of interaction strength between one variable and other variables, as demonstrated in Figure 1b.
= x). in
2

---

Under review as a conference paper at ICLR 2023
2 PRELIMINARIES
In this section, we introduce some basic concepts of explanation methods (Covert et al., 2021; Arrieta al.) in the interpretable AI community, along with definitions and key concepts about the Laplacian and the heat equation (Hsu, 2002; Grigoryan, 2009). The connections between the two fields where this paper is inspired are also presented. Throughout our discussion, we consider closed and connected manifold M endowed with a Riemannian metric g. A function f .M- + R is assumed to be bounded and twice continuously differentiable.
2.1 FEATURE IMPORTANCE & ATTRIBUTION
Let : RDD + R be trained neural network that predicts label y E R for a given input (21, TD) E DRD Post hoc interpretation methods aim at generating human-friendly explanations explain why y = fx). One of the most popular types of explanations feature importance, which assigns certain score to each feature 24 of a particular data point x. For differentiable f. the gradient VS: : + R.D is the simplest method that quantifies local importance of each input feature to the model's 's outcome. Attribution methods go step further to assign each feature a value thx (f. x) with the same physical meaning and unit of measurement as the model's I's output. It is then enforced that the values for all input features sum up to the difference between the outcome y and baseline b, f()) b-bb Eb=, vbe(f,x) Previous work usually chose the baseline as the model prediction on a baseline impul x', i.e., b == f(x'). However, the choice of baseline input is known to be non-trivial (Sturmfels et al., 2020). In this work, we directly choose the baseline instead and design a corresponding attribution method. An interesting setting of b is the locally averaged outcome of f in the neighborhood around x, that is, b = Ex [xx [f To avoid selecting a single neighborhood distribution N(x), we further suggest, informally, considering sequence of increasingly larger neighborhoods {AV 3x)0, where we expect / to govern the scale of neighborhood. To obtain this sequence, we turn Laplacian and heat equations introduced below.
The manifold hypothesis assumes that data in real world concentrates on low-dimensional submani- fold M embedded in the ambient space IR D of much higher dimension. The function be explained is expected to behave consistently only on M despite being learned in P.. For learned function f RR" + R. we may be interested in its mechanism in either the open world or closed world. Arbitrary input in RD will be allowed in the former setting, while the latter setting only accepts input on the data manifold. Thus, users should choose the appropriate target and manifold accordingly depending on the purpose of model interpretation. If the goal is to understand model behavior in the open world, if and I should be chosen; while fi.M the restriction of f on M, and M are suggested if the analysis is restricted to the data manifold.
2.2 THE LAPLACIAN ON A RIEMANNIAN MANIFOLD
Since we are interested in manifolds apart from Euclidean space, we introduce the Laplace- Laplace-Beirrami operator, AM. which generalizes the ordinary Laplacian to Riemannian manifold. There are multiple equivalent ways of introducing this operator (Hsu, 2002, Section 3.1).
Divergence of the gradient field: AMf div grad f. The gradient grad f is the unique vector field on M that satisfies (grad X), dX(() for any vector field X, where the differential df gives the directional derivative of f along X. The divergence diy X of a vector field measures how much it locally behaves like sink source. Laplacian will therefore be positive at minima and negative at maxima. More generally, it acts as a measure of deviation from local average. The relevance of Laplacian in work is then obvious: through closer look into Laplacian, it is possible to show, for example, why the model's 's output at a local maxima minima) is larger(less) than its neighborhood.
Trace of the Hessian: AMI = tr(v² f). In Euclidean space R", Hessian V is matrix of the f. In Riemannian manifold with its Levi-Civita connection V, Hessian is the second covariant derivative, such that given local coordinates {a²), 8² r äf a dxx @ dx', where r'
second partial derivatives, hence A = E: dz? fzz.f.
local expression for the Hessian tensor is V the Christoffel symbols of connection. Therefore the Laplacian contains information about the second order behavior of function, which is not available in gradients.
3

---

Under review as a conference paper at ICLR 2023
Hodge Laplacian: AMf df. In exterior caleulus, the codifferential d' is adjoint to the differential d and by f o = f [df..0)gdV g)g where Vg is the Riemannian volume. When acting on 1-forms, d' can be given by the divergence d" (ax) the dual of X by (Y = (X,Y), Y), and hence AM = div grad f -d" The Hodge Laplacian for differential forms, given by M (d'd dd*). coincides with Amf when acting on functions, i.c. O-forms. Generalization of Laplacian to differential forms allows us to analysis the gradient field of a function in addition to the function itself.
div(X) where ax is
Infinitesimal generator that generates Brownian motion on M: roughly speaking, it means 1 (z) limæio i (BIs(X)))X, connection provides a clue to the definition of a distribution on the neighborhood of x through Brownian motion.
x] f())), where {Xe)e>o is a Brownian path. This
2.3 HEAT EQUATION
With the above Laplacian defined, we can now introduce the most basic diffusion process governed by the following PDE, which describes how initial heat distribution :M R looks after being diffused for time > 0:
i u(1, x) Am
(1)
at
(0,x) f(x).
(2)
The solution, utt, x): (0) x M IR, can also be given in multiple roughly equivalent ways:
Convolution: u(t. x)= Jm le (). fy)) in which ( y) is the fundamental solution of this PDE, known the heat kernel. In Euclidean space, is just a Gaussian centered at the point x.
u(t, x) =[[((XX)) Xu = x]. As the heat kernel is also the transition density function of Brownian motion on M, the expectation of function value over ends of Brownian paths yields the solution on stochastically complete manifold (Hsu, 2002). Such a stochastic representation of PDE solutions is known the Faynman- Kac formula (Karatzas & Shreve, 2012).
Expectation:
Gradient flow (Santambrogio, 2016): It well-known that the solution the gradient flow for the Dirichlet energy f monotonically decreases in time under the heat flow, meaning that the smoothness of the solution is always increasing.
grad flgdV, in (MM) space. Consequently, the Dirichlet energy
is convenient to define the heat operators (Pt)co to represent the solution: x(t,x) = (x). The heat equation can be extended to the diffusion of tensor fields and differential forms using generalized Laplacian. For instance, heat equation on 1-forms may be defined with the Hodge Laplacian: 8 (t,x)) x) (t,(t x). Since [. commutes with the differential d, if the initial condition is set to be a closed 1-form, i.c., the differential of function: (0, = df(x), then the solution will have an interesting connection with the heat equation on functions (Hsu, 2002. Section 7.2):
((t,x) (x).
(3)
means that diffusing the differential of a function with Hodge Laplacian is equivalent to applying the differential operator to the solution of scalar heat equation.
3 MULTI-SCALE INTERPRETATION BASED ON HEAT DIFFUSION
By treating the model to be explained as the initial heat distribution, we analyze the solution of the heat equation to summarize the model at different scales from the point of view of an input x. The solution alone has already given local weighted average value of the function centered at x: (x) [x, = x]. The time ł plays the role vicinity quantification that defines the range of neighborhood through a stochastic view the area reachable by Brownian motion starting at x. For small value of t, the smoothed value is mainly affected by small neighborhood of x, hence reflecting only local information of the explained model. While as increases, the smoothed function tends to capture more global trends of the model. In this section, we show that it is possible to extract more detailed information from the solution by applying differential operators to { (P. )
4

---

Under review as a conference paper at ICLR 2023
3.1 SUMMARIZING GRADIENTS
By Eq. 3 and the duality of differential and gradient, taking the gradient of Ptf is equivalent to smoothing the gradient field of f through running a heat equation for vector fields up to time l. In Euclidean settings, the gradient of Pal is (x) = EEV locally averaged version of the initial gradient field. On more general manifolds, it does not make sense to add vectors living in different tangent spaces Tx, MM, and we need the parallel transport (Lee, 2018, Chapter 4) term acting as a mechanism to connect nearby tangent spaces. There is Feynman- Kac formula for this operation (Hsu, 2002, Theorem 7.2.1):
= x]. which is simply the
grad (x) [E[M grad f(X))X = x].
where Tr the stochastic parallel transport map that transfers vectors from the tangent space of X back to that of Xo along the Brownian path, and M, is related to the Ricci curvature tensor on M. One can interpret Eq. 4 as collecting derivative information from neighborhood by sending Brownian particles that travel for fixed time length, parallel transporting the gradients at the positions of particles back to the explained point and averaging them.
3.2 ATTRIBUTION
For attribution tasks, natural baseline to choose in our framework is the local average B[](X,)] Xo
x]. Formally, given a model function f :. C RDD + R. the input of interest x, and the solution Pas of heat equation initialized at f. we attribute (x) f(x) to input features {x, at each t.
In order to allocate the difference to the coordinates of ambient space RD,, we try to disaggregate the Laplace-Beltranne Beltrami operator using the the standard orthonormal basis {c Let {x be the orthogonal projection of the unit vector e, onto the tangent space TxM. Since the gradient field tangential to M, we have grad ʃ = E, (grad f fee:) E. (grad f, E, The projection of grad I' onto the ith dimension is achieved by grad, f (grad E (E, and we define the dual form of grad, f as f to be the "partial differential" operator. Further, we can compute the contribution of cach feature i by taking the divergence of the projected vector field, div ((E,)) In this representation, the flow of heat along the ith dimension for an input x up to time T is defined as,
/" div ===
HeatFlow; (x, T)
dt.
(5)
desirable property of HeatFlow is that the attributions add up to the difference between the values of f and its smoothed version as follows,
= f a J कt
(P.N()dtt= J" (x(P)))x) Σ Heat HeatFlow; (x, (6)
where the last equality is an immediate result of AMj div gradʃ = Σe (grdd, (gradf.&)) E.). Instead of computing div with respect to manifold, we can also globally express both div and grad using the standard basis {c:}, i.e., grad = CE, and div X -P &,X; where X; is the i-th component of vector field X expressed in ambient coordinates. This leads to another decomposition AMf Σ f. Furthermore, through the trace of the Hessian expression, also have AMf= =1 &E,, (Hsu, 2002, Corollary 3.1.5). Together, we have the following three decompositions,
D T / ((PIENE))))))))) dt. (7) div grad dt = Σ [ (((P))())) =
D
Σ
It is worth noticing that while they are the same when summed over D dimensions. the three summands for each dimension are not necessarily equal to each other. For simplicity of calculation, we adopt the last decomposition strategy based on the Hessian tensor in our experiments.
In Euclidean space, all the three decompositions just reduce to sum of the diagonal terms of the Hessian matrix. Hence, attribution for i-th dimension becomes integral of the second partial derivative:
T 8²
Heat Flow, RD (x, Txf) =
(P))(x) dt.
(8)
dx
1=
5

---

Under review as a conference paper at ICLR 2023
3.3 PROPERTIES IN EUCLIDEAN SPACE
In this section, some nice properties satisfied by our method in Euclidean space are discussed. All proofs given in the Appendix. We leave the study of analogous properties general manifolds for future work.
HeatFlow obeys attribution axioms. The evolvement of the model interpretation problem brings several desirable properties that a new attribution method should satisfy (Lundberg & Lee, 2017: Sundararajan et al., 2017; Friedman, 2004). The following proposition includes four axioms defined in (Sundararajan & Najmi, 2020).
Proposition 1. HoatFlow
satisfies the Dummy, Efficiency, Symmetry, and Linearity axioms.
HeatFlow recovers GAM. Generalized additive models (Hastie & Tibshirami, 2017; Lou et al., 2012) are family of inherently interpretable models based on the sum of univariate functions. The interaction between the features is simply additive in a GAM, therefore a user can interpret the contribution of each feature independently through looking into the corresponding univariate function. Ideally, if an attribution method is applied to a GAM, its output is expected to be consistent with the GAM itself. This is the case for HeatFlow.
Proposition 2. Suppose
IRD R; X Fi is smooth additive function, in which Vi E {I,..., D), , J3 R + R bounded continuous function in L (B)... Then RD f.t) f((.))
lim Hlot
HeatFlow detects additive structure. For given neural network f. is usually impossible to decompose into sum of univariate functions. How do we know if at least some variable, say, H;, contributes additively to f? One idea is to look into the its gradient Vf R R.D Since Euclidean space is the focus of this section, let slightly abuse the notations df grad f f. div X for simplicity. æ; is truly
d = grad, f = [V,],e, 0, = [V and d contributing to f additively, i.e., 3ʃ, R + R such that fxx) = f((2,)) + f=)(x_;), then the magnitude of the gradient projected onto the i-th direction can be fully characterized by fs via o df.. The following lemma shows that the converse is also true.
Lemma 1. Assume that there exists univariate function Ji R - R such that Vx E RDD. df((x)) = =, Then f(x) = fa(,,) + ii), which the function f-s D-D11 does not depend on Iz.
Based on this observation, in order to detect if x, contributes univariately, one can try to find function 9: 1 E whose gradient matches d,f everywhere. If such a Ii exists, it will depend only on a; because its partial derivatives with respect to (æ, ji will always be zero. A straightforward implementation of this idea to solve the optimization problem:
min dg 2 = / [ldg(xx) dxf(x))l3
(9)
It is well known that the Buler-Lagrange equation for this problem is the Poisson equation Ag d'd d'd.f. Interestingly, we discover that HeatFlow is solving this kind of equation.
Proposition (-)e()) HeatFlow?" (- f, solves the Poisson equation Agi. = d'd,(P.f f).
As result, for a function f satisfying Vx lim Ptf(x) = 0, e.g., f (TR)) the difference between the gradient of HeatFlow at large enough /. and the partial gradient of f approximately measures the degree of deviation from being univariate for a considered feature. Specifically, if the feature i rarely interacts with other features, the following residual should be small everywhere:
ry (x(x)
xx) = | HeatFlow, (x,,,,,ww/)))) f,t)) dP -xxf)
(10)
When this residual zero everywhere, according to Lemma 1, the feature contributes to P.f f additively. In Figure 1b, visualize the residuals at large t for the toy example. Interestingly, similar residuals are also discovered a different setting where the Shapley value (Shapley, 1953) is employed for attribution (Kumar al., 2021).

---

Under review as a conference paper at ICLR 2023
4 CONNECTION WITH EXISTING WORK
Close connections can be drawn between our method and many existing attribution approaches. We discuss in this section how method generalizes the literature of attributing prediction of model f()) on an input x to its features.
Vanilla Gradient (Grad) (Simonyan al., 2014). The Vanilla Gradient is defined as Grad(z) Vx)(x). It only characterizes first-order behavior of the model in an infinitesimal range.
Smooth Gradient (SG) (Smilkov et al., 2017). Given a user-defined variance a, the Smooth Gradient is defined as SG(x) Nxx,0 Vzf(z).
special case of our method in Euclidean settings: SG(x) = Vx + (0, o (x) = Vxu(t = o² /2,//)x)
Integrated Gradients (IG) (Sundararajan et al., 2017). Given a user-defined baseline input x', 2_
IG of feature is defined as IG, x') = (z, - (x))- So 8m; f(x' +axx- x'))aa where gradients are accumulated along the straight-line path in Euclidean space. Our method can also be viewed expectation over multiple path integrals: [ (X,))x) x] _ df = xx].
Expected Gradients (EG) (Erion et al., 2019). By introducing a distribution of baselines D, EG is the expectation of path integrals defined as EG, D) = Ex>~DI; x'). When a Gaussian baseline is adopted as D (xx,, o² I). it coinsides with explaining deviation from local average as E, (x;(x. D) = f(x
method in Euclidean settings,
When the training data itself is used, D D it coinsides with our method in manifold setting when t + 00, explaining the difference from global average of all data.
Blur Integrated Gradients (BlurIG) (Xu et al., 2020). BlurIG extends IG by considering the path of successively blurring the input image using Gaussian filter. It solves a heat equation in the 2D plane of a single image as contrast to data space in our method.
It is worth noting that the heat equation has also found applications in diffusion-based generative modeling -Sohl-Dickstein et al., 2015; Song et al., 2021; De Bortoli et al., 2022), in which certain design of forward diffusion process is equivalent to solving the heat equation (as a special case of the equation) initialized at p(x), the probability density function of data distribution.
5 EXPERIMENTS
To illustrate our multi-scale attribution methods, we demonstrate a sequence of attribution maps at discretized time steps on three image recognition tasks, including a synthetic image regression task, MNIST classification and facial age estimation'. We compare our method, HeatFlow, with four other methods, including Grad, IG. SG and BlurIG. As for SG, increasing levels of Gaussian noises are applied and resulting gradients are averaged over 100 samples. We denote noise level as the amount of noise to add to the input as fraction of the total spread, max x min x. As for BlurIG, partial accumulation of gradients along the path is calculated, where denotes variance of the 2D Gaussian kernel. detailed discussion and pseudo-algorithm of the numerical implementation of HeatF'low in end-to-end framework is presented in Appendix A.5. In the following experiments, all data manifold is learnt by VAEs. Since our method involves many differentiation operations, experiments" implemented using the JAX package (Bradbury et al., 2018) with Tesla V100 GPU. Further information, such as latent dimension of VAES and image sizes are summarized in Appendix A.7. For all attribution maps of following figures, red and blue pixels denote positive and negative values, respectively, and deeper color denotes higher absolute value and stronger contribution.
5.1 SYNTHETIC EXPERIMENTS FOR HIERARCHICAL FEATURE STRUCTURE
To study the ability of HeatFlow in uncovering different features with global or local effect, we design synthetic dataset with hierarchical feature structure. A latent code z [=o, =d-1] E [Uniform(0 (0, 2 is first sampled, based on which, labels and input images are generated as shown in Figure 2a. The underlying manifold is known, which is the Cartesian products of six circles. Local features are those pixels close to the upper-left corner, while global ones are those of larger squares.
I Face images from the UTKFaace https: http///suagggguuu....ute iitthub.io/uKKaace datasct. Source code available at: https: / /nonymouss.m.ooo 4open. wcwence/eeneeeee /heattt///eeeeeeeeeeeeeee exp a ainer-FFDC
7

---

Under review as a conference paper at ICLR 2023
Input
HeatFlow
True
Manifold
2.5
VAE
HeatFlow
d+12
VAE d-12
VAE
sin
True
Manifold
Grad
SG
10ae 5%
15%
25%
25%
45%
cos 4o
IG
Blur 1G
60
(a) Data generation.
(b) Heatmaps generated by various methods.
Figure 2: (a) Top: Generate labels. Bottom: Generate images. (b) Comparison of HeatFlow, Grad, SG, IG with black baseline and BlurIG on synthetic data. HeatFlow under true manifold, manifold learnt by VAE with latent dimension d = 12 and d = 6 are presented in the first three rows, respectively. On the fourth row, gradients on the true manifold collected each time step. The reconstructed inputs shown below the original input. Vanilla Grad and IG are presented below the reconstructed inputs. For HeatFlow, SG and BlurIG, different time steps, noise levels of Gaussian blur, and partial integration up kernel width shown, respectively.
As shown in Figure 2b, vanilla Grad mostly highlights local features that are close to the upper-left corner. Both Grad and SG misleadingly highlight squares lying the diagonal which stays constant among all images. For IG and BlurIG, more global features are concentrated on, however multi- scaled information is lacking. For HeatFlow, localness is meticulously controlled by heat diffusion on the manifold and when value of is small, only small squares which correspond to local features highlighted; value of grows, more global pixels, i.e. larger squares gradually shown. Furthermore, highlighted pixels tend to appear as an organized group in HeatFlow, suggesting its ability to attribute features with correlated contributions.
Manifold Mismatch. The heat kernel is known to be stable under perturbations of the underlying manifold (Sun al., 2009). Intuitively, such stability comes from the Brownian motion interpretation of heat flow: small perturbations will only affect a subset of the infinitely many Brownian paths. To study the reliance of alignment between the learned manifold and the true manifold, an ablation experiment is conducted where HeatFlow is run separately using the true underlying manifold and manifold learned by VAE with latent dimensionality of Figure 2b. It is observed that, when the correctness of the manifold increases, more detailed information is captured. The multi-scale property behaves as expected regardless of degree of manifold mismatch.
and 12, shown first three rows
5.2 MNIST AND UTKFACE
The same experiments are conducted on the MNIST classification task, where in Figure 3a, logits of neural network with 98% test accuracy is diffused over time. To handle all logits simultaneously. we use heuristic extension of HeatFlow described in the Appendix. Saliency maps are particularly noisy for vanilla Grad. SG and Blur IG also fail to provide succinet information as Gaussian noise added in Euclidean space introduce unreliable evaluations of gradients over off- tof-tthemmifoodd samples. For HeatFlow, concise attribution with pixels only on digit area are highlighted. It is also evident that the attribution changes heat flows. More interesting findings can be drawn if we study the logits of digits that are similar. In Figure 3b, we present heat flows and multi-scale attribution maps for three digits, 3, 6, 9, and compare them with their visually similar digits, {5, 8}. {0, 8). 4). respectively. It is observed that, in the last column, only pixels corresponding to the common features of the compared digits remain. At small pixels that separate digits apart are highlighted.
8

---

Under review as a conference paper at ICLR 2023
3
9
(a) Multi-scale attributions on MNIST.
(b) HeatFlow for different logits.
Figure 3: (a) Heat maps of the logit of label class on MNIST test samples. Original input is shown in the most upper-left corner. Vanilla Grad and IG are presented below the input. HeatFlow, SG and BlurIG with increasing level of time, noise and kernel width is shown on each row started from the second column. respectively. (b) Heat diffusion for MNIST samples comparing logits of different classes. Left: Change in function value. Right: HeatFlow attribution maps.
As for explaining the facial age estimation task, results are presented in Appendix A.9. Our method managed to locate structured features such eye brows, wrinkles, whether smile with teeth shown as important information for prediction of ages, where other methods failed to discover. We also emphasize that multi scale information summarized our method, such that attributed features which remains for large value of represent more global effects, such as wrinkles.
Quantitative Evaluation. We also include quantitative evaluation to compare HeatFlow with other methods using strategy adopted in (Jethani et al., 2021). This evaluation strategy utilizes the remove/retain-ang -entrrrrrnnnn ideas where evaluator trained to predict the label given an arbitrary subset of features, and then, performance of the evaluator assessed as the features were gradually ex- cluded/included according the absolute im- portance output by each explanation method. Using a set of 1000 test images for MNIST and 100 for UTKFace, accuracy and mean absolute error assessed respectively as we include or exclude the most important features ranging from - 100%, with curves visual- ized in Figure 4. HeatFlow is very competitive in terms of AUC, indicating that it is good at identifying important discrininative features.
Accorpicy
Teat 5)
Murg
(rclsoott))
EXC
MAC
Figure 4: The change of accuracy for MNIST (upper) and MAE for UTKFace (lower) as an increasing percentage pixels attributed to be important are included (left) excluded (right).
CONCLUSIONS
We have introduced a novel interpretation
framework, HeatFlow, which generates a sequence of feature attributions for an interested input, explaining the deviations of model's outcome from multi-scale local averages. Drawbacks of this work lie in the difficulties of manifold learning on more complex datasets, high-dimensional PDE solving, and high-order derivative computation. Our method will benefit from methodological advancements these directions. is also interesting to generalize HeatFlow to the union of disconnected manifolds, which may be more appropriate assumption for classification datasets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards hetter understanding of gradient-based attribution methods for deep neural networks. In International Conference on Learning Representations, 2018.
Alejandro Barredo Arricta, Natalia Flaz-Rodriguerz Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García. Sergio Gil-López, Daniel Molina, Richard Benjamins, al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion.
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. In International Conference Learning Representations, 2018.
Christian Beck, Sebastian Becker, Philipp Grohs, Nor Jaafari, and Amulf Jentzen. Solving the kolmogorov pde by means of deep learning. Journal of Scientific Computing. 2021.
Julius Berner, Markus Dablander, and Philipp Grohs. Numerically solving parametric families of high-dimensional kolmogorov partial differential equations via deep learning. Advances in Neural Information Processing Systems, 2020.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, and Skye Wanderman-Milne. JAX: composable transformations of Python+NumPy programs, 2018. URL http: / giggtiithuubb com/ google/jaa
Johann Brehmer and Kyle Cranmer. Flows for simultancous manifold learning and density estimation. In Advances Neural Information Processing Systems, 2020.
Ian Covert, Scott M Lundberg, and Su-In Lee. Explaining by removing: unified framework for model explanation. J. Much. Learn. Res., 2021.
Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modeling. arXiv preprint arXiv:220..0276. 2022.
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In Advances in Neural Information Processing Systems, 2019.
Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott M Lundberg, and Su-In Lee. Learning explainable models using attribution priors. 2019.
Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott M Lundberg, and Su-In Lee, Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature machine intelligence, 2021.
Eric J. Friedman. Paths and consistency in additive cost sharing. International Journal of Game Theory, 32:501-518, 2004.
Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
Alexander Grigoryan. Heat Kernel and Analysis on Manifolds. 2009.
Trevor Hastie and Robert Tibshirani. Generalized additive models. 2017.
Robin Hesse, Simone Schaub-Meyer, and Stefan Roth. Fast axiomatic attribution for neural networks.
In Advances in Neural Information Processing Systems, 2021.
Elton P Hsu. Stochastic analysis manifolds. 2002.
Neil Jethani, Mukund Sudarshan, Ian Connick Covert, Su-In Lee, and Rajesh Ranganath. Fastshap: Real-time shapley value estimation. In International Conference on Learning Representations, 2021.
10

---

Under review as a conference paper at ICLR 2023
Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi. Guided integrated gradients: An adaptive path method for removing noise. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Ioannis Karatzas and Steven Shreve. Brownian motion and stochastic calculus, volume 113, Springer Science & Business Media, 2012.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations. 2014.
Indra Kumar, Carlos Scheidegger, Suresh Venkatasuubammanian, and Sorelle Friedler. Shapley residuals: Quantifying the limits of the shapley value for explanations. In Advances in Neural Information Processing Systems, 2021.
John M Lee. Introduction to Riemannian manifolds, volume 176. Springer, 2018
Yin Lou, Rich Caruana, and Johannes Gehrke. Intelligible models for classification and regression. In Proceedings of the 18th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2012.
Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30. 2017.
Thomas März and Colin B. Macdonald. Calculus on surfaces with general closest point functions. SIAM Journal on Numerical Analysis, 5066):3303-3328, 2012.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning. 2014.
Steven J. Ruuth and Barry Merriman. A simple embedding method for solving partial differential equations on surfaces. Journal of Computational Physics, 124(3):1443-191, 2008. ISSN 0021- 9991.
Filippo Santambrogio. (Euclidean, metric, and Wasserstein] gradient flows: overview. Bulletin of Mathematical Sciences, 7:87-154, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In Proceedings of the IEEE international conference computer vision, 2017.
Lloyd S. Shapley. value for n-person games. In Contributions to the Theory of Games, 1953.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. Workshop at International Conference Learning Representations. Citeseer, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. 2017.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference Machine Learning, 2015.
Yang Song, Jascha Sohl-Dickstein Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. ccore-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https: /opp/eeveeeeeeeeeooe net forum? id=PxTIG12RRHS,
11

---

Under review as a conference paper at ICLR 2023
Suraj Srinivas and Francois Fleuret. Rethinking the role of gradient- based attribution methods for model interpretability. In International Conference on Learning Representations. 2021.
Ari Stern and Alexander Tettenhorst. Hodge decomposition and the shapley value of a cooperative game. Games and Economic Behavior, 2019.
Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 2020.
Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Computer graphics forum, volume 28, pp. 1383-1392. Wiley Online Library, 2009.
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In International conference on machine learning. PMLR, 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on muchine learning. PMLR, 2017.
Binxu Wang and Carlos R Ponce. geometric analysis deep generative image models and its applications. In International Conference Learning Representations, 2020.
Zifan Wang, Haofan Wang, Shakul Ramkumar, Piotr Mardziel, Matt Fredrikson, and Anupam Datta. Smoothed geometry for robust attribution. In Advances in Neural Information Processing Systems, 2020.
Shawn Xu, Subhashini Venugopalan, and Mukund Sundararajan. Attribution in scale and space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Huiting Yang, Liangyu Chai, Qiang Wen, Shuang Zhao, Zixun Sun, and Shengfeng He. Discovering interpretable latent space directions of gans beyond binary attributes. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 AXTOMATIC PROPERTIES IN EUCLIDEAN SPACE
Proof of Proposition 1. Since the derivative of function with respect dummy variable is always zero, the Dummy axiom is satisfied. By Eq. 6, the attributions always add up to the difference (P) (xx and hence proved the Efficiency axiom. The linearity is consequence of the fact that the heat operator is linear. For a function that is symmetric in two variables 2; and Tj, we prove below P'P. Ptf/Out 0² P.f/Ox² as long as 2; = z.
Let f(1 ..., Ti, F) In be a function of n variables, f :R + R. Without loss of generality, let T; and 2z be symmetric variables, i.e. f(... Lis .. I.....) ...)= f(... 2ij a. Tir ...) for Vz,, 2zz E IR Then we have
Pif F4 .... IJ,...
1
Σ
Th yx,)² (..., Mis ..., ...yy
(4Tl)m/2
exp
41
1
(Ix yx))² 2 (20, - -21) + (xy vz)²) {(... Pis... Ys Vjs...dy
(4Tt)m/2 exp
4t
kE(())
1
(Ix 1/63 + (0i 1/// + (xx 1/12) ((.., 1/y bbb....dy
(4Tt)m/2 exp
4t
kE(
-/, (4Tt)n/2 exp
(on 1/6b2 (25 v/)² + (xr - 18)22 (o77)smmn "/. Joy
41
(E.))
PPff (.. Fj.... D.....
where y' is the permutation y by exchanging the i-th and j-th variables. The third equality follows from the change of variables formula and the fact that the determinant of a permutation matrix is #1. Hence, draw the conclusion that the heat operator preserves the symmetry in Euclidean space. Next, further assuming that x; Ti = a and according to the limit for second derivative, we have,
0²
(.., a,..,, a....
f(... a.... 2f( a a,..., a, ...) + f f(... a h le.. ...).
-lim
h-+0
h2
f..., Cr a + h,...) -2)))-2((...
2f( a m a ...) + f(.. a,..., a - h....)
lim
h-+0
12
2²
0n (..,, an ).....
In other words, we have shown that the second derivatives of the symmetric function with respect to symmetric variables are equal when the values are equal. As a result, we have proved that 8 Ptf/ 02 P.f /0z] long F; = Iz-
A.2 DEVIATION FROM ADDITIVE CONTRIBUTION AND LINK WITH SHAPLEY VALUE
An interesting link exists between our method and the Hodge decomposition a coorperative game (Stem & Tettenhorst, 2019). Given a set of D players and a value function v 2( + R. each coalition of players S 2 PI V can be considered as a vertex of D-dimensional hypercube G = (V, E), where each edge corresponds to addition of single player & S to S. The gradient and divergence of exterior calculus operating on this graph, denoted by (d, d'). are defined as dv(S, U [i)) = v(S U (i)) uSS) and (d" (d) = D(.,)E f(b, a), respectively. du E (2) gives the marginal value contributed by player joining coalition. To specify marginal contribution of an individual player i (D), a partial gradient (VV - (EE)) is defined as
de Su(S (i)) v(S), i=j
(0
otherwise
13

---

Under review as a conference paper at ICLR 2023
Notice the corresponding definition of partial differentiation operator in the continuous space adopted in our method, d,f = grad; f. A key result in (Stem & Tettenhorst, 2019) is connecting imessentiality of games to the defined partial gradient operators in the following proposition, where in inessential game, for all S C V, v(5) = Eies of ((i)), meaning each player contributes a precise value v ((()) any coalition it participates in.
Proposition 4. (Stern & Tettenhorst, 2019, Prop 3.3) A game is inessential if and only if d, E im for all i [D].
This means a game is inessential if one can find games vi such that d,v = do; for each player In setting, there is notion of game but we consider in breaking down the given differentiable model f()) to additive univariate functions. This ability in separating each feature to contribute independently is the "inessentiality" of our problem.
In particular, we consider the simplest case where the function to be explained is itself an additive one, i.e. f: RDD + IR E (1rz). We prove Proposition that method Heat Flow, Rp (,, T,T.J. tends to recover each independent component as tends to infinity.
Proof. The solution of heat equation initialized at an additive function is the sum of ID solutions, as follow,
Pif(x) = / (ATI) exp ( (- - 41 Ix y113)
Σ
fe(y)) dy
1=1
= (AT()1/2 exp ( //4 (a: f(())) dy (P.f)(a)))
46
i=1
==1
Following this observation, is easy to derive that,
HoatFlow; Rn (x,
1r 0² (Pif dt ['
T 0² (t. ,- = du?
T a (P.f))))) Prf((z)) f(())) di
dx²
For fi (R), 20 (z,) O. Hence, Proposition 2 proved.
Next, we consider more general case, where some variables might interact with each other.
Proof of Lemma 1. Assume an arbitrary path over R" r(t) = (t)(t), xn (t))" such that the end points are r(0) xx and r(1) = x6, particularly, the i-th feature has value (xa); and (xx); b. By the fundamental theorem of calculus for line integrals, and if Øf di for any x, have, dz. dr.
S(x,) (xxx) / Of(((t)) dr(t),
di Ōx dt.
Of(r(t)) dea(t) dt
Oz: di
(f((()) dt,(t) dl Jo Ox db
1 Of(((t)) dx;(t) di
Σ
Ox, dl
$$1
[
(r(((t)) dz, dt (" dff(xx((t) dr,(t) dt Ox, dt 0f(r(1)) dz,(t) dt + f((b) fí(a).
Σ
dr, dt
dx, dt
= = Eyzs To af(r(t)) dz,(1) (t) + fa(z)) Since dz; dt path r(t) is arbitrary meaning the equation holds for any path between 0 and x, if the first term depends on I,, then by differentiating both sides with respect F., Ba. dxs and hence Lemma proved.
With x, set to 0 and assuming f(0) = 0. we have f(x)
af di would be contradicted
14

---

Under review as a conference paper at ICLR 2023
Next, we prove Proposition such that our method HeatFlow; equation Agji d' dgi.: = d' dddd(P..f f).
RD (,t,f) f) is solving the Poisson
Proof. First we emphasize that operator d'd; d; and the heat operator P. commute with each other since d² de commute with the Laplacian A as follows:
0
8
d' d' div
Oz
0ak
0
D
0²
2
dur
1=1
0
0²
Σ
Ad d.f.
do
dzy
With this fact, we have
d'a (eeatFlo,(xxx
d'd
d' d'd,(P-
T=0
d'd
Pr(d' (x))
1-
AP, d.)ddd)))x)
==0
J+ Ōt Pr dxf)(x)dT
(x(d'd,f((x)) d' (x
A.3 EXAMPLE ON MULTIVARIATE GAUSSIAN
Here, present simple example of method, HcatFlow, d-dimensional multivariate Gaussian with mean and diagonal covariance Σ diag(an,..... -a)- Firstly, the solution of heat equation is calculated follow,
Pif(x)= = / ()xx)))N(r:p.. Σ) dy
/ (4Tt)d/2 exp (
1
1
||x yl
1/2 exp (y- 1 -4) dy
4t
(2m)d/2|2)
=N(x: μ4, E + 2tl)
Consequently, we have the contribution of the i-th variable
/' (P.f) HeatFlow (x, T,f]))= 1 Ox?
RD
T 82
1" 120 (2T (2T))d/2 + 2/1 0; +
1
dt
A.4 ADDITIONAL RELATED WORK
A relevant literature from the perspective of generative models is identifying interpretable latent dimensions (Yang et al., 2021). In (Wang & Ponce, 2020), unsupervised discovery of interpretable axes is facilitated by exploring the latent space along the eigenvectors of the metric tensor defined by the decoder. Eigenvectors at different ranks encode qualitatively different type of changes. The main difference between work and these prior literature is that primary goal is interpret learned regressioonclassifiaaiion model with the help of a generative model to explore the data manifold, while their goal is explain the generative model itself.
15

---

Under review as a conference paper at ICLR 2023
The discussion in the main text assumes that we have access to parameterized representation of the manifold M. In practice, the manifold may be implicitly defined via a projection function that projects an arbitrary point onto the manifold through identifying its closest on on-manifold counterpart in terms of Fuclidean distance (Ruuth& & Merriman, 2008; März & Macdonald, 2012). Given such closest point function cp(x) TRD + .M, it is possible extend function f(x) . + TR to the surrounding space in RD by defining a new function f cp(x) : RDD + R. It has been shown that the intrinsic operations of grad and A are simplified as grad f(x) = Vlf o (x) and
AMf (x) = Aff cx)(x). The latter equation provides the fourth possibility to disaggregate the Laplace-Beltraide Beltrami operator.
A.5 NUMERICAL IMPLEMENTATION
In this section, we briefly discuss the numerical implementation of our method in an end-to-end framework from manifold learning to multi-scale explanation. Since there are numerous approaches for manifold learning and high-dimensional PDE solving, we claim our implementation is only one of many possible realizations and can be generalized to different manifold settings for various purposes.
Manifold Learning. Deep generative models strive to infer probability distribution from observa- tional data X' DR as well as learning the underlying data manifold (Brehmer & Cranmer, 2020; Arvanitidis al., 2018). Approaches such as VAEs (Kingma & Welling, 2014; Rezende et al., 2014) and GANs (Goodfellow et al., 2014) makes the assumption of a d-dimensional data manifold, M, embedded in the ambient space with d < D, through highly flexible generator, g 2 - M, where ze Z =dd is the latent variable. The local Jacobian of the generator function, J2 8g z=z. provides
local basis in the input space and G2 J J, is the Riemannian metric.
Heat Equation Solving. To solve the heat equation on manifold, especially a high- digendimensional one, utilize the Feynman-Kar formula and the framework proposed by (Beck et al., 2021; Berner et al., 2020) where one can simulate training data in order to learn solution P.f by means of deep learning. supervised learning problem is constructed via the predictor variables (x, t) and the dependent target variables = (Xx, Xa=x. The unique minimizer of the quadratic loss ming [E((x(x, t) - y) the solution of the heat equation. In practice, the following empirical error minimized over the function space of suitable neural networks H::
d arg min 1 " [(X,)) Xo x dEH
(11)
where {(x,, 4-) realization of i.i.d. samples uniformly drawn from M T). To simulate Brownian motion on a Riemannian manifold, we perform random walks over the learned manifold depending on the Laplacian adopted. In Fuclidean setting, random walks are realized by adding Gaussian noise. While in curved space, one usually resort to geodesic random walks performed directly latent variables. Detailed algorithms are presented in the Algorithm 2.
For classification models, to align with training of the original model, we minimize the cross entropy loss, instead of MSE in Eq. 11.
Gradient and Decomposition of Laplacian equation solving, we need to explicitly realize the projection &.- When local chart g is available, such as decoder function, we define JJ J)1J" as the projection matrix. The resulting explicit forms of f and E, are
To compute the gradient and the attribution after heat
grad (J"
G 10f/0z & (J"JJ
(G-10g:/0z
By the definition of Christoffel symbol which describes the metric connection of manifold, is easy to 8g
derive 13 Σm G (0, Gkm + Gjmm 2m Gjx)) Em,b G Hy where H 02 the Hessian matrix of the generator and Jhb dgs the Jacobian matrix. Following the definition dem
of Hessian tensor, we have
05 V2 V2I(E,, E.[ Ozz0dz
0s 0z;
0g: T G-1 1 H, J,G -, )
Og.
r
E Oz
Oz
ik
(12)
16

---

Under review as a conference paper at ICLR 2023
where y and , the Hessian and Jacobian matrix of the explained model with respect to latent variables, respectively. Substitute this implementation into the third decomposition strategies in Eq. 7 for each Ptf and then obtain multi-scale attribution.
The following pscudo-algorithm shows a reference implementation of HeatFlow. Notice that given training input data and function to be trained, the steps of manifold learning and heat equation solving only need to be computed once. The resulting manifold and heat kernel can be reused for explaining any further test input.
Algorithm 1: A reference implementation of HeatFlow
Require: D: training dataset, f : RR. >> IR: a trained model, x": a test input to explained, T:
total horizon D.T. Output V { Jit=l feature importance for each feature and level of localness t // Manifold Learning
learn the underlying manifold using VAE from D
get-: encoder g, decoder h, Jacobian of the encoder Jg. and metric G, = JJJ
// Heat Equation Solving
initialize model with similar structure as f and take extra input while learning not done do for each batch do sample {x(+) 10) uniformly from [),
for each = 1,...s do
(+) + RW 9 )
end
update é minimize loss -= -iE (x(x(1,)))))))))))))) ((()))))))
end
end
get solution ox(x, Rs (xf(() on the learned manifold
// Decomposition Laplacian
12 initialize L as a zero array =x(x")
14 for each i = = 1,.., do for each (. 1,..., do
calculate HeatFlow; (=) according to Eq. 12 accumulate ( Vii 1] + Heat 56
end 18
19 end
In order to simulate Brownian motion Riemannian manifold needed in solving high- dimensional heat equation, we resort to Algorithm 2, which is also adopted by (Arvanitidis et al., 2018).
Algorithm 2: Random walk on Riemannian manifold: RW s, S, G)
input Latent starting point E Rdd, step size number of steps T, metric tensor G output: Random walk path Txx
% for t
to T do L, U = (z)) (L eigenvalues, U cigenvectors)
v= UL de, e~ L.),
z= z=
Z(t.:) Z
end
17

---

Under review as a conference paper at ICLR 2023
A.6 DEFINITION OF MATHEMATICAL SYMBOLS
Table I: Definition of involved math symbols, in local coordinates on a Riemannian manifold and Cartesian coordinates in the special case Euclidean space, along with their intuitive meanings in natural language (intuitive meaning referenced from Wikipedia). Particularly for Cartesian coordinates in Euclidean space, a three-dimensional example also shown. Notice that the Einstein summation convention is used, implying summation over i and j. 0x/Ox' and et dæ' refer to the unnormalized local covariant and contravariant bases, gdd the inverse metric tensor, ax is dual of vector field X, and K, is square-integrable inner product.
Local Coordinates
Cartesian Coordinates
Symbol
Meaning
metric tensor:
8Gz identity matrix
(e ej}
Allow definition of distances and angles on manifolds, just the inner product on Eu- clidean space allows defining distances and angles there.
{gg
af 0z.,
gradient: grad f ff
81 dz, g g
Direction and rate of fastest increase; A tangent vector, which represents an infinites- imal change in (vector) in- put.
01
0f
0.
T
By
0=
(dual of grad) differential: df (V =
01
e
af e² Oz:
0z.
How much the (scalar) out- put changes for a given in- finitesimal change in (vector) input.
af
a
af
By
0=
0F,
divergence: div = .
1 det gF: Oz.
8mg
The volume density of the outward flux vector field from an infinitesimal volume around given point.
Vdt g
BF
BF dz
Gy
(adjoint of d) codifferential:
ar; 0x:
The amount of "stuff" flow- ing through surface locally per unit time, with velocity moving by the vector field.
(ax) div (X) ( = ((ff.
8F
8F,
OF.
8y
8º1
Laplacian: A V f
vg Sæ. (5 af
Local average deviation, how much the average value of function over small balls cen- tered a point deviates from its output.
dæy
0²
0²
0²1
0z2
0y
022
18

---

Under review as a conference paper at ICLR 2023
A.7 IMPLEMENTATION DETAILS
We summarize hyperparameter settings in our experiments in Table 2.
Table 2: Implementation details for the three experiments, synthetic, MNIST and Face.
Hyperparamctee//atasstt
Synthetic
MNIST
UTKFace
Number of generated training samples
60000
Number of random walk steps
5000
Image size
128 x 128
28 x 28
200 x 200 x 3
VAR latent dimension d
6/12
10
16
Step size 8 Explained model performance
0.1
0.02
0.05
MSE< 10-4
ACC> 98%
MAE< 3.0
19

---

Under review as a conference paper at ICLR 2023
A.8
MORE RESULTS FOR SYNTHETIC EXPERIMENTS
Input
True
Manifald
VAE
HeatFlow
VAE 2-12
6-12
VAE
HeatFlow
de-
VAE 6-6
grad R/Go)
Grad
SG
25%
35%
1G
Blur IG
36
200
Input
HeatFlow
True
Manifald
0.5
1.5
2.5
VAE
HeatFlow
sentrect
VAE
HeatFlow
d+b
grad R.
True
Manifold
9%
15%
25%
35%
49%
Input
True
Monifald
VAE
Heatflow
VAE 2-12
6-12
VAE
HeatFlow
&--
VAE 6-6
grad R,70)
Grad
SG
25%
35%
45%
IG
Blur
36
60
84 100
Figure Three more examples for synthetic experiments. HeatFlow under true manifold, and learned manifold by VAE with latent dimension d 12 and d 6 presented in the firt three rows, respectively. On the fourth row, gradients on the true manifold is collected at each time step. SG and Blur IG are presented the last rows.
20

---

Under review as a conference paper at ICLR 2023
HeatFlow
Manifold
VAE d=12
grad Rf(e)
IG
Blur 1G
300
True Manifold
VAE
HeatFlow
dx122
grdd R
True
Menifold
IG
Blur
Input
Munifold
VAE
HeatPlow VAE d=12
dx122
gndd Rf(t))
True
Manifold
Grad
SG
15%
25%
35%
45%
Figure 6: Three examples for another synthetic experiments. HeatFlow with true manifold, VAE with latent dimension d = 12 are presented in the first and second rows, respectively. On the thir row, gradients the true manifold is collected at each time step. SG and Blur IG are presented on the last two rows.
21

---

Under review as a conference paper at ICLR 2023
A.9
RESULTS FOR FACIAL AGE PREDICTION
VAE 6-16
Grad
SG
Blur IG
Input
HeatFlow
VAE 6-16
Iree
Grad
SG
Blur IG
Input
VAE 6-16
tma
Grad
Blur IG
Input
HeatFlow VAT 6-16
Grad
raw 2.%
725
Blur
Input
HeatFlow
27%
12.5%
Figure 7: Five test examples for facial age prediction explanation. HeatFlow using VAE with latent dimension are presented in the first row. Vanilla Grad and IG are shown below the original input. SG and Blur IG are presented the second and thir rows.
22

---

Under review as a conference paper at ICLR 2023
A. ABLATION: COMPARING HEATFLOW ON MANIFOLD AND EUCLIDEAN SPACE
2
3
Figure 8: Four examples comparing HeatFlow on the learned manifold and Euclidean space IR 28 Original input is shown on the most upper-left comner HeatFlow run on the learned manifold and Euclidean space is shown on the first and second rows, respectively.
A.11 EXAMPLE OF ATTRIBUTION FOR ERRONEOUS PREDICTION
f(.)
-5.932
t
1.616
6. 658
10.206
13.043
14.434
Figure 9: Example of attribution for erroneous prediction from the facial age prediction task. The prediction of age of this example is -5.9, which is negative and wrong. As heat flows, indicated by downward arrow, the prediction value of the smoothed model is presented beside each attribution result.
23


Reviews:
[{"review_id": "yH95cuVpr_", "reviewer": "Reviewer_VgMK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose HeatFlow, a framework based on the heat equation on a Riemannian manifold, to interpreting a representation of givn deep learning model. Especially, they focus to analyze the multi-sale behvariors of the model, which cannot be revealed by using a naive gradient-based model interpretation approach. The authors show that the proposed HeatFlow framework obeys some nice properties that frequently used in the interpretable AI field, e.g., attribution axioms. The authors compare the proposed method with four baselines including Grad, IG, SG and BlurIG", "summary_of_the_review": "While it is an educational guess, I think this paper contributes both a solid theoretical background and practial method for the relevant field. I would like to vote to accept this paper. However, since I am not an expert in this field, I cannot evaluate this paper exhaustively; thus, my rating might be updated after discussing with the authors and other reviewers.", "strength_and_weaknesses": "Disclaimer:\n\n I am not an expert of the model interpretaion and explainable AI fields. My evaulation on this paper is not exhaustive and might be incorrect.\n\nStrength:\n\n1. The paper is based on solid theoretical properties from differential geometry and PDE. It seems that the proposed method is technically sound.\n\n2. Experimental results show that the proposed HeatFlow is likely to interprete the model more convincely compared to other baselines.\n\nWeakness:\n\n1. The proposed method can only address a scalar case, i.e., 1D regression problem or binary classification. However, it can be by-passed by investigating each output of multi-class/multi-dimensional models separately.\n\n2. The paper is a little dense and hard to read. It might be because I am not familar with the math used in this paper.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "v0LMVFgG8sL", "reviewer": "Reviewer_s9nq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author proposes the HeatFlow. \nIt calculates the feature attribution using the model's outcome and local average values.\nIt provides multi-scale explainability results.", "summary_of_the_review": "Above all, I think this paper provides an interesting framework to apply the PDE method to the explainability task. \nHeatFlow can provide a fair attribution assignment by obeying four axioms in Shapley value.\nHowever, the paper is hard to follow, and more details should be included to help the readers to understand.\nI am willing to raise my score if the paper flow becomes easier to follow.", "strength_and_weaknesses": "Pros:\nThe proposed FeatFlow is proved to be an additive model and satisfies the four axioms of the Shapley value. \nThis proof provides solid theory support that FeatFlow can provide fair feature attributions.\n\nCons:\nThis paper is a little hard to follow.\nAlthough the authors define each symbol, \nit's still hard to connect these abstract symbols to the explainability task.\n\nFor example, in section 2.1, \nI think it can be better if more details about the Laplacian-Beltrami operator are given and provided some sample examples to show the relationship or motivation to apply this method on explainability.\n\nA detailed pseudo-algorithm may be a better way to show the methodology clearly.\n\nQuestion：\nIn Figure 4, it seems like IG gets the best result on the MNIST exclusion curve and it's better than the proposed HeatFlow.\nHowever, in the UTKFace exclusion curve, it seems like IG has a much lower MAE than heat and SG.\nCould you provide some insight into this phenomenon? I am curious about this.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "d1MuVPEsv1", "reviewer": "Reviewer_uXwu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper under review uses heat diffusion process to understand some multi-scale behavior of the learned model around a test point. Summary scales which characterizes the model on different scales can be drawn. ", "summary_of_the_review": "I would not suggest to accept this paper. It would be better developed before resubmit to a journal or a conference proceeding. ", "strength_and_weaknesses": "Strength: This paper is clearly written and well-organized. \nWeakness: The paper introduces many math notations, while no enough emphasis on why heat flow can help us better understand the functions represented by neural networks for instance. There is no really crucial part which help us advance the understanding of DNN for instance. The results showed here are too toy models. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "LyYcfvf-SCN", "reviewer": "Reviewer_Nuvr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new neural network interpretation and attribution framework which looks at heat flow to achieve multi-scale interpretation of a scalar-valued function", "summary_of_the_review": "The experimental evaluation needs to be more extensive, matching existing published models for interpretability and attribution (see Comment 1 above). Therefore, I do not recommend acceptance as things stand at this stage (but I welcome clarifications and improvements)", "strength_and_weaknesses": "\nPros:\n- It is an interesting and novel idea to use the diffusion equation for the attribution problem\n- The paper is generally well written\n\nCons:\n- Evaluation strategy and analysis (see comment 1 below)\n- Related literature (see comment 2 below)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject