Title: SmoothHess: ReLU Network Feature Interactions via Stein's Lemma
Venue: NeurIPS 2023
Accepted: yes
Keywords: Interpretability

Feature Interactions

Stein's Lemma
Abstract: Several recent methods for interpretability model feature interactions by looking at the Hessian of a neural network. This poses a challenge for ReLU networks, which are piecewise-linear and thus have a zero Hessian almost everywhere. We propose SmoothHess, a method of estimating second-order interactions through Stein's Lemma. In particular, we estimate the Hessian of the network convolved with a Gaussian through an efficient sampling algorithm, requiring only network gradient calls. SmoothHess is applied post-hoc, requires no modifications to the ReLU network architecture, and the extent of smoothing can be controlled explicitly. We provide a non-asymptotic bound on the sample complexity of our estimation procedure. We validate the superior ability of SmoothHess to capture interactions on benchmark datasets and a real-world medical spirometry dataset.

SmoothHess: ReLU Network Feature Interactions via Stein's Lemma

Abstract

Several recent methods for interpretability model feature interactions by looking at the Hessian of a neural network. This poses a challenge for ReLU networks, which are piecewise-linear and thus have a zero Hessian almost everywhere. We propose SmoothHess , a method of estimating second-order interactions through Stein's Lemma. In particular, we estimate the Hessian of the network convolved with a Gaussian through an efficient sampling algorithm, requiring only network gradient calls. SmoothHess is applied post-hoc, requires no modifications to the ReLU network architecture, and the extent of smoothing can be controlled explicitly. We provide a non-asymptotic bound on the sample complexity of our estimation procedure. We validate the superior ability of SmoothHess to capture interactions on benchmark datasets and a real-world medical spirometry dataset.

1 Introduction

As machine learning models are increasingly relied upon in a variety of high-stakes applications such as credit lending [50, 42] medicine [10, 67, 15], or law [38], it is important that users are able to interpret model predictions. To this end, many methods have been developed to assess the importance of individual input features in effecting model output [53, 76, 81, 66, 73]. However, one may achieve a deeper understanding of model behavior by quantifying how features interact to affect predictions. While diverse notions of feature interaction have been proposed [53, 57, 56, 63, 25, 77, 84], in this work, we focus on the intuitive partial-derivative definition of feature interaction [25, 4, 47, 85, 18, 39].

Given a function and point, the Interaction Effect [47] between a given set of features on the output is the partial derivative of the function output with respect to the features; intuitively, it represents the infinitesimal change in the function engendered by a joint infinitesimal change in each chosen feature. The Interaction Effect derives in part from Friedman and Popescu [25], who define the global interaction between a set of features over the data distribution as the expected square of the partial-derivative with respect to those features. As in prior works [47, 39], we eschew the expectation to focus on local interactions occurring around a given point x and avoid squaring partial-derivatives to maintain the directionality of the interaction. We focus on pair-wise feature interactions , which, in the context of the Interaction Effect, are the elements of the Hessian.

ReLU networks are a popular class of neural networks that use ReLU activation functions [28]. The use of ReLU has desirable properties, such as the mitigation of the vanishing gradient problem [28], and it is the sole activation function used in popular neural network families such as ResNet [32] and VGG [72]. However, ReLU networks are piece-wise linear [58] and thus have a zero Hessian almost everywhere (a.e.), posing a problem for quantifying interactions (see Figure 1(a)).

(

x

1

x

2

x

1

x

2

f

*

q

Σ

(a)

(b)

f

x

1

x

2

f

q

Σ

x

1

x

2

0.677

0.67

0.670

,

)

(

*

)(

,

)

A common approach for estimating the Hessian in ReLU networks is to take the Hessian of a smooth surrogate network which approximates the original: each ReLU is replaced with SoftPlus, a smooth approximation to ReLU [28, 22], before differentiating [39, 20]. However, this approach affords one only coarse control over the smoothing as each internal neuron is smoothed, leading to unwieldy asymmetric effects, as can be seen in Figure 2.

In this work, we propose SmoothHess : the Hessian of the ReLU network convolved with a Gaussian. Such a function is a more flexible smooth surrogate than SoftPlus as smoothing is done on the network output , and the covariance of the Gaussian allows one to mediate the contributions of points based on their Mahalanobis distance. Unfortunately, obtaining the Hessian of the convolved ReLU network is impossible using naïve Monte-Carlo (MC) averaging. However, by proving an extension of Stein's Lemma [78, 49], we show that such a quantity can be efficiently estimated using only gradient calls on the original network. For an illustration of ReLU network smoothing, see Figure 1(b).

Our main contributions are as follows:

We propose SmoothHess , the Hessian of the Gaussian smoothing of a neural network, as a model of the second-order feature interactions.

We derive a variant of Stein's Lemma, which allows one to estimate SmoothHess for ReLU networks using only gradient calls.

We prove non-asymptotic sample complexity bounds for our SmoothHess estimator.

We empirically validate the superior flexibility of SmoothHess to capture interactions on MNIST, FMNIST, and CIFAR10. We utilize SmoothHess to derive insights into a network trained on a real-world medical spirometry dataset. Our code is publicly available. ∗

The remainder of this paper is organized as follows: In Sec. 2, we summarize the gradient-based methods for feature importance and interactions that are most related to our work. In Sec. 3, we provide a technical preliminary covering the definitions and techniques underlying both our method and competing works. Next, in Sec. 4, we introduce our method, SmoothHess, explain how to estimate it, and provide sample complexity bounds for our estimation procedure. In Sec. 5, we experimentally demonstrate the ability of SmoothHess to model interactions. Finally, in Sec. 6, we summarize our method and results, followed by a discussion of limitations and possible solutions.

2 Related Work

Feature Importance and First-Order Methods: Methods that quantify feature importance fall into two categories: (i) perturbation-based methods (e.g., [53, 66, 17]), which evaluate the change in model outputs with respect to perturbed inputs, and (ii) gradient-based methods (e.g., [72, 76, 81]), which leverage the natural interpretation of the gradient as infinitesimally local importance for a given sample. Most relevant to our work are gradient-based approaches. The saliency map, as defined in [72], is simply the gradient of model output with respect to the input. Several variants are developed to address the shortcomings of the saliency maps. SmoothGrad [76] was developed

∇ 2

(

*

)(0)

10

3

(b)

(a)

-3

-2

-1

0.0

-1

0

1

2

4

5

6

7

8

9

10

SH :

vs

x

f

q

σ

1,2

10

σ

Smoother

-1

0

1

2

3

4

vs

∇ 2

x

f

β

(0)

1,2

log

10

β

Smoother

3

-1

0

1

2

4

5

6

7

8

9

10

∇ 2

x

(

f

*

q

σ

)(0)

1,2

∇ 2

x

f

β

(0)

1,2

log

σ

2

log

β

10

to address noise by averaging saliency maps (see also Sec. 3), and comes with sample complexity guarantees [3]. Sundararajan et al. [81] introduce Integrated Gradients, the path integral between an input and an uninformative baseline. This is extended to the Shapley framework by Erion et al. [24]. The Grad-CAM line of work [93, 70, 61] is similar in nature to the methods above, with the key distinction that importance is modeled over internal (hidden) layers.

Feature Interactions: A variety of methods have been proposed to estimate higher-order feature interactions, which can again be separated into perturbation-based [54, 82, 86, 57, 92] and gradientbased approaches. Among the latter, Tsang et al. [85] propose Gradient-NID, which estimates feature interaction strength as the corresponding Hessian element squared. Janizek et al. [39] propose Integrated Hessian, which extends Integrated Gradients to use a path-integrated Hessian. Lerman et al. [47] propose Taylor-CAM, a higher-order generalization of Grad-Cam [70]. Cui et al. [18] quantify global interactions for Bayesian neural networks in terms of the expected value of the Hessian over the data distribution. For classification ReLU networks, the CASO and CAFO explanation vectors exploit feature interactions in the loss function using Hessian estimates [74].

Unfortunately, due to their piecewise linearity, existing Hessian-based interaction methods cannot be readily applied to ReLU networks. Janizek et al. [39] replace each ReLU activation with SoftPlus post-hoc, before applying Integrated Hessians. Similarly, Tsang et al. [85] apply their method to networks with the SoftPlus activation instead of ReLU. For regression tasks, Lerman et al. [47] replace ReLU with the smooth activation function GELU [34] before training. Although SoftMax outputs and the cross-entropy loss admit higher order derivatives, pre- or post-hoc smoothing, as above, is necessary for finding interactions affecting logits, internal neurons, or regression outputs. Indeed, while Singla et al. [74] estimate interactions on the original ReLU network, they are only with respect to the loss function.

In contrast, we propose a method for quantifying feature interactions that works with any ReLU network post-hoc without requiring retraining or modifications to the network architecture. It also can be directly estimated with respect to model as well as intermediate layer outputs. Furthermore, our experiments in Sec. 5 show the superior ability of our method to model logit outputs, internal neurons, and SoftMax probabilities as compared to a SoftPlus smoothed network.

3 Technical Preliminary

ReLU Network Background: We denote a ReLU network by F : R d → R c , where c = 1 in the case of regression. We denote the function which we wish to explain as f : R d → R , an arbitrary neuron f ( l ) i (the i th neuron in the l th layer) in our ReLU network, or a SoftMax Probability log

2

for some class k ∈ { 1 , . . . , c } . We denote sample space X ⊆ R d and point x 0 ∈ X for which we wish to capture feature interactions affecting f ( x 0 ) . A general L -hidden-layer ReLU Network F = f ( L +1) : R d → R c may be defined recursively by [33, 58]:

with g (0) ( x 0 ) = x 0 ∈ R d . For a given layer l the dimension (number of neurons) is defined as n l ∈ N , i.e. f ( l ) ( x 0 ) , g ( l ) ( x 0 ) ∈ R n l (with n 0 = d ) with weight and bias W ( l ) ∈ R n l ,n l -1 , b ( l ) ∈ R n l . As stated above, ReLU networks are piecewise linear [58]. Specifically, each neuron f ( l ) i : R d → R , l ∈ { 1 , . . . , L +1 } , i ∈ { 1 , . . . , n l } , is a piecewise linear function, corresponding to a finite set of K ∈ N convex polytopes Q = { Q i } K i =1 , Q i ⊆ R d which form a partition of R d [30, 31, 58, 33, 71, 64, 7]. See Figure 1(a) for an example of a ReLU network.

SmoothGrad: SmoothGrad [76] is an extension of the saliency map ∇ x f ( x 0 ) [72] developed to reduce noise. SmoothGrad is an average over saliency maps; formally, it is E δ [ ∇ x f ( x 0 + δ )] where δ ∼ N (0 , Σ) and the covariance Σ ∈ R d × d is a hyperparameter usually chosen to be isotropic Σ = σ 2 I, σ > 0 . SmoothGrad is estimated by sampling n perturbation vectors { δ i } n i =1 , δ i ∼ N (0 , Σ) , ∀ i ∈ { 1 , . . . , n } and averaging via ˆ G SG n ( x 0 , f, Σ) = 1 n ∑ n i =1 ∇ x f ( x 0 + δ i ) . Unfortunately, the analogous Hessian average 1 n ∑ n i =1 ∇ 2 x f ( x 0 + δ i ) ≈ E δ [ ∇ 2 x f ( x 0 + δ )] is not useful for quantifying feature interactions in ReLU networks as E δ [ ∇ 2 x f ( x 0 + δ )] = 0 .

SoftPlus: A common approach to assessing higher-order derivatives for ReLU networks is by differentiating a smooth surrogate f β : R d → R where a SoftPlus replaces every ReLU in f [28, 39, 20, 21]. The SoftPlus function s β ( x 0 ) = 1 β log(1 + e βx 0 ) , β > 0 , is a smooth approximation of ReLU, where β is a parameter inversely proportional to the level of smoothing [28, 22]. SoftPlus approaches ReLU and f β and ∇ f β approximate f and ∇ f with arbitrary accuracy as β →∞ [20]. Dombrowski et al. [20] establish empirical and theoretical connections between ∇ x f β ( x 0 ) and SmoothGrad. They observe visual similarities between ∇ x f β ( x 0 ) and SmoothGrad at appropriate β values. Further, in the simple case of a single neuron with no bias, they prove that replacing ReLU with SoftPlus corresponds to a Gaussian-like smoothing of the gradient. However, this theory does not extend to networks with multiple neurons and layers.

Stein's Lemma: Stein's Lemma [78] is central to our analysis. We relate a variant of Stein's Lemma [51], which extends the results of Stein [78] to hold for multivariate normal distributions with arbitrary covariance matrices:

Lemma 1. (Stein's Lemma [51]) Given x 0 ∈ R d , covariance matrix Σ ∈ R d × d , multivariate normal random vector δ ∈ R d distributed from δ ∼ N (0 , Σ) and almost everywhere differentiable function g : R d → R for which E δ [ | [ ∇ x g ( x 0 + δ )] i | ] < ∞ for each i ∈ { 1 , . . . , d } , then

A zero-th order oracle associated with a function g : R d → R is an oracle which, when provided with any input x ∈ R d , returns g ( x ) . Likewise, a first order oracle (or gradient oracle) returns ∇ x g ( x ) . In the context of this work, where g is a ReLU network, zero-th order and first order oracle calls amount to network forward passes and backpropagation calls, respectively.

Given one has access to g as a zero-th order oracle, the LHS of Stein's Lemma may be estimated by sampling a set of perturbations { δ i } n i =1 ; δ i ∼ N (0 , Σ) , querying the zero-th order oracle g ( x 0 + δ i ) for each δ i , and MC-estimating, i.e:

Such an approach is useful for estimating E δ [ ∇ x g ( x 0 + δ )] in the RHS of Eq. (3) when the gradient of g is impossible or expensive to obtain but g itself may be efficiently queried as a zero-th order oracle (see e.g., [44, 48, 60]). Extending Liu [51]'s work, Lin et al. [49] present a second-order variant of Stein's Lemma expressing the expected Hessian in terms of the gradient:

Lemma 2. (First-Order Oracle Stein's Lemma [49]) Given x 0 ∈ R d , covariance matrix Σ ∈ R d × d , multivariate normal random vector δ ∈ R d distributed from δ ∼ N (0 , Σ) and continuously differentiable function g ( z ) : R d → R with locally Lipschitz ∗ gradients ∇ g : R d → R d , then

Complexity bounds have been derived for similar identities, which express the Hessian using zero-th order information [6, 94, 23]. However, Lemma 2 fails for ReLU networks. This is precisely because ReLU networks are piecewise linear and, therefore, are not continuously differentiable . In the next section, we directly address this through our method for estimating a smoothed Hessian for ReLU networks.

4 SmoothHess

Our main contributions are: (1) We propose SmoothHess , the Hessian of the network convolved with a Gaussian, for modeling feature interactions. (2) We use Stein's Lemma to prove that SmoothHess may be estimated for ReLU networks using only gradient oracle calls. (3) We prove non-asymptotic sample complexity bounds for our SmoothHess estimator.

Gaussian Convolution as a Smooth Surrogate: An alternative smooth-surrogate to f β is h f, Σ : R d → R , the convolution of f with a Gaussian:

where Σ ∈ R d × d is a covariance matrix, q Σ ( z -x 0 ) = (2 π ) -d 2 | Σ | -1 2 exp( -1 2 d ( x 0 , z ) Σ ) is the density function of the Gaussian distribution N (0 , Σ) , | · | is the determinant and d ( x 0 , z ) Σ = ( x 0 -z ) T Σ -1 ( x 0 -z ) ∈ R is the Mahalanobis distance between x 0 and z .

The Gaussian-smoothed function h f, Σ is infinitely differentiable and does not suffer from the limitations of surrogates obtained from internal smoothing. Here, smoothing is done on the output image of f and thus the relationship between h f, Σ ( x 0 ) , f ( z ) , z and x 0 is made explicit by Eq. (5): the relative contribution of f ( z ) to h f, Σ ( x 0 ) is proportional to the exponentiated negative half Mahalanobis distance -1 2 d ( x 0 , z ) Σ . The ability to adjust Σ gives a user fine-grained and localized control over smoothing; the eigenvectors and eigenvalues of Σ respectively encode directions of input space and a corresponding locality for their contribution to h f, Σ .

We define SmoothHess as the Hessian of h f, Σ :

Definition 1. (SmoothHess) Given ReLU network f : R d → R , point to explain x 0 ∈ R d , covariance matrix Σ ∈ R d × d and q Σ : R d → R , the density function of Gaussian distribution N (0 , Σ) , then SmoothHess is defined to be the Hessian of f convolved with q Σ evaluated at x 0 :

Well-known properties of the Gaussian distribution may be used to encode desiderata into the convolved function and, accordingly, to SmoothHess through the choice of the covariance. For instance, it is known that as d →∞ the isotropic Gaussian distribution N (0 , σ 2 I d × d ) converges to U ( S d -1 σ √ d ) , a uniform distribution over the radius σ √ d sphere [88]. Thus, given large enough d , as is commonly encountered in deep learning datasets, one may choose Σ = ( r/ √ d ) I to approximately ensure that h f, Σ ( x 0 ) incorporates information from the radius r sphere around x 0 . We exploit and validate this intuition in our experiments (see Table 1).

Finally, we must highlight the strong connection between SmoothHess and SmoothGrad [76], which Wang et al. [89] prove is equivalent to the gradient of the same smooth surrogate: ∇ x h f, Σ ( x 0 ) . Thus,

SmoothGrad and SmoothHess together define a second-order Taylor expansion of h f, Σ at x 0 , which can be used as a second-order model of f around x 0 .

SmoothHess Computation via Stein's Lemma: We relate our method for estimating SmoothHess for ReLU networks. As stated above, Lemma 2 does not hold for ReLU networks. However, we extend the arguments of Wang et al. [89] and Lin et al. [49] to show that the LHS from Lemma 2 is equivalent to SmoothHess for all Lipschitz continuous functions ∗ :

Proposition 1. Given x 0 ∈ R d , L -Lipschitz continuous function g : R d → R , covariance matrix Σ ∈ R d × d and random vector δ ∈ R d distributed from δ ∼ N (0 , Σ) with density function q Σ : R d → R , then

where ∗ denotes convolution.

In other words, even though Lemma 2 does not hold for ReLU networks, Stein's Lemma is indeed evaluating a Hessian: namely, SmoothHess given by Eq. (6). The proof consists of moving the Hessian operator on the RHS of Eq. (7) into the integral defined by ( g ∗ q Σ )( x 0 ) . The resulting expression is simplified into a form for which Lin et al. [49] prove is equivalent to the LHS of Eq. (7). The proof is provided in App. B.

Proposition 1 opens up the possibility of an MC-estimate of SmoothHess that only require access to a first-order oracle ∇ f . This is computed by sampling a set of n ∈ N perturbations { δ i } n i =1 , δ i ∼ N (0 , Σ) , and for each δ i querying ∇ x f ( x 0 + δ i ) before taking the outer product δ i [ ∇ x f ( x 0 + δ i )] T ∈ R d × d , Monte-Carlo averaging and finally symmetrizing:

Each first-order oracle call has the same time complexity as one forward pass and may be computed efficiently in batches using standard deep learning frameworks [62, 1, 13]. As n is finite, ˆ H ◦ n ( x 0 , f, Σ) is not guaranteed to be symmetric in practice. Thus, we symmetrize our estimator in Eq. (8b). A straightforward consequence of Proposition 1 is that lim n →∞ ˆ H n ( x 0 , f, Σ) = ∇ 2 x h f, Σ ( x 0 ) , which we formally show in the Proof of Theorem 1 in App. C.

Estimation of SmoothGrad may be amortized with SmoothHess, obtaining ∇ x h f, Σ ( x 0 ) at a significantly reduced cost . The main computational expense when estimating SmoothGrad is the querying of the first-order oracle. As this querying is part of SmoothHess estimation, the gradients which we compute may be averaged at a minimal additional cost of O ( nd ) to obtain a SmoothGrad estimate. Likewise, SmoothHess may be obtained at a reduced cost of O ( nd 2 ) , the cost of the outer products, during SmoothGrad estimation. For details of our algorithm, see App. D.

Last, we prove non-asymptotic bounds for our SmoothHess estimator:

Theorem 1. Let f : R d → R be a piece-wise linear function over a finite partition of R d . Let x 0 ∈ R d , and denote { δ i } n i =1 , a set of n i.i.d random vectors in R d distributed from δ i ∼ N (0 , Σ) . Given ˆ H n ( x 0 , f, Σ) as in Eq. (8) , for any fixed ε, γ ∈ (0 , 1] , given n ≥ 4 ε 2 [max(( C + √ d + √ 1 c + log 4 γ ) 2 , ( C -√ d + √ 1 c -log 4 γ ) 2 )] then

where H = ∇ 2 x [( f ∗ q Σ )( x 0 )] , C + , C -c + , c -> 0 are constants depending on the function f and covariance Σ and q Σ : R d → R is the density function of N (0 , Σ) .

We elect to use the non-asymptotic bounds presented in Theorem 3.39 of Vershynin [87], which hold for sums of outer products of a sub-gaussian random vector with itself. This poses a challenge as ˆ H n is the sum of outer products of two sub-gaussian random vectors which are not necessarily equal . To deal with this issue, we separately prove non-asymptotic bounds for the outer products

of the positive and negative eigenvectors of the summands. This is accomplished using an identity expressing the eigenvalues of the summands in terms of Σ -1 δ and ∇ x f ( x 0 + δ ) , which we state and prove in App. C. The proof is completed by applying the triangle inequality and union bound to combine the two bounds into Eq. (9). Our proof is included in App. C.

5 Experiments

5.1 Experimental Setup

Datasets and Models : Experiments were conducted on a real-world spirometry regression dataset, three image datasets (MNIST [45], FMNIST [90] and CIFAR10 [43]), and one synthetic dataset (Four Quadrant). The Four Quadrant dataset consists of points x ′ ∈ R 2 sampled uniformly from the grid [ -2 , 2] × [2 , 2] ⊆ R 2 , with a spacing of 0 . 008 . The label of a point y ( x ′ ) = Kx ′ 1 x ′ 2 ∈ R is chosen based upon its quadrant, with K = 5 , 3 , 12 , -10 for Quadrants 1 , 2 , 3 and 4 respectively. We train a 5-layer network on MNIST and FMNIST a ResNet-18 [32] on CIFAR10 and a 6-layer network on Four Quadrant. A 10-layer 1-D convolutional neural network was trained on the spirometry regression dataset. Additional model, hyperparameter and dataset details are outlined in App. E.

Hardware: Experiments were performed on an internal cluster using NVIDIA A100 GPUs and AMDEPYC223 7302 16-Core processors.

Metrics : Gradient-based explainers are evaluated using a Taylor expansion-like function as a proxy. We estimate a gradient Hessian pair ˜ G ( x 0 ,g ) ∈ R d , ˜ H ( x 0 ,g ) ∈ R d × d for some function g : R d → R around point x 0 . Here g is either a smooth-surrogate of f or f itself. We define the following Taylor expansion-like function:

For readability, we remove the dependence on x 0 from ˜ G ( x 0 ,g ) and ˜ H ( x 0 ,g ) . Eq. (10) is almost equivalent to the Taylor-expansion of g at x 0 , with the key difference that the zero-th order term is f ( x 0 ) as opposed to g ( x 0 ) . This is done to isolate the impact of the explainers ˜ G g and ˜ H g . Setting ˜ H = 0 yields ˜ f x 0 , ˜ G g , 0 (∆) , a first-order Taylor expansion-like function. For brevity, we refer to ˜ f x 0 , ˜ G g , ˜ H g simply as a Taylor expansion below. We use the following two metrics to quantify the efficacy of SmoothHess:

Perturbation MSE : We introduce the Perturbation Mean-Squared-Error ( P MSE ) to assess the ability of ˜ H and/or ˜ G to capture the behaviour of f over a given neighborhood. Given point x 0 and radius ε > 0 , the P MSE directly measures the fidelity of ˜ f x 0 , ˜ G g , ˜ H g to f when restricted to the ball B ε ( x 0 ) :

A low P MSE value indicates that ˜ f x 0 , ˜ G g , ˜ H g is a good fit to f when restricted to B ε ( x 0 ) , and thus that ˜ G g and ˜ H g capture the behaviour of f over B ε ( x 0 ) . P MSE may be estimated for a point

x 0 by sampling a set of n ∈ N points { x ′ i } n i =1 uniformly from B ε ( x 0 ) , computing the errors and MC-averaging: 1 n ∑ n i =1 ( ˜ f x 0 , ˜ G g , ˜ H g ( x ′ i ) -f ( x ′ i )) 2 ≈ P MSE ( x 0 , f, ˜ f x 0 , ˜ G g , ˜ H g , ε ) .

Adversarial Attacks : Given ˜ G and ˜ H , one can use ˜ f x 0 , ˜ G, ˜ H to construct adversarial attacks of any desired magnitude ε > 0 . We denote ∆ ∗ ∈ R d as the perturbed input, and corresponding attack vectors by ∆ ∗ -x 0 . Given only ˜ G g (i.e. ˜ H g = 0 ), the first-order Taylor expansion attack yields: ∆ ∗ -x 0 = -ε ˜ G g \∥ ˜ G g ∥ 2 . Otherwise, the attack may be framed as the solution to a quadratic optimization minimizing the second-order Taylor expansion:

Although this optimization is non-convex (as ˜ H g is not guaranteed to be positive-semi-definite), it can be solved exactly [12]. For implementation details, and a discussion of the similarities and differences with Singla et al. [74], see App. D. We set g to be the smoothed version of the predicted class SoftMax probability function. Given a set of test points { x i } n i =1 , n ∈ N , we validate the efficacy of our attacks using the average post-hoc accuracy metric 1 n ∑ n i =1 I [ argmax t F ( x i ) = argmax t F (∆ ∗ i )] , where I denotes the indicator function, and ∆ ∗ i the output after x i is attacked.

Setup : We provide the details for our experiments below:

Four Quadrant: We train the network to memorize the Four Quadrant dataset. We measure the interactions at x 0 = (0 , 0) T using both SmoothHess and SoftPlus Hessian, estimated with granularly sampled σ 2 ∈ { 1 e -3 , . . . , 1 } and β ∈ { 1 e -1 , . . . , 4 } .

P MSE : We set g to be the smoothed version of a predicted class logit or a penultimate neuron. The penultimate neuron is chosen to be the maximally activated neuron on average over the train data by the "three," dress and cat classes for MNIST, FMNIST and CIFAR10, respectively. We compute P MSE for three neighborhood sizes ε ∈ { 0 . 25 , 0 . 50 , 1 . 0 } . While over one-hundred values of β are checked on a validation set before selection, only three values of σ are checked for SmoothHess and SmoothGrad, based on the common-sense criterion σ = ε/ √ d : σ ∈ { ε/ 2 √ d, 3 ε/ 4 √ d, ε/ √ d } outlined in Sec 4. The standard deviation of P MSE results is reported in App. F.

Adversarial Attacks: Adversarial attacks are performed after selecting the best β and σ from a held-out validation set. We refrain from using the criterion σ = ε/ √ d for adversarial attacks, as they rely upon the extremal, as opposed to average, behavior of f . We check between ≈ 10 and ≈ 30 values of σ and β on the held-out validation set before selecting the values resulting in the most effective attacks. Attacks are performed using magnitudes ε ∈ { 0 . 25 , 0 . 50 , 0 . 75 , 1 . 25 , 1 . 75 } for MNIST and FMNIST and ε ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 1 . 0 } for CIFAR10, which is easier to successfully attack at lower magnitudes due to it's complexity. For more details see App. D.

Competing Methods : We compare SmoothHess with other gradient-based methods that model f locally; i.e. those which can be associated with the Taylor expansion around a smooth surrogate of f or f itself. Specifically, we compare with the Gradient and Hessian of SoftPlus smoothed network f β [20, 39], SmoothGrad [76] and the vanilla (unsmoothed) Gradient. We also compare adversarial attacks with random vectors scaled to the attack magnitude ε as a baseline.

SmoothGrad

SmoothHess

Sample 1: Early Cough

Sample 2: Late Cough

SmoothHess

SmoothGrad

Spirometry Curve

SG

Interaction

Interaction

SH

0.04

0.010

4000

4000

E

3000

3000

0.005

noothHess

ni

2000

1000

0.00

-0.021000

h

E

2000

0.000

-0.005

E

0

2

D

-0.04

0

-0.010

Time (s)

Time (s)

0.010

4000

E

3000

0.02

G

3000

0.005

noothHess

2000

1000

0.00

-0.02

h

E

2000

1000

0.000

-0.005

E

0

-0.04

0

-0.010

6

6

Time (s)

Time (s)

Additional experiments are provided in App. F. We compare the efficacy of SmoothHess with the unsmoothed Hessian (for adversarial attacks on SoftMax) and Swish [65] smoothed networks, both of which our method outperforms. We also run the P MSE experiment using a ResNet101 model, validating the superior ability of SmoothHess to capture interactions in larger networks.

5.2 Results

Symmetric Smoothing - Four Quadrant Dataset: We investigate the ability of both SmoothHess and the SoftPlus Hessian to capture local feature interactions symmetrically. Results are shown in Figure 2. At each value of σ 2 , aside from extremely small σ 2 < 1 e -2 . 5 , the off-diagonal element of SmoothHess is approximately equal to 2 . 5 , the average interaction over the quadrants. The offdiagonal element of the SoftPlus Hessian is essentially never equal to 2 . 5 . The results for SmoothHess follow from the fact that an isotropic covariance was used, the rotation invariance of which weights points that are equidistant from x equally. The SmoothHess off-diagonal element is not ≈ 2 . 5 at small σ because, despite the network being trained to memorize the data, such a small neighborhood will inherently reflect noise. The inability of SoftPlus to capture interactions symmetrically follows from the fact that smoothing is done internally on each neuron, which does not guarantee symmetry.

Perturbation Mean-Squared-Error: We show SmoothHess can be used to capture interactions occurring in a variety of neighborhoods around x . It is shown in Table 1 that SmoothHess + SmoothGrad (SH + SG) achieves the lowest P MSE for all 18 combinations of dataset, function and neighborhood size ε . We emphasize that this is despite the fact that only three values of σ were validated for SG + SH and SG on a held-out set compared to over one-hundred values of β for SP (H + G) and SP G. This indicates the superior ability of SmoothHess to model f over a diverse set of neighborhood sizes ε . Further, it can be seen that second-order methods SH + SG and SP (H + G) achieve significantly lower P MSE than their first-order counterparts, SG and SP G, respectively, sometimes by an order of magnitude. This confirms the intuition that higher order Taylor expansions of the smooth surrogate provide more accurate models of f . Interestingly, while SH + SG is clearly superior to SP (H + G), we see that SG and SP G are tied in many cases. This could indicate that the symmetric properties of Gaussian smoothing are comparatively more important when higher-order derivatives are considered. However, more investigation is needed.

Adversarial Attacks: We use the interactions found by SmoothHess to generate adversarial attacks on the classifier. We see from Table 2 that the Taylor expansion associated with SmoothHess and SmoothGrad generates the most powerful adversarial attacks for all datasets and attack magnitudes ε , aside from CIFAR10 at ε = 1 . 0 where both methods successfully change most class predictions. The superiority of the SmoothHess attacks indicates that the Gaussian smoothing is capturing the extremal behavior of the predicted SoftMax probability more effectively than the other methods. One hypothesis for SmoothGrad outperforming SmoothHess in the largest neighborhood for CIFAR10 is that the network behavior is highly complex over a large area, and adding higher order terms decreases performance.

Qualitative Analysis of FEV 1 Prediction using Rejected Spirometry:

A spirometry test is a common procedure used to evaluate pulmonary function, and is the main diagnostic tool for lung diseases such as Chronic Obstructive Pulmonary Disease [40]. During an exam, the patient blows into a spirometer, which records the exhalation as a volume-time curve. Recent works have investigated the use of deep learning on spirograms for tasks such as subtyping [11], genetics [16, 91], and mortality prediction [36]. Additional background on spirometry is outlined in App. E.

Traditionally, exhalations interrupted by coughing are discarded for quality control reasons. We train a CNN on raw spirograms from the UK Biobank [79] to predict the patient's "Forced Expiratory Volume in 1 Second" (FEV 1 ), a metric frequently used to evaluate lung health, using efforts that were rejected due to coughing. In Fig. 3, we apply SmoothHess and SmoothGrad on two spirometry samples to understand their FEV 1 predictions. In Sample 1, coughing occurs within the first 4 seconds of the curve, as evidenced by the early plateauing of the curve which indicates a pause in exhalation. In contrast, Sample

2 exhibits no detectable coughing within the first 4 seconds. To improve the interpretability of the results, we group the features into 0.5 second time intervals. FEV 1 is traditionally measured in the initial 2 seconds of the non-rejected samples (see App. E), therefore we calculate SmoothHess interactions with respect to the first 0.5 second time interval.

FEV 1 is known to be strongly affected by the presence of coughing [55]. The time intervals where coughing occurs, indicated by plateaus in the spirometry curves in Figure 3, should be an important signal for any model trained to predict FEV 1 . Indeed, we observe that for Sample 1, the SmoothGrad attribution for the first 0.5s interval is relatively low, with strong interactions occurring at the coughinduced plateaus. In contrast, the first 0.5s interval for Sample 2 shows high importance, with lower magnitude interactions that may be indicative of small fluctuations in the volume-time curve.

In Figure 4 we present the SmoothHess matrix for Sample 1 in Fig. 3, applied on two CNN models with different convolution kernel width. Interestingly, the smaller kernel width constrains the interactions to features that are spatially close together. In contrast, the features in the large kernel model have long-ranging interactions. These results agree with the intuition that smaller kernel widths, which constrain the receptive field of neurons up until the final layers, may correspondingly limit feature interactions.

6 Conclusion and Future Work

We introduce SmoothHess, the Hessian of the network convolved with a Gaussian, as a method for quantifying second-order feature interactions. SmoothHess estimation, which relies only on gradient oracle calls, and cannot be performed with naive MC-averaging, is made possible by an extension of Stein's Lemma that we derive. We provide non-asymptotic complexity bounds for our estimation procedure. In contrast to previous works, our method can be run post-hoc, does not require architecture changes, affords the user localized flexibility over the weighting of input space, and can be run on any network output. We experimentally validate the superior ability of SmoothHess to capture interactions over a variety of neighborhoods compared to competing methods. Last, we use SmoothHess to glean insight into a network trained on real-world spirometry data.

Limitations: The outer product computations in SmoothHess estimation have space and time complexity O ( d 2 ) . When d ≫ 0 this becomes expensive: for instance ImageNet [19] typically has d 2 ≈ 10 10 . This is a common problem for all interaction effect methods. We leave as future work to explore computationally efficient alternatives or approximations to these outer products; a potential remedy is to devise an appropriate power method to estimate the top eigenvectors of SmoothHess, rather than its entirety, which fits well our estimation via sampling low-rank matrices.

References

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: a system for large-scale machine learning. In Osdi , volume 16, pages 265-283. Savannah, GA, USA, 2016.

Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence , 34(11):2274-2282, 2012.

Sushant Agarwal, Shahin Jabbari, Chirag Agarwal, Sohini Upadhyay, Steven Wu, and Himabindu Lakkaraju. Towards the unification and robustness of perturbation and gradient based explanations. In International Conference on Machine Learning , pages 110-119. PMLR, 2021.

Chunrong Ai and Edward C Norton. Interaction terms in logit and probit models. Economics letters , 80(1):123-129, 2003.

David Alvarez-Melis and Tommi S. Jaakkola. On the robustness of interpretability methods. CoRR , abs/1806.08049, 2018.

Krishnakumar Balasubramanian and Saeed Ghadimi. Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics , pages 1-42, 2022.

Randall Balestriero and Richard Baraniuk. Mad max: Affine spline insights into deep learning, 2018. URL https://arxiv.org/abs/1805.06576 .

Naman Bansal, Chirag Agarwal, and Anh Nguyen. SAM: The Sensitivity of Attribution Methods to Hyperparameters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8673-8683, 2020.

Mahsa Bazzaz and Seth Cooper. Active learning for classifying 2d grid-based level completability. arXiv preprint arXiv:2309.04367 , 2023.

Andrew Beam and Ben Kompa. Second opinion needed: Communicating uncertainty in medical artificial intelligence. NPJ Digital Medicine , 4, 2021.

Sandeep Bodduluri, Arie Nakhmani, Joseph M. Reinhardt, Carla G. Wilson, Merry-Lynn McDonald, Ramaraju Rudraraju, Byron C. Jaeger, Nirav R. Bhakta, Peter J. Castaldi, Frank C. Sciurba, Chengcui Zhang, Purushotham V. Bangalore, and Surya P. Bhatt. Deep neural network analyses of spirometry for structural phenotyping of chronic obstructive pulmonary disease. JCI Insight , 5(13), July 2020. doi: 10.1172/jci.insight.132781.

Stephen Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax .

Boris Chen, Amir Ziai, Rebecca Tucker, and Yuchen Xie. Match cutting: Finding cuts with smooth visual transitions. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 2114-2124, 2022. URL https://api.semanticscholar.org/ CorpusID:252846493 .

Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368 , 2019.

Justin Cosentino, Babak Behsaz, Babak Alipanahi, Zachary R. McCaw, Davin Hill, Tae-Hwi Schwantes-An, Dongbing Lai, Andrew Carroll, Brian D. Hobbs, Michael H. Cho, Cory Y. McLean, and Farhad Hormozdiari. Inference of chronic obstructive pulmonary disease with deep learning on raw spirograms identifies new genetic loci and improves risk models. Nature Genetics , April 2023. ISSN 1546-1718. doi: 10.1038/s41588-023-01372-4.

Ian Covert, Scott Lundberg, and Su-In Lee. Feature removal is a unifying principle for model explanation methods. arXiv preprint arXiv:2011.03623 , 2020.

Tianyu Cui, Pekka Marttinen, and Samuel Kaski. Recovering pairwise interactions using neural networks. CoRR , abs/1901.08361, 2019. URL http://arxiv.org/abs/1901.08361 .

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Ann-Kathrin Dombrowski, Maximilian Alber, Christopher J. Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame, 2019. URL https://arxiv.org/abs/1906.07983 .

Ann-Kathrin Dombrowski, Christopher J Anders, Klaus-Robert Müller, and Pan Kessel. Towards robust explanations for deep neural networks. Pattern Recognition , 121:108194, 2022.

Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating second-order functional knowledge for better option pricing. Advances in neural information processing systems , 13, 2000.

Murat A Erdogdu. Newton-stein method: An optimization method for glms via stein's lemma. The Journal of Machine Learning Research , 17(1):7565-7616, 2016.

Gabriel Erion, Joseph D. Janizek, Pascal Sturmfels, Scott M. Lundberg, and Su-In Lee. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature Machine Intelligence , 3(7):620-631, July 2021. ISSN 2522-5839. doi: 10.1038/s42256-021-00343-w.

Jerome Friedman and Bogdan Popescu. Predictive learning via rule ensembles. The Annals of Applied Statistics , 2, 12 2008. doi: 10.1214/07-AOAS148.

Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805 , 2017.

Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of Neural Networks Is Fragile. Proceedings of the AAAI Conference on Artificial Intelligence , 33(01):3681-3688, July 2019. ISSN 2374-3468. doi: 10.1609/aaai.v33i01.33013681.

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 315-323. JMLR Workshop and Conference Proceedings, 2011.

Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning , 110:393-416, 2021.

Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. Advances in neural information processing systems , 32, 2019.

Boris Hanin, Ryan S Jeong, and David Rolnick. Deep relu networks preserve expected length. In International Conference on Learning Representations , 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016.

Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. 2019.

Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. 2016.

Davin Hill, Aria Masoomi, Sandesh Ghimire, Max Torop, and Jennifer Dy. Explanation uncertainty with decision boundary awareness. arXiv preprint arXiv:2210.02419 , 2022.

Davin Hill, Max Torop, Aria Masoomi, Peter Castaldi, Edwin K. Silverman, Sandeep Bodduluri, Surya P Bhatt, Taedong Yun, Cory Y McLean, Farhad Hormozdiari, Jennifer Dy, Michael Cho, and Brian D Hobbs. Deep learning utilizing suboptimal spirometry data to improve lung function and mortality prediction in the uk biobank. medRxiv , 2023. doi: 10.1101/2023.04. 28.23289178. URL https://www.medrxiv.org/content/early/2023/04/29/2023.04. 28.23289178 .

Cheng-Yu Hsieh, Chih-Kuan Yeh, Xuanqing Liu, Pradeep Ravikumar, Seungyeon Kim, Sanjiv Kumar, and Cho-Jui Hsieh. Evaluations and Methods for Explanation through Robustness Analysis. arXiv:2006.00442 [cs, stat] , April 2021.

Zihan Huang, Charles Low, Mengqiu Teng, Hongyi Zhang, Daniel E. Ho, Mark S. Krass, and Matthias Grabmair. Context-aware legal citation recommendation using deep learning. CoRR , abs/2106.10776, 2021. URL https://arxiv.org/abs/2106.10776 .

Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. J. Mach. Learn. Res. , 22:104-1, 2021.

David P. Johns, Julia A. E. Walters, and E. Haydn Walters. Diagnosis and early detection of COPD using spirometry. Journal of thoracic disease , 6(11):1557-1569, November 2014. ISSN 2072-1439 2077-6624. doi: 10.3978/j.issn.2072-1439.2014.08.18.

Zulqarnain Khan, Aria Masoomi, Davin Hill, and Jennifer Dy. Analyzing the effects of classifier lipschitzness on explainers. arXiv preprint arXiv:2206.12481 , 2022.

William Knauth. The self-simplifying machine: Exploiting the structure of piecewise linear neural networks to create interpretable models. arXiv preprint arXiv:2012.01293 , 2020.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Jeffrey Larson, Matt Menickelly, and Stefan M. Wild. Derivative-free optimization methods. Acta Numerica , 28:287-404, 2019. doi: 10.1017/S0962492919000060.

Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: //yann.lecun.com/exdb/mnist/ .

G. Leoni. A First Course in Sobolev Spaces . Graduate studies in mathematics. American Mathematical Soc., 2009. ISBN 978-0-8218-8415-7. URL https://books.google.com/ books?id=W3RLWwnY0RkC .

Samuel Lerman, Charles Venuto, Henry Kautz, and Chenliang Xu. Explaining local, global, and higher-order interactions in deep learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1224-1233, 2021.

Cong Han Lim, Raquel Urtasun, and Ersin Yumer. Hierarchical verification for adversarial robustness. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 6072-6082. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/ lim20b.html .

Wu Lin, Mohammad Emtiyaz Khan, and Mark Schmidt. Stein's lemma for the reparameterization trick with exponential family mixtures, 2019. URL https://arxiv.org/abs/1910. 13398 .

Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue , 16(3):31-57, 2018.

Jun S Liu. Siegel's formula via stein's identities. Statistics & probability letters , 21(3), 1994-10. ISSN 0167-7152.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016.

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems , 30, 2017.

Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable ai for trees. Nature machine intelligence , 2(1):56-67, 2020.

Andrew Z Luo, Eric Whitmire, James W Stout, Drew Martenson, and Shwetak Patel. Automatic characterization of user errors in spirometry. In 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) , pages 4239-4242. IEEE, 2017.

Aria Masoomi, Chieh Wu, Tingting Zhao, Zifeng Wang, Peter Castaldi, and Jennifer Dy. Instance-wise feature grouping. Advances in Neural Information Processing Systems , 33: 13374-13386, 2020.

Aria Masoomi, Davin Hill, Zhonghui Xu, Craig P Hersh, Edwin K Silverman, Peter J Castaldi, Stratis Ioannidis, and Jennifer Dy. Explanations of black-box models based on directional feature interactions. In International Conference on Learning Representations , 2021.

Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems , 27, 2014.

V.C. Moore. Spirometry: step by step. Breathe , 8(3):232-240, 2012. ISSN 1810-6838. doi: 10. 1183/20734735.0021711. URL https://breathe.ersjournals.com/content/8/3/232 .

Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics , 17:527-566, 2017.

Daniel Omeiza, Skyler Speakman, Celia Cintas, and Komminist Weldermariam. Smooth gradcam++: An enhanced inference level visualization technique for deep convolutional neural network models. arXiv preprint arXiv:1908.01224 , 2019.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans. Pattern Anal. Mach. Intell. , 27(8):1226-1238, aug 2005. ISSN 0162-8828. doi: 10.1109/TPAMI.2005.159. URL https://doi.org/10.1109/TPAMI.2005.159 .

Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In international conference on machine learning , pages 2847-2854. PMLR, 2017.

Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941 , 2017.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135-1144, 2016.

Abhijit Guha Roy, Jie Ren, Shekoofeh Azizi, Aaron Loh, Vivek Natarajan, Basil Mustafa, Nick Pawlowski, Jan Freyberg, Yuan Liu, Zach Beaver, et al. Does your dermatology classifier know what it doesn't know? detecting the long-tail of unseen conditions. Medical Image Analysis , 75: 102274, 2022.

H. Royden and P. Fitzpatrick. Real Analysis . Pearson Modern Classics for Advanced Mathematics Series. Pearson, 2017. ISBN 9780134689494. URL https://books.google.com/ books?id=7GP3MAAACAAJ .

Patrick Schwab and Walter Karlen. CXPlain: Causal explanations for model interpretation under uncertainty. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. CoRR , abs/1610.02391, 2016. URL http://arxiv. org/abs/1610.02391 .

Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam. Bounding and counting linear regions of deep neural networks. In International Conference on Machine Learning , pages 4558-4566. PMLR, 2018.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR , abs/1312.6034, 2014.

Sahil Singla, Eric Wallace, Shi Feng, and Soheil Feizi. Understanding impacts of high-order loss approximations and features in deep learning interpretation. In International Conference on Machine Learning , pages 5848-5856. PMLR, 2019.

Dylan Slack, Sophie Hilgard, Sameer Singh, and Himabindu Lakkaraju. Reliable Post hoc Explanations Modeling Uncertainty in Explainability. In Neural Information Processing Systems (NeurIPS) , 2021.

Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 , 2017.

Daria Sorokina, Rich Caruana, Mirek Riedewald, and Daniel Fink. Detecting statistical interactions with additive groves of trees. In Proceedings of the 25th international conference on Machine learning , pages 1000-1007, 2008.

Charles M Stein. Estimation of the mean of a multivariate normal distribution. The annals of Statistics , pages 1135-1151, 1981.

Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliott, Jane Green, Martin Landray, Bette Liu, Paul Matthews, Giok Ong, Jill Pell, Alan Silman, Alan Young, Tim Sprosen, Tim Peakman, and Rory Collins. Uk biobank: An open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLOS Medicine , 12(3):1-10, 03 2015. doi: 10.1371/journal.pmed.1001779. URL https://doi.org/10.1371/journal.pmed.1001779 .

Adam Summerville, Sam Snodgrass, Matthew Guzdial, Christoffer Holmgård, Amy K Hoover, Aaron Isaksen, Andy Nealen, and Julian Togelius. Procedural content generation via machine learning (pcgml). IEEE Transactions on Games , 10(3):257-270, 2018.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning , pages 3319-3328. PMLR, 2017.

Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The Shapley Taylor Interaction Index. In Proceedings of the 37th International Conference on Machine Learning , pages 9259-9268. PMLR, November 2020.

Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 4(2): 26-31, 2012.

Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. In International Conference on Learning Representations , 2018.

Michael Tsang, Dehua Cheng, Hanpeng Liu, Xue Feng, Eric Zhou, and Yan Liu. Feature interaction interpretability: A case for explaining ad-recommendation systems via neural interaction detection. In International Conference on Learning Representations , 2019.

Michael Tsang, Sirisha Rambhatla, and Yan Liu. How does this interaction affect me? interpretable attribution for feature interactions. Advances in neural information processing systems , 33:6147-6159, 2020.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 , 2010.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Zifan Wang, Haofan Wang, Shakul Ramkumar, Piotr Mardziel, Matt Fredrikson, and Anupam Datta. Smoothed geometry for robust attribution. Advances in Neural Information Processing Systems , 33:13623-13634, 2020.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR , abs/1708.07747, 2017. URL http:// arxiv.org/abs/1708.07747 .

Taedong Yun, Justin Cosentino, Babak Behsaz, Zachary R McCaw, Davin Hill, Robert Luben, Dongbing Lai, John Bates, Howard Yang, Tae-Hwi Schwantes-An, Anthony P Khawaja, Andrew Carroll, Brian D Hobbs, Michael H Cho, Cory Y McLean, and Farhad Hormozdiari. Unsupervised representation learning improves genomic discovery for lung function and respiratory disease prediction. medRxiv , 2023. doi: 10.1101/2023.04.28.23289285. URL https://www.medrxiv.org/content/early/2023/04/29/2023.04.28.23289285 .

Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivariate shapley interactions in dnns. In The AAAI Conference on Artificial Intelligence (AAAI) , 2021.

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2921-2929, 2016.

Jingyi Zhu. Hessian estimation via stein's identity in black-box problems. arXiv preprint arXiv:2104.01317 , 2021.

A Societal Impacts

Machine learning models have become increasingly pervasive in society, from medicine [10, 67, 15] and law [38] to entertainment[14, 9, 80]. Therefore, it is important that users of the technology understand the factors underlying model predictions. To this end we propose SmoothHess for quantifying the feature interactions affecting model output. Potential applications of our method are widespread; SmoothHess may be used to find interactions influencing a model to predict whether a customer will default on a credit loan or if a patient has melanoma. The deeper understanding of the model gleaned from SmoothHess may be used to improve decision making. For instance, a doctor may notice that the SmoothHess feature interactions between the pixels in an image of a lesion don't "make sense", indicating that the model should not be trusted and that a more granular human assessment is required. However, such applications are highly sensitive and the cost of inaccurate predictions, or, in this case, misinterpreted attributions, can be high. An inaccurate interpretation can instill unwarranted trust, or mistrust, in a user. In the most extreme cases this may lead to sub-optimal decision making such as the confident denial of a credit loan to a trustworthy customer or misdiagnosis of a benign lesion as malignant.

In addition, it is important to note the challenges related to ensuring robust, stable, and trustworthy explanations. In particular, recent works have uncovered issues related to the sensitivity of the explainer to small changes in the input [5, 41], adversarial attacks [27, 20, 37], or hyperparameter tuning [8]. Methods have been proposed that attempt to quantify explanation uncertainty [75, 69, 35], however further challenges remain. Thus, as with all methods for explaining machine learning model predictions, we recommend that SmoothHess is used in tandem with the careful consideration of domain experts, who are best equipped to interpret interactions in the context of their field.

B Proof of Proposition 1

B.1 Preliminary

We make use of a Lemma from Lin et al. [49] in our proof below, which we relate here:

Lemma 3. (Lin et al. [49]) Denote x 0 ∈ R d , locally-Lipschitz continuous function h ( z ) : R d → R , covariance matrix Σ ∈ R d × d , random vector z ∈ R d distributed from z ∼ N ( x 0 , Σ) . If E z [ | h ( z ) | ] < ∞ , then

B.2 Main Result

Proposition 1. Given x 0 ∈ R d , L -Lipschitz continuous function g : R d → R , covariance matrix Σ ∈ R d × d and random vector δ ∈ R d distributed from δ ∼ N (0 , Σ) with density function q Σ : R d → R , then

where ∗ denotes convolution.

Proof. We define random vector z = x 0 + δ ∈ R d distributed from z ∼ N ( x 0 , Σ) , and for which we denote the density function as p Σ . We begin by showing E z [ | g ( z ) | ] = E δ [ | g ( x 0 + δ ) | ] < ∞ . We are given that g is L -Lipschitz, i.e. ∀ a, b ∈ R d

for some L > 0 .

Now, for any fixed δ ∈ R d we have

Thus, we may bound the expectation E δ [ | g ( x 0 + δ ) | ] :

Here g ( x 0 ) < ∞ as it is a constant, and it may be seen that L E δ [ ∥ δ ∥ 2 ] < ∞ using a simple change of variables. Defining β = Σ -1 2 δ ∼ N (0 , I ) one may write E δ [ ∥ δ ∥ 2 ] = E β [ ∥ Σ 1 2 β ∥ 2 ] . As a straightforward consequence of Cauchy-Schwarz, for any fixed β , one may write

where ∥·∥ F denotes the Frobenius norm. Noting that ∥ β ∥ 2 ∼ X d we use Eq. (17) to see

where Γ( · ) denotes the Gamma function.

Next, we move the Hessian operator inside the integral.

Lemma 3 may be applied as g is Lipschitz, and thus locally-Lipschitz, and E z [ | g ( z ) | ] < ∞ , yielding

Using a change of variables from z to x 0 + δ we write

which, when combined with Eq. (19) and Eq. (20), completes the proof

C Proof of Theorem 1

We begin by establishing a result expressing the eigenvalues of the symmetrization of a rank 1 matrix in closed form:

Lemma 4. Given x, y ∈ R d denote A = xy T + yx T ∈ R d × d . The following facts hold:

Matrix A will have the following eigenvalues

It has d -2 eigenvalues equal to 0

The other two eigenvalues are denoted by λ + ( A ) and λ -( A ) will have the following form:

λ + ( A ) and λ -( A ) are non-negative and non-positive respectively.

Given x and y are sampled from sub-gaussian distributions, then λ + ( A ) and λ -( A ) are sub-exponential random variables.

Proof. We divide the proof into three sections

We have rank ( A ) ≤ 2 , A ∈ S d . Therefore ∃ Q = [ e 1 | e 2 | . . . | e d ] ∈ R d × d s.t. Q T Q = QQ T = I , the column vectors e i ∈ R d are orthonormal and

where Λ = diag ([ λ 1 , λ 2 , 0 , . . . , 0]) ∈ R d × d . As eigenvalues are invariant to change of basis, A has eigenvalues λ 1 and λ 2 and the other d -2 eigenvalues are equal to 0 .

It can be seen that span ( { x, y } ) = span ( { e 1 , e 2 } ) . As span ( { e 1 , e 2 } ) = C ( A ) this can be shown by proving span ( { x, y } ) = C ( A ) . We know ∃ z x ∈ R d : z x ⊥ x and ∃ z y ∈ R d : z y ⊥ y . Thus we have

̸

From the above, we see that x, y ∈ C ( A ) . We know rank ( A ) ≤ 2 . If rank ( A ) = 2 we have x = y and thus span ( { x, y } ) = C ( A ) . If rank ( A ) = 1 we have x = 0 , y = 0 and still span ( { x, y } ) = C ( A ) . If rank ( A ) = 0 we have x = y = 0 and clearly span ( { x, y } ) = { 0 } = C ( A ) .

̸

̸

As { e i } d i =1 are orthonormal, we have that x T e j = y T e j = 0 , ∀ j > 2 . We define

We define Q 2 = [ e 1 | e 2 ] ∈ R d × 2 and

We first show that ∥ ¯ x ∥ 2 and ∥ ¯ y ∥ 2 are equal to ∥ x ∥ 2 and ∥ y ∥ 2 respectively:

Next, we use Eq. (28) to show that ∥ ˜ x ∥ 2 and ∥ ˜ y ∥ 2 are equal to ∥ x ∥ 2 and ∥ y ∥ 2 respectively:

We show an equality between inner products ˜ x T ˜ y = x T y :

˜

x

˜

y

= (

e

T

1

x

)(

e

T

1

y

) + (

e

T

2

x

)(

e

T

2

y

) = ¯

x

¯

y

= (

Q

x

)

Q

y

=

x

QQ

y

=

(30a)

T

T

T

T

T

T

T

Now, one may write,

The following facts hold as a result of Eq. (31d):

λ 1 = 2˜ x 1 ˜ y 1

λ 2 = 2˜ x 2 ˜ y 2

˜ x 1 ˜ y 2 + ˜ x 2 ˜ y 1 = 0

Now, we show

Which can be derived as such:

where the second equality comes from the following:

Proving that Eq. (32) holds. Finally, we combine Eq. (29) and Eq. (30) with Eq. (32) to express the eigenvalues of A as:

which we use to denote λ + ( A ) = λ 1 , λ -( A ) = λ 2 .

As | x T y | ≤ ∥ x ∥ 2 ∥ y ∥ 2 , it follows that λ + ( A ) = x T y + ∥ x ∥ 2 ∥ y ∥ 2 ≥ 0 and λ -( A ) = x T y -∥ x ∥ 2 ∥ y ∥ 2 ≤ 0 .

We denote D = { 1 , . . . , d } It can be seen that x i y i is sub-exponential ∀ i ∈ D , as a sub-gaussian times a sub-gaussian is sub exponential. Thus, it follows that

as the sum of sub-exponential random variables is sub-exponential. Further we see that x 2 i and y 2 i are sub-exponential as the square of a sub-gaussian is sub-exponential. As the sum of sub-exponentials is sub-exponential we have that ∑ d i =1 x 2 i , ∑ d i =1 y 2 i are both subexponential random variables. As the square root of a sub-exponential is sub-gaussian we have that

As a sub-gaussian times a sub-gaussian is sub-exponential, from Eq. 37 we have that

Now we see from Eq. and 36 Eq. 38 that both λ + ( A ) and λ -( A ) are the sum of subexponential random variables and thus are sub-exponential.

We now use the result of Lemma 4 to prove the sample complexity bounds for SmoothHess in Theorem 1:

Theorem 1. Let f : R d → R be a piece-wise linear function over a finite partition of R d . Let x 0 ∈ R d , and denote { δ i } n i =1 , a set of n i.i.d random vectors in R d distributed from δ i ∼ N (0 , Σ) . Given ˆ H n ( x 0 , f, Σ) as in Eq. (8) , for any fixed ε, γ ∈ (0 , 1] , given n ≥ 4 ε 2 [max(( C + √ d + √ 1 c + log 4 γ ) 2 , ( C -√ d + √ 1 c -log 4 γ ) 2 )] then

where H = ∇ 2 x [( f ∗ q Σ )( x 0 )] , C + , C -c + , c -> 0 are constants depending on the function f and covariance Σ and q Σ : R d → R is the density function of N (0 , Σ) .

Proof. As x 0 , f and Σ are fixed, we refer to ˆ H n ( f, x 0 , Σ) as ˆ H n for brevity. We denote D = { 1 , . . . , d } . We begin by explicitly expressing our estimator ˆ H n in terms of δ i and ∇ x f ( x 0 + δ i ) . From Eq. (8) we have

Now, we show the convergence of our estimator ˆ H n :

Lemma 5. lim n →∞ ˆ H n = H

Proof. From Proposition 1 it is clear to see that lim n →∞ H ◦ n = H :

Next, we show it is also the case that lim n →∞ H ◦ T n = H :

Now, as we have lim n →∞ H ◦ n = lim n →∞ H ◦ T n = H , it follows that

We establish the following notation to be used below: given a fixed vector δ ∈ R d one may construct matrix A δ ∈ R d × d by:

(Symmetry of Hessian) (41c)

It can be seen from Eq. 39 and Lemma 5 that H and ˆ H n may be expressed in terms of matrices A δ and A δ i :

Next, we establish that the random vectors Σ -1 δ and ∇ f ( x 0 + δ ) are sub-gaussian:

Lemma 6. The random vectors Σ -1 δ and ∇ f ( x 0 + δ ) are sub-gaussian.

Proof. As Σ -1 δ is Gaussian it is sub-gaussian. Now, we show that ∇ f ( x 0 + δ ) is a sub-gaussian random-vector. We have that f is piecewise-linear over a partition of R d with finite cardinality L . Let us denote this partition as Q = { Q i } L i =1 , Q i ⊆ R d , where, when restricted to a given Q ∈ Q we have

where V Q ∈ R d , A Q ∈ R are the affine coefficients associated with the region Q . Then it is the case that ∇ f : R d → R is a bounded function, where, aside from a set of measure 0 , M ⊆ R d (the boundaries of regions Q ) where ∇ f is not defined, one has

Thus, ∇ f ( x 0 + δ ) is a bounded random vector and therefore is sub-gaussian.

Given the operators λ + , λ -: R d × d → R as defined in the statement of Lemma 4 and fixed vector δ ∈ R d , we denote λ + δ := λ + ( A δ ) , λ -δ := λ -( A δ ) and the corresponding unit eigenvectors as v + δ ∈ R d and v -δ ∈ R d respectively. We denote random vectors w + δ , w -δ ∈ R d by

where √ λ + δ and v + δ are a random variable random vector pair coming from the same δ . An immediate consequence of Lemma 6 is that w + δ and w -δ are sub-gaussian random vectors:

Lemma 7. w + δ and w -δ are sub-gaussian random-vectors

Proof. Using Lemma 4(3), Lemma 4(2) and Lemma 6 we see that that λ + δ and -λ -δ are non-negative sub-exponential random variables and thus that √ λ + δ and √ -λ -δ are sub-gaussian random variables. We may say w + δ is a sub-gaussian random vector if ⟨ w + δ , z ⟩ is a sub-gaussian random variable ∀ z ∈ R d . Let us fix arbitrary z ∈ R d . As √ λ + δ is sub-gaussian we have

Now, ∀ t ≥ 0

Thus defining K ( z ) 1 := K 1 ∥ z ∥ 2 we see that ∀ t ≥ 0

Thus ⟨ w + δ , z ⟩ is sub-gaussian for arbitrary z ∈ R d . Therefore, w + δ is a sub-gaussian random-vector. The same argument holds to show that w -δ is a sub-gaussian random vector.

Given any fixed δ , it can be seen that

Thus one may re-write H and ˆ H n from Eq. 44 as:

Because { δ i } n i =1 are i.i.d. random vectors and w + δ i , w -δ i are fully determined by δ i , it follows that { w + δ i } n i =1 and { w -δ i } n i =1 are both sets of i.i.d. random vectors.

We denote the following:

It can be seen from the RHS of Eq. (52) that

We now aim to decompose H in terms of H + , H -, in order to derive separate concentration bounds. To this end, we prove the following lemma:

Lemma 8. H + and H -exist.

Proof. Let us consider the random matrix w + δ w + T δ ∈ R d × d . As ( w + δ ) k ∈ R is sub-gaussian ∀ k ∈ D , the element ( w + δ w + T δ ) ij ∈ R is sub-exponential as the product of two sub-gaussian's, ∀ i, j ∈ D . Thus, E δ [( w + δ w + T δ ) ij ] exists ∀ i, j ∈ D . The same argument can be made to show E δ [( w -δ w -T δ ) ij ] exists ∀ i, j ∈ D .

In light of Lemma 8, the LHS of Eq. (52) may be decomposed as

Before deriving separate concentration bounds on ˆ H + n and ˆ H -n we note that

Finally, we bound the deviation of ˆ H + n and ˆ H -n from their expectations. Let us fix ε, γ ∈ (0 , 1] . From Eq. (55), and the fact that { w + δ i } n i =1 is a set of i.i.d. sub-gaussian random vectors, Theorem 3.39, Remark 3.40 of Vershynin [87] may be applied, yielding: ∀ t ≥ 0

where ε + n = C + √ d √ n + t √ n and C + , c + > 0 are constants depending on the sub-gaussian norm of w + δ i . Let us select t = √ log(4 /γ ) c + . Plugging into Eq. (57), we get

Let us consider n + = 4 ε 2 ( C + √ d + √ log(4 /γ ) c + ) 2 . One may see that

Thus, given n = n + one has ε + n = max( ε 2 , ( ε 2 ) 2 ) = ε 2 , because ε 2 < ε ≤ 1 , and that

In fact, because ε + n is monotonically decreasing in n , given n ≥ n + Eq. (60) holds.

The same logic above may be used to show that there exists constants C -, c -> 0 depending on the sub-gaussian norm of w -δ such that, given n ≥ n -= 4 ε 2 ( C -√ d + √ log(4 /γ ) c -) 2 one has

Finally, we combine the two bounds from Eq. (61) and Eq. (60). Given n ≥ max( 4 ε 2 ( C + √ d + √ log(4 /γ ) c + ) 2 , 4 ε 2 ( C -√ d + √ log(4 /γ ) c -) 2 ) we have

D Implementation Details

D.1 Quadratic Optimization

Given a function f : R d → R , point x 0 ∈ R d gradient Hessian pair G ∈ R d , H ∈ R d × d and a magnitude constraint ε > 0 , we aim to solve the following optimization:

as f ( x 0 ) is constant, the problem above is equivalent to

where we have replaced ∆ -x 0 , which can be interpreted as the output after the attack with δ = ∆ -x 0 , the attack vector itself.

The optimization problem in Eq. (64) is non-convex as H is not guaranteed to be positive semidefinite. However, as Slater's constraint qualification is satisfied, i.e. ∃ δ ∈ R d s.t. ∥ δ ∥ 2 < ε , Eq. (64) may be solved exactly[12]. Specifically, the solution may be obtained by solving an equivalent convex optimization problem:

where S d denotes the the set of symmetric matrices and ⪰ indicates the block matrix [ X,γ ; γ T , 1] ∈ R ( d +1) × ( d +1) is constrained to be positive semi-definite.

However, the optimization in Eq. (65) is expensive to solve when d ≫ 0 as there are O ( d 2 ) variables. For instance, MNIST and FMNIST have d 2 ≈ 6 . 0 · 10 6 and CIFAR10 has d 2 ≈ 10 8 . Thus, before converting Eq. (64) into Eq. (65), we elect to reduce the dimension of the optimization problem to k ∈ N , k ≪ d .

Let us consider the eigendecomposition Q Λ Q T = H . Here the columns of Q ∈ R d × d are orthonormal eigenvectors of H . Given the d eigenvalues { λ 1 , . . . , λ d } , sorted such that i < j = ⇒ | λ i | ≥ | λ j | , we have Λ = diag ( λ 1 , . . . , λ d ) . We remove the last d -k columns from Q and d -k columns and rows from Λ to construct ˜ Q ∈ R d × k , ˜ Λ ∈ R k × k . Thus, we have a low-rank approximation of H :

Thus, Eq. (64) is approximately equivalent to another optimization which uses this low-rank approximation for H :

Defining ˜ δ = ˜ Q T δ ∈ R k , Eq. (67) is approximately equivalent to

where the constraint is simplified to ∥ ˜ δ ∥ 2 ≤ ε from ∥ ˜ Q ˜ δ ∥ 2 ≤ ε as

Finally, ˜ δ ∗ ∈ R k , the optimal solution to Eq. (68), is projected back to R d yielding an approximate solution to Eq. (67):

Choosing k: The choice of k ∈ N is determined using a threshold hyperparameter T ∈ (0 , 1] . Given T , k is chosen to be the smallest number of (sorted) eigenvalues/eigenvectors which account for a proportion of the total eigenvalue magnitude that is at least T :

For the values of T used for each dataset see App. E

D.1.1 Similarities and Differences with CASO/CAFO

Our quadratic optimization is closely connected to the CAFO/CASO explanation vectors proposed by Singla et al. [74]. Both methods use optimizations to minimize a value outputted by the network, and the CASO method also uses a second-order Taylor expansion in their objective. Additionally, the proposed Smooth-CASO method is equivalent to an attack using SmoothGrad and SmoothHess on the cross-entropy loss function (when their regularizers are set to 0 ), which admits higher order derivatives.

However, there are some key differences. First, our optimization is meant for use with arbitrary network outputs as opposed to just the loss, as is the case with CAFO/CASO. While the vanilla loss Hessian modeled by Singla et al. [74] is proven to be positive semi-definite, this is not the case for arbitrary functions which one may wish to attack. Thus methods to optimize non-convex quadratic objectives, such as those outlined in App. D.1, are required. Second, our goal is not to generate an explanatory feature importance vector for analysis, as is an important motivation for CAFO/CASO, but to assess the quality of the gradient Hessian pair used to attack the function. For this reason, we do not use sparsity constraints such as in Singla et al. [74], which are in part meant to improve the interpretability of CAFO/CASO as explainers. Last we stress that the techniques used to find the cross-entropy loss Hessian for CASO and Smooth-CASO cannot be used for internal neurons, logits or regression valued output in ReLU networks, due to their piecewise-linearity.

D.2 SmoothHess

The SmoothHess estimation procedure, and the amortization with SmoothGrad estimation, is presented below in Algorithm 1. While empirically our SmoothHess estimator converges, we have additionally found that reflecting each point δ i in the perturbation set { δ i } n i =1 about the origin to create an augmented perturbation set { δ i } n i =1 ∪ {-δ i } n i =1 before estimation can result in faster per-sample convergence.

Algorithm 1 Joint SmoothHess and SmoothGrad Estimation

d d

D.3 Alternative Covariance Matrices

While in this work we use isotropic covariance matrices of the form Σ = σ 2 I, σ > 0 , SmoothHess can be estimated using arbitrary positive definite covariance matrices Σ ∈ R d × d . Such a covariance matrix can be set according to the users preference, encoded using the fact that the eigenvectors of Σ represent directions of interest and their corresponding eigenvalues represent levels of smoothing.

In the simplest case the user already has an orthornomal eigenvector basis in mind, a 1 , . . . , a d ∈ R d as well as desired levels of smoothing σ 1 , . . . , σ d > 0 . In this case Σ may be set by constructing eigenvector and eigenvalue matrices Q = [ a 1 | . . . | a d ] ∈ R d × d and Λ = diag ( σ 1 , . . . , σ d ) ∈ R d × d and simply multiplying Σ = Q Λ Q T .

Alternatively, the user may only have k < d orthonormal eigenvectors in mind, a 1 , . . . , a k ∈ R d , for which they wish to smooth at specific levels σ 1 , . . . , σ k > 0 . A procedure such as Gram-Schmidt may be used to find a k +1 , . . . , a d ∈ R d which extends a 1 , . . . , a k to an orthonormal basis of R d . Again, the smoothing levels along the eigenvectors a k +1 , . . . , a d may be chosen according to user preference. For instance, one may select σ k +1 = . . . = σ d ≈ 0 if minimal smoothing is desired along these directions. Just as above Σ may be set by constructing eigenvector and eigenvalue matrices Q = [ a 1 | . . . | a d ] ∈ R d × d and Λ = diag ( σ 1 , . . . , σ d ) ∈ R d × d and simply multiplying Σ = Q Λ Q T .

E Experiment Setup

E.1 Datasets and Models

In this work we make use of six datasets, two synthetic datasets (Four Quadrants, Nested Interactions) three benchmark datasets (MNIST, FMNIST, CIFAR10) and a real world medical dataset (Spirometry). Below we describe these datasets and training details.

Four Quadrant . The Four Quadrant dataset consists of points x ∈ R 2 sampled from the grid [ -2 , 2] × [ -2 , 2] with a spacing of 0 . 008 . A 6-layer fully connected ReLU network was trained using RMSProp [83] on the Four Quadrant dataset achieving a final mean-squared-error of ≈ 1 e -4 . Training lasted for 40 , 000 iterations with a batch size of 128 and a starting learning rate of 1 e -3 which was decayed by a factor of 1 e -1 at iterations 5000 , 10 , 000 and 20 , 000 .

Nested Interactions . The Nested Interactions dataset consists of points x ∈ R 2 sampled from the grid [ -2 , 2] × [ -2 , 2] with a spacing of 0 . 008 . A 6-layer fully connected ReLU network was trained using RMSProp on the Nested Interactions dataset achieving a final mean-squared-error of ≈ 1 e -1 . Training lasted for 200 , 000 iterations with a batch size of 64 and a starting learning rate of 1 e -3 which was decayed by a factor of 1 e -1 at iterations 40 , 000 , 80 , 000 , 120 , 000 and 160 , 000 . For more details on Nested Interactions see App. F.3.

MNIST MNIST consists of 70,000 28x28 greyscale images, each corresponding with one of the digits 0 -9 . There are 60 , 000 and 10 , 000 images in the pre-defined train and test sets respectively. We further split the train set into 50 , 000 images for training and 10 , 000 for validation. A 5-layer fully connected network with dimensions 500 -300 -250 -250 -250 was trained with stochastic gradient descent for 30 epochs, with a batch size of 128 and a starting learning rate of 1 e -2 which was decayed by a factor of 1 e -1 at iterations 4 , 000 and 8 , 000 . Final accuracies of ≈ 100% (Train) ≈ 100% (Val) and ≈ 98% (Test) were achieved. All images are flattened. For P MSE all methods were evaluated on 200 test points. For adversarial attacks all methods were evaluated on 400 test points. The threshold for the quadratic optimization attack was set to T = 0 . 98 .

FMNIST . FMNIST consists of 70,000 28x28 greyscale images, each corresponding with one of 10 articles of clothing. There are 60 , 000 and 10 , 000 images in the pre-defined train and test sets respectively. We further split the train set into 50 , 000 images for training and 10 , 000 for validation. Final accuracies of ≈ 93% (Train) ≈ 93% (Val) and ≈ 88% (Test) were achieved. The network and training details are identical to that used for MNIST above. All images are flattened. For P MSE all methods were evaluated on 200 test points. For adversarial attacks all methods were evaluated on 400 test points. The threshold for the quadratic optimization attack was set to T = 0 . 98 .

CIFAR10 CIFAR10 consists of 60,000 3x32x32 RGB color images, each corresponding with an animal or vehicle. There are 50,000 and 10,000 images in the pre-defined train and test sets respectively. We further split the train set into 40 , 000 images for training and 10 , 000 for validation. AResNet-18 [32] was trained on CIFAR10 for 55 epochs using a batch size of 128 . The first 5 epochs were used for warmup with a starting learning rate of 1 e -2 ending at 0 . 5 . For the rest of training a cosine decay schedule was used [52] decaying down to 1 e -5 by the final epoch. Augmentations used for training were (i) Random Horizontal Flip ( p = 0 . 5) (ii) Color Jitter ( p = 0 . 8) with brightness, contrast and saturation values equal to 0 . 4 and hue value 0 . 1 (iii) Random Grayscale ( p = 0 . 2) . Final accuracies of ≈ 85% (Train, Augmentations) ≈ 96% (Val, No Augmentations) and ≈ 90% (Test, No Augmentations) were achieved. No augmentations were applied to the validation/test data when evaluating explainers. For P MSE all methods were evaluated on 100 test points. For adversarial attacks all methods were evaluated on 200 test points. The threshold for the quadratic optimization attack was set to T = 0 . 8

Spirometry. The Spirometry dataset uses raw exhalation curves, measured in volume over time, recorded during a spirometry exam. Each spirometry curve is measured in 10ms intervals, which we downsample to 50ms intervals and limit to 15s in total length, resulting in 300 features. We use the UK Biobank dataset, which is a large, population-based study conducted in the United Kingdom. Participant statistics have been previously reported in Sudlow et al. [79]. The UK Biobank records 2-3 exhalation efforts for each participant, using a Vitalograph Pneumotrac 6800 device ∗ . If two efforts are recorded as passing acceptability criteria and are also reproducible ( ≤ 5% difference in Forced Vital Capacity (FVC) and Forced Expiratory Volume in 1 Second (FEV 1 )), then the third

effort is omitted. A common metric used to evaluate lung health is the Forced Expiratory Volume in 1 Second (FEV 1 ), which is the maximum volume of air that can be expelled by the participant in 1 second [59]. Note that a participant's FEV 1 measurement is taken as the maximum FEV 1 over all recorded exhalation efforts during a single visit.

The spirometer automatically evaluates effort against a number of acceptability criteria. One such criteria is the detection of coughing. In our experiment, we use a subset of exhalation efforts where coughing was detected and train a CNN to predict the participant's final FEV 1 measurement. This subset contains 8,721 samples, which we split into training (80%) and test (20%) partitions. We follow the preprocessing in Hill et al. [36] to ensure that participants have at least one effort that passes quality control where FEV 1 can be measured. The trained CNN includes 10 convolution blocks. Each block contains a 1-d convolution of kernel width 200 and 20 channels, batch normalization, dropout (p= 0 . 5 ), and skip connection. The model is trained using mean squared error (MSE), achieving 0.563 MSE on the train set and 0.547 MSE on the test set.

E.2 Hyperparameters

For ease of reading, we use the following notation below: [ a : b : c ] = { a, a + c, a +2 c, . . . , b -2 c, b -c } ⊆ R denotes the set of points between a (inclusive) and b (exclusive) at intervals of size c . Here a and b are chosen such that a < b and ( b -a ) mod c = 0 . An example is: [0 . 1 : 1 . 0 : 0 . 1] = { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .

E.2.1 P MSE

For each of the 18 combinations of dataset, function and neighborhood size ε the performance of β and σ are validated on a held out set before selection. 200 validation points are used to choose σ, β for MNIST and FMNIST and 50 validation points are used for CIFAR10. For SmoothHess + SmoothGrad and SmoothGrad only three values of σ are validated, based on the common sense criterion σ = ε/ √ d : given neighborhood size ε and dataset with dimension d , σ is chosen from σ ∈ { ε/ 2 √ d, 3 ε/ 4 √ d, ε/ √ d } .

The following values of β are checked on a validation set:

The values of σ for SH and SH + SG, and β for SP H + G and SP G which achieve the lowest P MSE on the validation data are shown in Table 3. The results for SoftPlus β are interesting: (i) We see that MNIST and FMNIST results, for both class logit and interior neuron, are consistent for fixed ε . Further we see that CIFAR10 results between class logit and interior neuron are consistent for fixed ε . (ii) The optimal value of β seems to be approximately proportional to the value of ε . While our results in Table 1 show that SmoothHess is better at capturing local interactions than the SoftPlus Hessian, the results in Table 3 indicate that the relationship between f , f β and β warrants further exploration.

E.2.2 Adversarial Attacks

200 validation points were used to choose β and σ 2 for MNIST and FMNIST and 50 validation points were used to choose β and σ 2 for CIFAR10.

The following values of σ 2 were validated for adversarial attacks:

MNIST and FMNIST: σ 2 ∈ { 0 . 001 , 0 . 005 } ∪ [0 . 01 : 0 . 1 : 0 . 01] ∪ [0 . 15 : 1 . 00 : 0 . 05]

CIFAR10: σ 2 ∈ { 5 e -05 , 7 . 5 e -05 }∪ [0 . 0001 : 0 . 0006 : 0 . 0001] ∪{ 0 . 00075 }∪ [0 . 001 : 0 . 006 : 0 . 001]

The following values of β were validated for adversarial attacks:

MNIST and FMNIST: β ∈ [1 : 20 : 1] ∪ [20 : 100 : 5] ∪ [100 : 210 : 10] .

CIFAR10: β ∈ [1 : 10 : 1] ∪ [10 : 45 : 5]

The values of σ 2 and β that achieve lowest validation post-hoc accuracy are shown in Table 4. Following intuition, it is generally the case that parameters corresponding with increased smoothing (larger σ 2 and smaller β ) achieve better results (lower post-hoc accuracy) for large ε , and parameters corresponding to less smoothing (smaller σ 2 and larger β ) achieve better results for small ε .

F Additional Experiments

F.1 P MSE

Comparison with Swish: In Table 1, P MSE results are presented for five methods: the first (SmoothGrad) and second (SmoothHess + SmoothGrad) order Taylor expansions of the ReLU network f convolved with a Gaussian, the first and second order Taylor expansions of the SoftPlus smoothed network and the vanilla (unsmoothed) Gradient. We present Table 5, a version of Table 1 which includes additional results comparing with Swish [65] smoothed networks. Swish, an alternative smooth activation to SoftPlus, is formally defined as Sw β ( x ) = x sigmoid ( βx ) where sigmoid ( x ) = 1 1+exp( -x ) and β is a hyperparamter determining the level of smoothing. It can be seen in Table 5 that Swish is generally less effective then SoftPlus, and is outperformed by our method at each combination of dataset and locality.

Standard Deviation: We report the standard deviation of the P MSE for each method, dataset, function and neighborhood size ε in Table 6. Of the 18 P MSE results, SmoothHess + SmoothGrad attains the lowest standard deviation for 15 and ties with SoftPlus Hessian + SoftPlus Gradient for 2 . The standard deviation of SoftPlus Hessian + SoftPlus Gradient is the lowest for FMNIST internal neuron at ε = 0 . 5 , achieving 2 . 4 e -7 while SmoothHess + SmoothGrad achieves 2 . 5 e7 .

ResNet101: We repeat our P MSE experiment for the predicted class logits of CIFAR10 using a ResNet101, reporting results in Table 7. It can be seen in the leftmost column that our method, SH + SG, achieves superior performance to the competing methods at each locality, indicating that SmoothHess can generalize to larger network architectures. The standard deviation of P MSE , as well as choices of σ 2 / β (as selected from a validation set) are reported in the center and rigthmost columns, respectively.

F.2 Adversarial Attacks

Comparison with Vanilla Hessian: One may use the vanilla Hessian of the predicted SoftMax probability, which admits higher order derivatives, to construct adversarial attacks. Table 8 is an updated version of Table 2 which includes results for attacks using the vanilla Hessian + vanilla Gradient (H + G), in the third to last row.

We see that inclusion of the vanilla Hessian generally results in more effective attacks then use of the vanilla gradient alone. H + G ties SH+SG and SP H + G for lowest post-hoc accuracy at the

smallest magnitude ( ε = 0 . 25 ) for the simplest dataset (MNIST). However, as no smoothing is done, the attacks generated from the H + G are generally significantly weaker then those generated using smooth surrogates. In fact, aside from MNIST with ε = 0 . 25 , the second order vanilla H + G attacks achieve higher post-hoc accuracy then first-order method SmoothGrad for all datasets and values of ε . This is especially apparent for CIFAR10, the most complex dataset.

F.3 Nested Interactions

We use the Nested Interactions dataset to highlight SmoothHess's ability to capture different interactions occurring at various localities around a given point. In this experiment we measure interactions around the origin x 0 = (0 , 0) T ∈ R 2 .

Just like the Four Quadrant dataset, the Nested Interactions dataset consists of points x ∈ R 2 sampled uniformly from [ -2 , 2] × [ -2 , 2] ⊂ R 2 with a spacing of 0 . 008 . We establish different interactions occurring around the origin x 0 , based upon the distance from x 0 . Specifically, we set the label for a given point x by: x ∈ B 0 . 6 ( x 0 ) = ⇒ y ( x ) = 1 2 x 2 1 + x 1 x 2 , x ∈ B 1 . 2 ( x 0 ) \ B 0 . 6 ( x 0 ) = ⇒ y ( x ) = x 1 x 2 , x ∈ R 2 \ B 1 . 2 ( x 0 ) = ⇒ y ( x ) = -5 x 1 x 2 .

In words, the interaction between features x 1 and x 2 is 1 inside the radius1 . 2 ball around x 0 and is -5 outside of this ball. The interaction between x 1 and itself is 1 inside the radius0 . 6 ball around x 0 and 0 outside of this ball. The interaction between x 2 and itself is 0 over all of R 2 .

We train a 6-layer neural network on the Nested Interactions dataset and estimate SmoothHess and SoftPlus Hessian for σ 2 ∈ { 1 e -6 , . . . , 1 e 1 } and β ∈ { 1 e -1 , . . . , 4 . 0 } respectively. The interaction results for x 1 with itself, x 1 with x 2 and x 2 with itself as a function of the level of smoothing ( σ or β ) are reported in Figure 5.

As the target function y ( x ) is discontinuous, it is not possible for a network to memorize the Nested Interactions dataset. Thus, there very well may be interactions occurring in the network which are not described as above; the interactions we know occur in the data are not a pure "gold-standard". That being said, Figure 5 shows that SmoothHess captures the interactions as we know occur in the data, and the SoftPlus Hessian does not. This suggests that, to a large extent, both (i) the network has

memorized the data and (ii) SmoothHess captures the network behaviour while the SoftPlus Hessian does not.

F.4 Qualitative Comparison

We present a visual comparison of the interactions found by SmoothHess with those from other methods. Namely, we consider methods that can be interpreted as the quadratic term in a second-order Taylor expansion around a smooth surrogate network: SoftPlus Hessian and Swish Hessian.

We show interactions found between super-pixels of CIFAR10 test images. To this end, we utilize the Simple Linear Iterative Clustering (SLIC) [2] algorithm to segment the image into 20-25 super-pixels. We sum interactions between each pair of features in each pair of super-pixels, before visualization.

Results are shown in Figure 6 for the predicted class logit of a ResNet18 model for three CIFAR10 test images. Here, each row corresponds to a separate image. Test images are visualized in column 1. Columns 2 -4 correspond to the three methods. For each image, interactions between one chosen

2

(a)

-5.0

-4.0

-3.0

-2.0

-6.0

-1.0

0.0

x

1

,

x

1

x

1

,

x

2

x

2

,

x

2

2.0

-1.0

0.0

1.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

-3.0

-2.0

-4.0

-5.0

SmoothHess values

Smoother

-1.0

0.0

1.0

2.0

3.0

4.0

(b)

Smoother

2.0

-1.0

0.0

1.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

-3.0

-2.0

-4.0

-5.0

log

10

σ

2

log

10

β

SH :

vs

∇ 2

x

(

f

*

q

σ

)(0)

log

10

σ

vs

∇ 2

x

f

β

(0)

log

10

β

super-pixel (outlined in black) and each other super-pixel are visualized as a heatmap overlaid upon the image. In order to facilitate comparison across images and methods, we standardize the heatmap colorbar to range between the most negative and most positive interaction values on a per-image and method basis.

One interesting trend seen in each case is that there is a strong positive interaction between the chosen super-pixel and one other super-pixel which (a) is spatially nearby and (b) contains the class object of interest. For example, in the first row, the side-view mirror of the car positively interacts with the front wheel. In the second row, the tip of the frogs head can be seen to interact positively with the side of the head. In the third row, the upper and lower portions of the dogs front leg have a strong positive interaction.

Due to the subjectivity of this comparison, we include quantitative results above each image. Specifically, we indicate the P MSE each method achieves within an ε = 0 . 25 ball around each image. Optimal smoothing parameters were chosen for each method for this task (see Table 3). It can be seen that SmoothHess achieves the lowest P MSE in each case by a wide margin. Thus, SmoothHess may be the preferable option if one wishes for a visualization which best reflects the network's behaviour in an ε = 0 . 25 ball around the image.

Interaction

Min Negative

Interaction

No Interaction

Test Image

SmoothHess

SoftPlus Hessian

Swish Hessian

/u1D4AB

MSE

= 2.0

e

-

4

/u1D4AB

MSE

= 1.0

e

-

4

/u1D4AB

MSE

= 1.0

e

-

4

/u1D4AB

MSE

= 7.0

e

-

4

/u1D4AB

MSE

= 3.0

e

-

4

/u1D4AB

MSE

= 4.0

e

-

4

/u1D4AB

MSE

= 1.5

e

-

3

/u1D4AB

MSE

= 7.0

e

-

4

/u1D4AB

MSE

= 8.0

e

-

4

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFor smooth functions the joint interaction between pairs of features and their impact on the function output can be modeled and quantified via second order partial derivatives. The Hessian of a neural network can therefore provide insight as to how interactions between pairs of features impact predictions made by the network. ReLU networks are piecewise linear and therefore have a zero Hessian almost everywhere, as a result a different approach to quantifying interactions is needed. One way to overcome this issue is to consider a smoothed approximation of the network. Prior works substitute ReLU for Softplus and compute the Hessian of the corresponding smooth network, however, the sensitivity to the $\\beta$ parameter of the Softplus activation is pronounced. Furthermore this method affords poor control over the level of smoothing. This work considers instead estimating interactions by computing the Hessian of the convolution of the network with a Gaussian - this Hessian is referred to as SmoothHess. Furthermore, using Stein's Lemma it is shown that one can efficiently estimate SmoothHess from first order information alone using Monte Carlo simulation. Non-asymptotic bounds on the error of this estimate are also provided in Theorem 3. Experimental results demonstrating improvements of this method over existing techniques, at least in the context of relatively simple problems such as MNIST and CIFAR10, are provided.\n\nSTRENGTHS:\nThis paper appears to provide a new and natural method with which to estimate the impact of interactions between pairs of features on the network output in the setting in which the network of the Hessian is zero almost everywhere. The method seems principled and outperforms existing methods on a number of benchmarks. The technical results appear correct although I confess I did not check them thoroughly. The paper is generally well written and clear. I am not an expert in this space so cannot say much to as to its broader context with respect to prior works.\n\nWEAKNESSES:\nOne minor concern might be around how one actually chooses the covariance matrix in practice. However, choosing hyperparameters is a problem for the other methods as well it seems and furthermore this method appears to give you more direct control of the smoothing.\n\nQUESTIONS:\n1. Are these techniques limited to ReLU networks or could one in fact apply them to a more general class of functions for which the first and second order partial derivatives are not defined everywhere (non-smooth) and or zero almost everywhere?\n\n2. How does one choose the covariance matrix and hence the degree of smoothing? \n\n3. Can you identify any (potentially pathological) examples in which SmoothHess really fails to model interaction effects? In short could you provide further comment on potential failure modes of the method and highlight any warnings one should be aware of when using it?\n\nLIMITATIONS:\nLimitations of the work, in particular the computational complexity with regard to the input dimension, are highlighted and discussed in the conclusion. A link to code is included in the supplementary material. Potential social impacts are also discussed in the supplementary material: the authors are careful to point out that as with any method used to explain decisions made by a model one needs to treat such outputs with care and due oversight, and that it is best to work in tandem with domain experts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe goal of this paper is to model the point-wise interactions between input features in a ReLU network. ReLU networks are piece-wise linear, which inhibits using Hessian to model feature interaction (since it’s zero almost everywhere). This paper proposes SmoothHess, a new method to compute feature interaction by the Hessian of a smooth surrogate of the ReLU network. The authors prove that SmoothHess can be estimated using only gradient calls, and prove non-asymptotic sample complexity bounds.\n\nOne result that’s particularly elegant is the connection between SmoothHess and SmoothGrad: combined, they define a local, second-order Taylor expansion of the network. The author designed several evaluations of the proposed method based on this connection.\n\nThe main baseline that SmoothHess is compared against is SoftPlus Hessian which is based on an alternative smooth surrogate of ReLU. These methods are evaluated as local second-order estimates of the ReLU model: they are evaluated by the accuracy of their local approximation, as well as the capability to guide adversarial attacks. SmoothHess performs best in both tasks on three standard image classification datasets. The authors also performed qualitative analysis on a regression problem from the medical domain.\n\nSTRENGTHS:\nAlthough feature interaction is a well-studied topic, I find the proposed method to be quite novel. The motivation of the problem, i.e., the piece-wise linearity of ReLU networks is very clear and the survey of related work seems thorough.\nThe proof sketches provided in the main paper are explained clearly, with detailed references to pieces of proofs from other work.\nOverall, I find the paper to be very well-written and easy to follow. It strikes a great balance between having detailed theoretical results and clearly conveying the intuition.\n\nWEAKNESSES:\nThe main weakness of the paper is the comprehensiveness of experiments. Due to computational cost of computing SmoothHess, all experiments are conducted on relatively small-scale datasets and small-ish models. Although I do not see reasons why this method would not work on larger models, it would make the paper much stronger to have empirical evidence.\n\nI also find the qualitative analysis a bit dry. Part of this is due to my lack of familiarity with the Spirometry regression task. But I cannot confidently evaluate the significance of that result, in particular regarding what it means for interpretability of ReLU networks in general. Can this method improve tasks that can benefit from interpretability, like error detection, trust calibration, debugging, auditing, or knowledge discovery? I'm not sure.\n\nThis is a bit disappointing since interpretability is a primary motivation for this work. It would help to at least have some qualitative comparison (and with more competing methods) on more familiar datasets to at least help us understand the behavioral differences between SmoothHess and existing methods.\n\nQUESTIONS:\nRegarding Equation 10: if my understanding is correct, what Equation 10 does is it defines how SmoothHess (and competing methods) can be used in conjunction with gradient methods to define a local, second-order model of the underlying ReLU model. This equation is then used as the basis of experiments which evaluates the goodness of fit from several aspects. Is this understanding correct? Are there alternatives to Equation 10? Is there an alternative way to do similar experiments without Equation 10? Is it possible for this specific definition (for example by using f instead of g as the first term) to bias the results?\n\nLIMITATIONS:\nThe authors have adequately address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SmoothHess, a method to define a smooth approximate Hessian for networks where many components of the Hessian may be zero (e.g. ReLU networks). This can aid in tasks where interactions between features are needed, but where those components of the exact Hessian are zero. SmoothHess is defined as the convolution of the output of the network with the Gaussian. It is easy to show, using Stein's lemma, that this is equivalent to the expectation of the gradient of a perturbed output over that Gaussian distribution. They then do a number of experiments. In the first two sets of experiments (perturbation MSE and adversatrial attakcs) on three datasets to show the advantages of using SmoothHess over alternative smoothings of ReLU, such as softPlus. In both experiments, SmoothHess significantly outperforms SofPlus. They also test this on a 4 quadrants dataset with similar results. They also apply this to a real-world medical dataset of spirometry to measure feature interactions.\n\nSTRENGTHS:\nThe paper shows the benefit of using a smoothened Hessian of ReLU networks for tasks involving feature interactions. The benefit of the method is that it is post-hoc and can be applied to the output of a trained network without a modification to the architecture. The theoretical derivations are easy to follow and well organized. The experiments also make a good case for using their construction, showing superiority to SoftPlus.\nOverall, the simplicity of the method is its main strength.\n\nWEAKNESSES:\nMy main reservation is how impactful the work can be. It is a neat idea, but with a rather limited scope. For example, given the expense of computing SmoothHess and the fact that we only need it for diagonal blocks of the Hessian of ReLU networks (explained next) makes the impact rather limited for me. Maybe the authors can explain or emphasize more the breadth of the problems where this would be useful or required?   \nOne minor issue to raise is that, as I understand it, the Hessian of ReLU networks is *not* zero almost everywhere. In multi-layer networks, only the Hessian components involving weights in the same layer are zero. All cross layer Hessian terms should be nonzero. Let $W^l_i$ denote a component indexed by $i$ in the weights of layer $l$. The Hessian term   \n$${\\partial^2 F(x) \\over \\partial W^l_i \\partial W^k_j} $$   \nis in general not zero, even in ReLU networks. \nIn other words, the paper's claims about vanishing Hessian only hold for block diagonal components of the Hessian. They need to argue for why these components are more important than off-diagonal blocks.  \nAdditionally, the baselines in the experiment are rather limited. SoftPlus is not the only smooth alternative to ReLU. The swish activation and similar functions could have been compared as well.\n\nQUESTIONS:\n1. Why only consider diagonal Hessian components, while off-diagonal blocks of ReLU networks can be non-zero?\n2. How do baselines other than SoftPlus (e.g. swish) perform?\n3. In the spirometry experiment, what is the conclusion? Can you trust or validate that the interactions revealed by SmoothHess are real? Does it help predict which datapoints to discard, or is it about which ones not to drop? \n4. regarding the quadratic computation of SmoothHess, is there an alternative for feature interactions which is not quadratic?\n\nLIMITATIONS:\nThe paper does discuss limitations, notably the quadratic cost in feature dimensions, which is prohibitively expensive for large models/datasets. I don't see an immediate societal impact issue.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose to compute Hessian of a ReLU neural network by a convolution of the network function with a Gaussian distribution function. It's equivalent (or cover) some prior work, e.g., SmoothGrad. The method is generally easy to understand, without modifying the network structure. A non-asymptotic bound on the sample complexity is provided.\n\nSTRENGTHS:\nStrengths:\n-- The proposed smooth-surrogate is easy and straightforward to understand, the computation method is somewhat novel using Stein's Lemma, following [40] with a mild extension to all Lipschitz continuous functions, which addresses the challenge posed by the zero Hessian of piecewise-linear ReLU networks. The paper provides a non-asymptotic bound on the sample complexity of the estimation procedure. \n\n-- Proposed SmoothHess can be applied post-hoc and does not require any modifications to the ReLU network architecture. This allows analysis and interpretation of the interactions of existing ReLU neural networks without retraining or redesigning the model.\n\n-- Efficient sampling: SmoothHess utilizes an efficient sampling algorithm, which only requires network gradient calls, making it computationally feasible to large-scale networks and complex datasets.\n\nWEAKNESSES:\nWeaknesses:\n-- Scope limited to ReLU networks: While SmoothHess addresses the challenge of zero Hessian in ReLU networks, its applicability is limited to this specific activation function. It would be useful to discuss the potential generalizability of the method to other types of neural networks with different activation functions.\n\n-- Generalize beyond simple datasets: SmoothHess can be applied to large models and datasets with its simple computations, it would be beneficial to clarify the generalizability of SmoothHess and the potential limitations in terms of dataset diversity.\n\nQUESTIONS:\nSome notations are confusing, e.g., caption of figure 2, is it $\\log(10\\sigma)$ or $\\log_{10}{\\sigma}$? \n\nalso the text, \"Aside from at minute\", what does \"minute\" mean here?\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes an approach for measuring feature interactions for neural networks that have no\nuseful second order gradients or Hessians (notably those employing ReLU). The approach convolves a\nsmooth normal with the network as a proxy with usable Hessians and then offers a means of\nestimating them using sampling of first-order gradients, thereby making it more tractable than\nmethods that rely on computing Hessians naively. Experiments evaluate the proposed method and\nseveral adaptations of existing methods for computing 1) the 3rd-term (second order gradient) of\nthe Taylor expansion of the output of a neural network around an input, and 2) for adversarial\ninputs via optimization using (1). In these experimental terms, the proposed approach (aided by\nSmoothGrad for the 2nd Taylor term) performs better than those based on only the first order\ngradient or those that use include both first and second order gradient but proxy ReLU models using\nSmoothPlus activation.\n\nSTRENGTHS:\n+ Almost strictly better experimental results than the compared-to methods.\n\n+ Hessians for networks which don't naively have them due to technicalities should be generally\n  useful in a wide variety of applications.\n\nWEAKNESSES:\n- Experimentation focuses on agreement between a network and one evaluated as per Taylor expansion\n  using SmoothHess. While this may be a fairly general test, it does not offer any insight into\n  benefits in cases where the method would actually be useful to use, with no trivial alternatives.\n  Presumably in most cases we can already evaluate the network regardless of Hessian non-existence\n  so at surface level there is no need for Taylor expansion and thus the Hessian of the smooth\n  proxy. The adversarial attack is also based on use of Taylor expansion of smoothed model as proxy\n  so it does not improve this problem. The final case-study does present interaction results but is\n  difficult to follow (see questions below). I suggest, then, that the paper include evaluation of\n  the methods that focus on the Hessian, instead of network output. One example is to test\n  adversarial attacks on gradient-based attributions by gradient descent on the gradient (i.e.\n  using Hessian). Compare to other methods for attacking attributions.\n\n- Tradeoffs with respect to the parameters are not explored or described. The use of smoothing\n  serves to increase the impact of network regions further away from each point but smoothing is a\n  double-edged sword. If smoothing (and subsequent sampling) is insufficient compared to the sizes\n  of linear network regions, the results may not be useful. On the other hand, large enough\n  smoothing reduces the network to a constant.\n\n  One impact of this tradeoff is how fast gradient descent can arrive at good solutions. Thus in\n  addition to the above suggestion of testing usefulness of the Hessian, I also suggest that the\n  effect of the parameters (or smoothness) have on that usefulness. For example, sufficiently\n  significant smoothness I expect to result in useless gradients of gradients for attacking\n  attributions.\n\nSmaller things:\n\n- What is δ (without subscript) in Equation 8a? Did you mean δᵢ ?\n\n- On line 117, the expectation and perturbation are presumed not significant to the point that the\n  Hessian is 0 almost everywhere.\n\n- By \"second-order interactions\" did you mean interactions via second-order gradients? Interactions\n  seem already second-order so saying second-order interactions is suggesting third-order\n  gradients.\n\n- In Table 1, are the two best (bolded) results in the 4th numerical column identical or is it just\n  that the printout identical?\n\n- Around line 255, you write \"We denote Δ* ... as the output after x₀ is attacked\". This may be a\n  bit confusing as I think you mean Δ* as the perturbed input, not the network output. Is that\n  right?\n\n- Line 349 writes \"Interestingly, the smaller kernel width constraints the interactions to features\n  within a small locality.\" Why is that interesting? Isn't that expected?\n\n- Typo near \"between between\".\n\n- The Adversarial Attacks paragraph on line 254 and Equation 12a is missing the target class of the\n  attack.\n\nQUESTIONS:\n- Conceptually, when sampling as per MC-estimation, if samples end up not spanning more than 1\n  linear region of the original network, would the estimate be zero in the same sense as second\n  order gradient is zero?\n\n- What is the impact of smoothness/covariance on sample bound as per Equation 9?\n\n- Does \"SP H + G\" mean \"SoftPlus (Hessian + Gradient)\" or \"(SoftPlus Hessian) + (ReLU Gradient)\"?\n  Line 285 suggests the latter but the label/caption suggest the first. If the first, the\n  comparison may not be as fair as it could be.\n\n- Example of Figure 3 is difficult to understand. Why are the arrows labeled \"interaction\" crossing\n  from one method to another? Also, SmoothGrad vs. SmoothHess is indicated by both color and the\n  separate graphs? Did you mean that green/red are indicators of positive or negative interactions\n  for both of those methods or for one respectively? Or perhaps the second of each pair of graphs\n  shows the interaction of the other timesteps with the first timestemp only? Also, what is the\n  kernel width for the example shown in Figure 3?\n\n- The covariance matrix is set in a specific way in the text but the benefit of the option to set\n  it otherwise are mentioned. How would one go about setting it alternatively and for what\n  purposes?\n\nLIMITATIONS:\nThe work describes experiments with adversarial attacks which could use some mention with relation\nto negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author introduces a novel method to approximate the Hessian of a smoothed\nloss function at a specific point. By incorporating a variation of Stein’s lemma,\nthey cleverly estimate the gradient over a smoothed surrogate of a ReLU network\nat that point. To gauge the method’s effectiveness, they measure its performance\nby testing the normalized L2 distance between the loss function and its first-\norder and second-order approximations within a ball centered around the given\npoint.\n\nSTRENGTHS:\nThe method offers a straightforward and comprehensible approach, presenting\na powerful and novel tool for approximating the Hessian of a ReLU network at\na specific point. The author’s analysis is comprehensive and remarkably easy\nto follow. The experiments conducted in the paper appear relevant, and the\nobtained results show promising potential.\n\nWEAKNESSES:\nWhile the ideas and methods of the paper are solid, I came across some read-\nability issues.\n\n1.\nThe statement of Theorem 3 (line 211) is confusing. The real positive\n$\\delta$ is used without declaration, while the random variables $\\delta_1,\\ldots,\\delta_n$ are declared. This makes the reader wonder how one can compare a probability with a random vector, and what is the meaning of this inequality which is essentially a random variable itself. I had to read the proof in the appendix to fully understand the statement of the theorem. I think using a different symbol for $\\delta$ might increase readability.\n\n2.\nI would add the definitions of the oracles used in the article for clarity.\n\nQUESTIONS:\n1. line 103 \"An general L-hidden...\" is a typo?\n\n2. line 287 \"with the non-smooothed...\" is a typo?\n\nLIMITATIONS:\nThe article address the time and space complexity of the method in the presence of a high-dimensional input.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-08-01"}