Title: Trade-off Between Efficiency and Consistency for Removal-based Explanations
Venue: NeurIPS 2023
Accepted: yes
Keywords: AI interpretability

explainable AI

deep learning theory
Abstract: In the current landscape of explanation methodologies, most predominant approaches, such as SHAP and LIME, employ removal-based techniques to evaluate the impact of individual features by simulating various scenarios with specific features omitted. Nonetheless, these methods primarily emphasize efficiency in the original context, often resulting in general inconsistencies. In this paper, we demonstrate that such inconsistency is an inherent aspect of these approaches by establishing the Impossible Trinity Theorem, which posits that interpretability, efficiency, and consistency cannot hold simultaneously. Recognizing that the attainment of an ideal explanation remains elusive, we propose the utilization of interpretation error as a metric to gauge inefficiencies and inconsistencies. To this end, we present two novel algorithms founded on the standard polynomial basis, aimed at minimizing interpretation error. Our empirical findings indicate that the proposed methods achieve a substantial reduction in interpretation error, up to 31.8 times lower when compared to alternative techniques.

Trade-off Between Efficiency and Consistency for Removal-based Explanations

Abstract

In the current landscape of explanation methodologies, most predominant approaches, such as SHAP and LIME, employ removal-based techniques to evaluate the impact of individual features by simulating various scenarios with specific features omitted. Nonetheless, these methods primarily emphasize efficiency in the original context, often resulting in general inconsistencies. In this paper, we demonstrate that such inconsistency is an inherent aspect of these approaches by establishing the Impossible Trinity Theorem, which posits that interpretability, efficiency, and consistency cannot hold simultaneously. Recognizing that the attainment of an ideal explanation remains elusive, we propose the utilization of interpretation error as a metric to gauge inefficiencies and inconsistencies. To this end, we present two novel algorithms founded on the standard polynomial basis, aimed at minimizing interpretation error. Our empirical findings indicate that the proposed methods achieve a substantial reduction in interpretation error, up to 31.8 times lower when compared to alternative techniques ‡ .

1 Introduction

Most existing explanation approaches are removal-based [18], which involve the sequential process of eliminating certain input features, examining the subsequent alterations in the model's behavior, and ascertaining each feature's impact through observation. However, most of these methods [60, 38] primarily focus on the original input with all features, often yielding inconsistent outcomes in alternative scenarios. Consequently, the resulting interpretations may not explain the network's behavior consistently even in a small neighborhood of the input, as demonstrated in Figure 1.

Inconsistency is a non-trivial concern. Imagine a doctor treating a diabetic patient with the help of an AI system. The patient has features A, B, and C, representing three positive signals from various tests. The AI recommends administering 4 units of insulin with the following explanation: A, B, and C have weights 1 , 1 , and 2 , respectively, amounting to 4 units in total. The doctor might then ask the AI: what if the patient only has A and B, but not C? One might expect the answer to be close to 2 , as A+B has a weight of 2 . However, the network, being highly non-linear, might output a different suggestion like 3 units, explaining that both A and B have a weight of 1 . 5 . Such inconsistent

behaviors can significantly reduce the doctor's confidence in the interpretations, limiting the practical value of the AI system.

this

movie

is

very

good

12.7%

-3.6%

+

6.8%

-

=

15.0%

13.2%

+

+

prediction

baseline

-

94.3%

50.4%

+

Interpretation

(

)

this

movie

is

good

-5.3%

+

9.1%

-

=

19.2%

+

prediction

baseline

-

87.7%

50.4%

+

14.3%

Interpretation

(

)

Consistency (see Definition 2.2) is certainly not the only objective for interpretability. Being equally important, efficiency is a commonly used axiom in the attribution methods [72, 22, 59], also called local accuracy [38] or completeness [60], stating that the model's output should be equal to the network's output for the given input (see Definition 2.3). Naturally, one may ask the following question:

Q1: Can we generate an interpreting model that is both consistent and efficient?

Unfortunately, this is generally unattainable. We have proved the following theorem in Section 2:

Theorem 1 (Impossible trinity, informal version). For post-hoc interpreting models, interpretability, efficiency and consistency cannot hold simultaneously.

A few examples following Theorem 1:

(a) Most attribution methods are interpretable and efficient, but not consistent.

(b) The original (deep) network is consistent and efficient, but not interpretable.

(c) If one model is interpretable and consistent, it cannot be efficient.

However, consistency is necessary for many scenarios, leading to the follow-up question:

Q2: For consistent interpreting models, can they be approximately efficient?

The answer depends on the definition of 'approximately efficient'. We introduce a new notion called truthfulness , which serves as a natural relaxation of efficiency, or partial efficiency. We divide the functional space of a network f into two subspaces: the readable part and the unreadable part. We call the interpreting model g truthful if it can accurately represent the readable part of f , denoted as V (see Definition 2.4). The unreadable part is not merely a 'fitting error'; it truthfully represents the higher-order non-linearities in the network that our interpretation model g , even at its best, cannot cover. In short, what g conveys is true, though it may not encompass the entire truth. Due to Theorem 1, this is essentially the best that consistent algorithms can achieve.

Truthfulness is a parameterized notion, depending on the choice of the readable subspace. While there are theoretically infinite possible choices, we follow previous researchers on interpretability with non-linearities [61, 41, 63], using the basis that induces interpretable terms like x i x j or x i x j x k . These capture higher-order correlations and are easy to understand. The resulting subspace has the standard polynomial basis (or equivalently, the Fourier basis).

When truthfulness is parameterized with the polynomial basis, designing consistent and truthful interpreting models is equivalent to learning the Fourier spectrum (see Algorithm 1 and Lemma 3.4). However, exact consistency is not always necessary for most real-world applications, as approximate consistency is usually sufficient. We formally use the number of different interpreting models (in log scale) as a metric for inconsistency. Our last question is:

Q3: When a little inconsistency is allowed, can we get better interpreting models?

We affirmatively answer this question in Section 4 by introducing a new algorithm called Harmonicalocal. We then apply it multiple times to develop Harmonica-anchor and Harmonica-anchorconstrained, which offer smaller interpretation errors with a small degree of inconsistency.

In this paper, we focus on removal-based explanations [18], which implies that f and g are Boolean functions. We remark that empirically most networks are not Boolean functions, i.e., the input variables are real valued. However, for every explanation algorithm (including LIME [47] and SHAP [38]) in the removal-based framework, the input x is not modified to an arbitrary real value. Instead, x is fixed, and the algorithm only retain or remove each feature of x for generating the explanations. When x is fixed with n features, it becomes natural to use Boolean functions to represent both the network f and the interpreting model g for their outputs in all 2 n feature-removal scenarios. In other words, the algorithms in the framework only consider a fixed x each time, and given this x , there are only 2 n possible outcomes, even for the models with real inputs. Therefore, treating f and g as Boolean functions is not a simplification, but an accurate characterization of the removal-based framework (see Figure 1 in [18] for an illustration).

Our new algorithms, including Harmonica-local, Harmonica-anchor, and Harmonica-anchorconstrained, are all based on the Harmonica algorithm from Boolean functional analysis [27], which has rigorous theoretical guarantees on recovery performance and sampling complexities. In Section 5, we demonstrate that on datasets like IMDb and ImageNet, our algorithms achieve up to 31 . 8 x lower interpretation error compared with other methods.

In summary, our contributions are:

We prove the impossible trinity theorem for removal-based explanations, demonstrating that interpretable algorithms cannot be consistent and efficient simultaneously.

When a small inconsistency is allowed, we propose new algorithms using Harmonica-local and empirically demonstrate that these algorithms achieve significantly lower interpretation errors compared with other methods.

For interpretable algorithms that are consistent but not efficient, we introduce a new notion called truthfulness, which can be regarded as partial efficiency. Due to the impossible trinity theorem, this is the best achievable outcome when consistency is required.

2 Our Framework on Interpretability

We consider a Hilbert space H equipped with inner product ⟨· , ·⟩ , and induced norm ∥ · ∥ . We denote the input space by X , the output space by Y , which means H ⊆ X → Y . We use G ⊂ H to denote the set of interpretable functions, and F ⊂ H to denote the set of machine learning models that need interpretation. In this paper, if not mentioned otherwise we focus on models that are not self-interpretable, i.e., f ∈ F \ G .

Definition 2.1 (Interpretable and Interpretation Algorithm) . We call A model g is interpretable , if g ∈ G . An interpretation algorithm A takes f ∈ H , x ∈ X as inputs, and outputs A ( f, x ) ∈ G for interpreting f on x .

As we mentioned previously, for many interdisciplinary fields, the interpretation algorithm should be consistent.

Definition 2.2 (Consistent) . Given f ∈ H , an interpretation algorithm A is consistent with respect to f , if A ( f, x ) remains the same (function) for every x ∈ X .

Efficiency is an important property of the attribution methods.

Definition 2.3 (Efficient) . A model g ∈ H is efficient with respect to f ∈ F on x ∈ X , if g ( x ) = f ( x ) .

The following theorem states that one cannot expect to achieve the best of all three worlds.

Theorem 1 ( Impossible Trinity for Removal-based Explanations ) . For any interpretation algorithm A and function sets G ⊂ F ⊆ H , there exists f ∈ F such that with respect to f , either A is not consistent, or A ( f, x ) is not efficient on x for some x ∈ X .

Proof. Please refer to Appendix B for all the proofs.

Theorem 1 says efficiency is too restrictive for consistent interpretations. However, being inefficient does not mean the interpretation is wrong, it can still be truthful. Recall a subspace V ⊂ H is closed if whenever { f n } ⊂ V converges to some f ∈ H , then f ∈ V . We have:

Definition 2.4 (Truthful gap and truthful) . Given a closed subspace V ⊆ H , g ∈ G ⊆ V and f ∈ F , the truthful gap of g to f for V is:

When T V ( f, g ) = 0 , we say g is truthful for subspace V with respect to f , and we know (see e.g. Lemma 4.1 in [56]) ∀ v ∈ V, ⟨ f -g, v ⟩ = 0 .

Truthfulness means g fully captures the information in the subspace V of f , therefore it can be seen as a natural relaxation of efficiency. To characterize the interpretation quality, we introduce the following notion.

Definition 2.5 (Interpretation error) . Given functions f, g ∈ X → Y , the interpretation error between f and g with respect to measure µ is

Notice that interpretation error is only a loss function that measures the quality of the interpretation, instead of a metric in ℓ p space. Therefore, µ can be a non-uniform weight distribution following the data distribution. If µ is uniform distribution over X , we abbreviate I p,µ ( f, g ) as I p ( f, g ) . For real-world applications, interpreting the model over the whole X is unnecessary, so µ is usually defined as a uniform distribution on the neighborhood of input x (under a certain metric), in which case we denote the distribution as N x .

3 Applying Our Framework to Removal-based Explanations

Now we focus on interpreting removal-based explanations [38, 18]. Removal-based explanations are post-hoc, which means they are generated based on a target network for a fixed input, by removing features from that input. There are three choices affecting the removal-based explanations: how features are removed, what behavior is analyzed after feature removal, and how to summarize the feature influence. For example, SHAP [38] considers all possible subsets of the features, and analyzes how holding out different features affects functional value, and finally summarizes the differences based on the Shapley value calculation.

Therefore, removal-based explanations can be represented as Boolean functions, as feature subset S ⊆ [ n ] can be represented as Boolean input: f ∈ {-1 , 1 } n → R , where -1 means the specific feature i ∈ S is removed, 1 means the specific feature is retained. We use -1 / 1 instead of 0 / 1 to represent the binary variables for ease of exposition using the Fourier basis.

3.1 Fourier Basis and Truthful Gap

Fourier analysis is a handy tool for analyzing Boolean functions. Due to the space limit, we defer the more comprehensive introduction on Fourier analysis for Boolean functions to Appendix A, and only present the necessary notions here.

Definition 3.1 (Fourier basis) . For any subset of variables S ⊆ [ n ] , we define the corresponding Fourier basis as χ S ( x ) ≜ Π i ∈ S x i ∈ {-1 , 1 } n →{-1 , 1 } .

The Fourier basis is also called polynomial basis in the literature. It is a complete orthonormal basis for Boolean functions, under the uniform distribution on {-1 , 1 } n . We remark that this uniform distribution is used for theoretical analysis and algorithm design, and is different from the measure µ for interpretation quality assessment in Definition 2.5.

Definition 3.2 (Fourier expansion) . Any Boolean function f ∈ {-1 , 1 } n → R can be expanded as

where ˆ f S = ⟨ f, χ S ⟩ is the Fourier coefficient on S .

Now we define the notion of C -Readable function.

Definition 3.3 ( C -Readable function) . Given a set of Fourier bases C , a function f is C -readable if it is supported on C . That is, for any χ S ̸∈ C , ⟨ f, χ S ⟩ = 0 . Denote the corresponding subspace as V C .

The Readable notion is parameterized with C , because it may differ case by case. If we set C to be all the single variable bases, only linear functions are readable; if we set C to be all the bases with the degree at most 2 , functions with pairwise interactions are also readable. Moreover, if we further add one higher order term to C , e.g., χ { x 1 ,x 2 ,x 3 ,x 4 } , it means we can also reason about the factor x 1 x 2 x 3 x 4 in the interpretation, which might be an important empirical factor that people can easily understand. Starting from the bases set C , we have the following formula for computing the truthful gap.

Lemma 3.4 (Truthful gap for Boolean functions) . Given a set of Fourier bases C , two functions f, g ∈ {-1 , 1 } n → R , the truthful gap of g to f for C is

With the previous definitions, it becomes clear that finding a truthful interpretation g is equivalent to accurately learning a Boolean function with respect to the readable bases set C . Intuitively, it means we want to find algorithms that can compute the coefficients for the bases in C . In other words, we want to find the importance of the bases like x 1 , x 2 x 5 , x 2 x 6 x 7 , etc.

3.2 Representative Algorithms

Applying the impossible trinity theorem to removal-based explanations, there are two notable algorithms on the extremes.

Shapley values: efficient but not consistent. Let N = { 1 , 2 , ..., n } represent a set of n players, and let v : 2 N → R be a characteristic function that assigns a real value to each coalition of players. The Shapley value of player i is given by:

where | S | is the number of players in coalition S , and the sum is taken over all possible coalitions S that do not include player i . One of the most important properties of Shapley values is efficiency , i.e. ∑ i ∈ N ϕ i ( v ) = v ( N ) -v ( ∅ ) (or denoting explanation function g ≜ ∑ i ∈ N ϕ i ( v ) + v ( ∅ ) ). As Shapley value based explanations do not focus on consistency, its explanation is efficient only for the original input, but not for other scenarios when certain features are removed.

Harmonica: consistent but not efficient. What can we do if we want to address consistency? As mentioned in Definition 2.4, we do not expect the algorithm to be efficient, but it can still be truthful with respect to a subspace V , which is naturally represented with the polynomial basis. Therefore, in order to learn truthful explanations for given subspace V , it is natural to consider LASSO regression over the coefficients on the polynomial basis. Harmonica (Algorithm 1) fulfills this requirement [27] and has superior complexity shown in Theorem 2.

Algorithm 1: Harmonica [27]

Given uniformly randomly sampled x 1 , · · · , x T , evaluate them on f : { f ( x 1 ) , ...., f ( x T ) } .

Solve the following regularized regression problem.

Output the polynomial g ( x ) = ∑ S,χ S ∈ C α S χ S ( x ) .

Theorem 2 (Complexity of Harmonica) . Given f ∈ {-1 , 1 } n → R , a ( ϵ/ 4 , s, C ) -bounded function, Algorithm 1 finds a function g with interpretation error at most ϵ in time O (( T log 1 ϵ + | C | /ϵ ) · | C | ) and sample complexity T = ˜ O ( s 2 /ϵ · log | C | ) .

We defer detailed descriptions and theoretical guarantees of these algorithms to Appendix B.1, C and D, discussion on the comparison of our algorithms with the existing algorithms to Appendix E.

3.3 Quantifying Inconsistency

All the Fourier coefficients together are called the Fourier spectrum of f . We can define the following distance between two functions based on their spectrums as the quantitative measure of the inconsistency between different explanations.

Definition 3.5 (Spectrum (Fourier) distance for Boolean functions) . Given two interpretations g ( · ) , h ( · ) ∈ {-1 , 1 } n → R , we define the spectrum p -distance between them as:

As there may be many candidates p for evaluating interpretation error and spectrum distance. In the following, we will investigate the most natural choice for p . As the Fourier expansion uniquely determined a Boolean function. We would like the distance between their expansion to have the same tendency as the interpretation error, thus the spectrum can encode enough information to reflect the accuracy of the interpretation. We shall first discuss the most 'strict' case, i.e. the L 0 norm.

̸

Denote the support of an Boolean function f as supp( f ) ≜ { x ∈ {-1 , 1 } n | f ( x ) = 0 } . The support of Fourier coefficients (Fourier spectrum), is defined by supp( ˆ f ) ≜ { S ⊆ [ n ] | ˆ f ( S ) = 0 } . Based on Proposition C.1, we have the following uncertainty principle for removal-based explanations : Theorem 3 (Uncertainty Principle for Removal-based Explanations) . Assume f, g ∈ {-1 , 1 } n → R and f = g . When evaluated under L 0 norm, the interpretation error and spectrum distance (inconsistency value) have the following uncertainty principle (can be seen as a quantitative version of Impossible Trinity Theorem 1):

̸

We can see that the L 0 norm is too strict for evaluating the differences between interpretations (inconsistency). Fortunately, the following proposition shows that the L 2 relaxed version is much better, or to say, natural, due to Parseval's identity.

Proposition 3.6 ( L 2 norm is natural) . When taking µ as a uniform distribution, for f, g ∈ {-1 , 1 } n → R , we have I 2 ( f, g ) = D 2 ( f, g ) .

Now we present a theorem on the trade-off between efficiency and consistency, quantified by total interpretation error and total inconsistency value (spectrum distance).

Theorem 4 (Trade-off between Efficiency and Consistency) . Given a function f : {-1 , 1 } n → R and a set of N interpretable functions { g 1 , g 2 , . . . , g N } defined on disjointed sets X i ( X = ⋃ X i ), each g i : {-1 , 1 } n → R , let ¯ g denote the average (spectrum) of g i across the space X = {-1 , 1 } n . Define (we use L 2 norm as default):

Then, we have the following inequality:

In addition, denote g † as the globally consistent interpretation g † defined on {-1 , 1 } n with minimal interpretation error (can be obtained by Harmonica), we have:

̸

Algorithm 2: Harmonica-anchor

Input: anchor number k , model f , a distance metric function d ( · , · ) : ( χ S , χ S ) → R which calculates distance between two anchors (e.g., L p norm or Hamming distance), sampling number T , regularization coefficient λ 1 .

Output: interpretation models g i with coefficients α i ∈ R | C | , i = 1 , 2 , 3 , ..., k .

1: Fix random bases b i , i = 1 , 2 , 3 , ..., k as anchors.

2: Randomly sample T bases b 1 , b 2 , ..., b T and accordingly, we calculate f ( x 1 ) , f ( x 2 ) , ..., f ( x T ) .

3: Assign each basis to the anchor with minimal distance and index using x

d i = min { argmin j d ( b i , b j ) } , i = 1 , 2 , 3 , ...T and j = 1 , 2 , 3 , ..., k .

4: for i = 1 , 2 , 3 , ..., k do

5: Solve the following regularized regression problem:

6: end for

This theorem illuminates the inherent trade-offs in designing interpretable models. Specifically, to reduce I total, one would typically need to increase D total unless f itself is close to an interpretable model ¯ g . Theorem 4 can also be seen as a quantitative version of the Impossible Trinity Theorem 1 for removal-based explanations.

Now we established the lower bound of I total + D total by I 2 ( f, g † ) (the same as D 2 ( f, g † ) ), showing the trade-off between efficiency and consistency, and also bridging the locally consistent interpretations g i and globally consistent interpretation g † .

4 Trade-off Between Efficiency and Consistency

Harmonica recovers functions across the entire function space. However, in our setting, if N x is small, it is sufficient to recover a small neighborhood of the function. This inspires us to apply Harmonica to a local space instead of the entire space. By doing so, we obtain more concentrated samples in the local neighborhood. Consequently, minimizing the interpretation error in this neighborhood becomes more manageable. We refer to Harmonica with samples in the local neighborhood as Harmonica-local.

If we relax the consistency requirement, meaning that users are willing to accept minor inconsistencies in interpretations when a slight modification is made to the input, we can achieve smaller interpretation error with Harmonica-anchor, as shown in Algorithm 2. Intuitively, Harmonic-anchor applies multiple Harmonica-local algorithms to different subspaces of the input. Specifically, for a given neighborhood region N x , we now randomly select k x bases b x,i , ( i = 1 , 2 , 3 , ..., k x ) as interpretation anchors instead of only one basis as Harmonica does. We calculate k x interpretation models g x,i on each anchor and denote the coefficient of g x,i as α x,i . For simplicity, we omit the subscript x in the following.

The number of bases k is highly related to consistency, so we could define the log value log k to evaluate the inconsistency of the interpretation models. The minimum inconsistency value is 0 , as in Harmonica ( k = 1 ), while for attribution methods, the inconsistency should be almost n , which is the number of input variables. The maximum inconsistency value depends on the neighborhood region N x , since the number of interpretation models should not exceed the number of points in N x . As a special case, the Harmonica algorithm achieves 0 inconsistency, making it the most consistent algorithm.

However, if one only considers the number of different interpretation algorithms, users may still encounter a high level of inconsistency empirically when the interpretations significantly differ from one another. Thus, another critical constraint to add is the spectrum distance constraint among different interpretations of different anchors. Based on Theorem 4, we introduce Harmonica-anchorconstrained in Algorithm 3 (in Appendix). In this algorithm, λ 2 serves as a penalty coefficient for the difference between g i , and we solve the problem using iterative gradient descent. Intuitively, a larger

λ 2 restricts the expressive power of the interpretation models g i , leading to a larger interpretation error.

5 Experiments

5.1 Analysis on Polynomial Functions

To investigate the performance of different interpretation methods, we manually examine the output of various algorithms including LIME [47], SHAP [38], Shapley Interaction Index [45], Shapley Taylor [61, 26], Faith-SHAP [62], Harmonica, and Low-degree (Appendix D) for lower-order polynomial functions.

We observe that all algorithms can accurately learn the coefficients of the first-order polynomial. For the second-order polynomial function, only Shapley Taylor, Faith-SHAP, Harmonica, and Low-degree can learn all the coefficients accurately. For the third-order polynomial function, only Faith-SHAP, Harmonica, and Low-degree succeed. Due to space constraints, we defer the details to Appendix F.

5.2 Experimental Setup

In the rest of this section, we conduct experiments to evaluate the interpretation error I p, N x ( f, g ) and truthful gap T V C ( f, g ) of Harmonica and other baseline algorithms on language and vision tasks quantitatively. In our experiments, we choose 2nd order and 3rd order Harmonica algorithms, which correspond to setting C to include all terms with order at most 2 and 3 .

The baseline algorithms chosen for comparison include LIME [47], Integrated Gradients [60], SHAP [38], Integrated Hessians [30], Shapley Taylor interaction index, and Faith-SHAP, where the first three are first-order algorithms, and the last three are second-order algorithms.

The two language tasks we select are the SST-2 [55] dataset for sentiment analysis and the IMDb [40] dataset for movie review classification. The vision task is the ImageNet [33] for image classification. To demonstrate the capability of our interpretation framework applied to vision tasks, we have generated two examples in Figure 2, compared with LIME, Integrated Gradients (IG), and SHAP. Note that all of these methods are applied to the same ground-truth image segmentation provided by the MS-COCO dataset [35]. For ablations on using the SLIC superpixels [2], please refer to Appendix H.

Cat

Harmonica-2nd

LIME

IG

SHAP

Horse

Harmonica-2nd

LIME

IG

SHAP

For the SST-2 dataset, we attempt to interpret a convolutional neural network (see details in Appendix G) trained with the Adam [32] optimizer for 10 epochs. The IMDb dataset contains long paragraphs, and each paragraph has multiple sentences. By default, we use periods, colons, and exclamations to separate sentences. For the ImageNet [33] dataset, we aim to provide class-specific interpretation, meaning that only the class with the maximum predicted probability is considered for each sample. We use the official pre-trained ResNet-101 [28] model from PyTorch.

1

2

4

8

16

Subspace radius

10

2

10

1

Interpretation error

22.8x

IMDb

1

2

4

8

Subspace radius

10

1

10 0

Interpretation error

31.8x

ImageNet

Harmonica (2nd order)

Harmonica (3rd order)

LIME

SHAP

Integrated Gradients

Integrated Hessians

Faith-Shap (2nd order)

Harmonica-anchor (k=3)

Harmonica-anchor (k=5)

Harmonica-anchor (k=7)

Harmonica-anchor (k=9)

5.3 Results on Interpretation Error

For a given input sentence x with length l , we define the induced neighborhood N x by introducing a masking operation on this sentence. The radius 0 ≤ r ≤ l is defined as the maximum number of masked words.

Figure 3 displays the L 2 interpretation error evaluated under different neighborhoods with a radius ranging from 1 to ∞ for all the considered datasets. Here, ∞ represents the maximum sentence length, which may vary for different data points. We also inspect L 1 and L 0 norms. Here, L 2 and L 1 are defined according to Eqn. (2) with p = 2 and p = 1 , respectively. And L 0 denotes ∫ X 1 | f ( x ) -g ( x ) | ≥ 0 . 1 dµ ( x ) . We can see that Harmonica consistently outperforms all the other baselines on all radii.

Note that for IMDb in Figure 3, we make a slight modification such that the masking operation is performed on sentences in one input paragraph instead (we also change the definition of radii accordingly). For ImageNet in Figure 3, the interpretation error is evaluated on 1000 random images, and the masking operation is performed on 16 superpixels in one input image instead (we also change the definition of radii accordingly). We can see that when the neighborhood's radius is greater than 1, Harmonica outperforms all the other baselines. Specifically, when evaluated on ImageNet dataset, Harmonica-anchor ( k = 9 ) achieves 31 . 8 times lower interpretation error I 2 , N 1 compare to Integrated Gradients [60]. Limited by space, the detailed numerical results and interpretation error under other norms are presented in Appendix I. In contrast, Harmonica-local achieves more accurate local consistency but high error outside the neighborhood. The relevant results are deferred in Appendix J due to limited space.

As mentioned in Section 4, Harmonica-anchor further reduces the interpretation error by learning each subspace separately. Figure 3 shows that the L 2 interpretation error of Harmonica-anchor achieves lower error than Harmonica on various datasets, and the error further declines as we increase the number of anchors. Full results of other norms and the results of Harmonica-anchor-constrained are shown in Appendix L.

5.4 Results on Truthful Gap

For convenience, we define the set of bases C d up to degree d as C d = { χ S | S ⊆ [ n ] , | S | ≤ d } . We evaluate the truthful gap on the set of bases C 3 , C 2 , and C 1 . For implementation details on

calculating the truthful gap, please refer to Appendix K. Table 1 shows the C 3 truthful gap evaluated on different datasets. We can see that Harmonica outperforms all the other baseline algorithms. Due to space limitations, C 2 and C 1 results are provided in Appendix K.

5.5 More Experimental Results

For more experimental results on different image segmentation methods, different neural network architectures, and different choices of baselines, please refer to Appendix H. For more results on the Low-degree algorithm, please refer to Appendix M.

6 Related Work

Interpretability is a critical topic in machine learning, and we refer the reader to [20, 37] for insightful general discussions. Below we discuss different types of interpretable models.

Removal-based explanations. Covert et al. [18] presents a unified framework for removal-based model explanation methods, connecting 26 existing techniques such as LIME [47], SHAP [38], Meaningful Perturbations [21], and permutation tests [10]. LIME [47] is a classical method that samples data points following a predefined sampling distribution and computes a function that empirically satisfies local fidelity.

The Shapley value [50, 72, 24] originates from cooperative game theory and is used to allocate the total value generated by a group of players among individual players. It is the unique kind of method that satisfies a few important properties, including efficiency, symmetry, dummy, additivity, etc. Shapley values have been extensively applied to machine learning model explanations [38, 39, 57, 59, 67, 78, 23, 77] and feature importance [17]. Recent research has focused on developing efficient approximation methods for Shapley values [38, 15, 5, 16, 31, 26, 68]. Many works have generalized Shapley values to higher-order feature interactions [45, 24, 61, 41, 63, 1, 62].

Gradient-based explanations. Gradient-based explanations are popular for deep learning models, such as CNNs. These methods include SmoothGrad, Integrated Gradient, GradCAM, DeepLift, LRP, etc. [53, 54, 52, 60, 74, 49, 6, 51, 13, 42, 52, 48]. Although these methods have seen extensive use in various areas, gradient-based methods are often time-consuming and can be insensitive to random model parameterization [3].

Model-specific interpretable models. Interpretable or transparent (white-box) models are inherently ante-hoc and model-specific. One primary goal of utilizing interpretable models is to achieve inherent model interpretability. Prominent approaches include Decision Trees ([66, 7, 75]), Decision Rules ([69, 58]), Decision Sets ([34, 70]), and Linear Models ([64, 65]). Moreover, another research direction attempts to use an interpretable model surrogate to approximate the original black-box models. [14] employs a two-layer additive risk model for interpreting credit risk assessments. [8] suggests an approach called model extraction that greedily learns a decision tree to approximate f . However, these methods primarily rely on heuristics and lack theoretical guarantees.

7 Conclusion

In this paper, we tackled the problem of generating consistent interpretations and introduced the impossible trinity theorem of interpretability, under a formal framework for understanding the interplay between interpretability, consistency, and efficiency. Since a consistent interpretation cannot be efficient, we relaxed efficiency to truthfulness, meaning the interpretation matches the target function in a specific subspace. This led to the problem of learning Boolean functions and the proposal of new algorithms based on the Fourier spectrum and a localized version of Harmonica. Our methods showed lower interpretation errors and improved consistency compared to the existing approaches.

While our work offers theoretical insights, many open questions and challenges remain in building more interpretable, consistent, and efficient models. We hope our work serves as a foundation for future research in the area of explainable AI.

Acknowledgments

Thanks to Jiaye Teng for the useful discussions. This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects 'Program on New Generation Artificial Intelligence' (Grant No. 2021AAA0150000).

References

Kjersti Aas, Martin Jullum, and Anders Løland. Explaining individual predictions when features are dependent: More accurate approximations to shapley values. Artificial Intelligence , 298: 103502, 2021.

Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine S¨ usstrunk. Slic superpixels. Technical report, EPFL, 2010.

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems , 31, 2018.

Zeyuan Allen-Zhu and Yang Yuan. Improved svrg for non-strongly-convex or sum-of-nonconvex objectives. In International Conference on Machine Learning , pp. 1080-1089. PMLR, 2016.

Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a polynomial time algorithm for shapley value approximation. In International Conference on Machine Learning , pp. 272-281. PMLR, 2019.

Sebastian Bach, Alexander Binder, Gr´ egoire Montavon, Frederick Klauschen, Klaus-Robert M¨ uller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one , 10(7):e0130140, 2015.

Randall Balestriero. Neural decision trees. ArXiv , abs/1702.07360, 2017.

Osbert Bastani, Carolyn Kim, and Hamsa Bastani. Interpretability via model extraction. ArXiv , abs/1706.09773, 2017.

William Beckner. Inequalities in fourier analysis. Annals of Mathematics , 102(1):159-182, 1975.

Leo Breiman. Random forests. Machine learning , 45:5-32, 2001.

S´ ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´ e J´ egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pp. 9650-9660, 2021.

Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 782-791, 2021.

Chaofan Chen, Kangcheng Lin, Cynthia Rudin, Yaron Shaposhnik, Sijia Wang, and Tong Wang. An interpretable model with globally consistent explanations for credit risk. ArXiv , abs/1811.12615, 2018.

Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. L-shapley and c-shapley: Efficient model interpretation for structured data. ArXiv , abs/1808.02610, 2019.

Ian Covert and Su-In Lee. Improving kernelshap: Practical shapley value estimation using linear regression. In International Conference on Artificial Intelligence and Statistics , pp. 3457-3465. PMLR, 2021.

Ian Covert, Scott M Lundberg, and Su-In Lee. Understanding global feature contributions with additive importance measures. Advances in Neural Information Processing Systems , 33: 17212-17223, 2020.

Ian Covert, Scott M Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. The Journal of Machine Learning Research , 22:209-1, 2021.

Amir Dembo, Thomas M Cover, and Joy A Thomas. Information theoretic inequalities. IEEE Transactions on Information theory , 37(6):1501-1518, 1991.

Derek Doran, Sarah Schulz, and Tarek R. Besold. What does explainable ai really mean? a new conceptualization of perspectives. ArXiv , abs/1710.00794, 2017.

Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision , pp. 3429-3437, 2017.

Eric Friedman and Herve Moulin. Three methods to share joint costs or surplus. Journal of Economic Theory , 87(2):275-312, 1999.

Christopher Frye, Colin Rowat, and Ilya Feige. Asymmetric shapley values: incorporating causal knowledge into model-agnostic explainability. Advances in Neural Information Processing Systems , 33:1229-1239, 2020.

Michel Grabisch and Marc Roubens. An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of Game Theory , 28:547-565, 1999.

Tom Gur and Omer Tamuz. Testing booleanity and the uncertainty principle. arXiv preprint arXiv:1204.0944 , 2012.

Mark Hamilton, Scott M. Lundberg, Lei Zhang, Stephanie Fu, and William T. Freeman. Axiomatic explanations for visual search, retrieval, and similarity learning. In International Conference on Learning Representations , 2022.

Elad Hazan, Adam R. Klivans, and Yang Yuan. Hyperparameter optimization: A spectral approach. ArXiv , abs/1706.00764, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770-778, 2016.

Isidore I Hirschman. A note on entropy. American journal of mathematics , 79(1):152-156, 1957.

Joseph D. Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. J. Mach. Learn. Res. , 22:104:1-104:54, 2021.

Neil Jethani, Mukund Sudarshan, Ian Connick Covert, Su-In Lee, and Rajesh Ranganath. Fastshap: Real-time shapley value estimation. In International Conference on Learning Representations , 2021.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Arxiv , abs/1412.6980, 2015.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60:84 - 90, 2012.

Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. Faithful and customizable explanations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pp. 131-138, 2019.

Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll´ ar. Microsoft coco: Common objects in context, 2015.

Nathan Linial, Yishay Mansour, and Noam Nisan. Constant depth circuits, fourier transform, and learnability. Journal of the ACM (JACM) , 40(3):607-620, 1993.

Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue , 16(3):31-57, 2018.

Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems , 2017.

Scott M. Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv:1802.03888 , 2018.

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies , pp. 142-150, 2011.

Aria Masoomi, Davin Hill, Zhonghui Xu, Craig P. Hersh, Edwin K. Silverman, Peter J. Castaldi, Stratis Ioannidis, and Jennifer G. Dy. Explanations of black-box models based on directional feature interactions. In International Conference on Learning Representations , 2022.

Gr´ egoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and KlausRobert M¨ uller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern recognition , 65:211-222, 2017.

Ryan O'Donnell. Analysis of boolean functions . Cambridge University Press, 2014.

Maxime Oquab, Timoth´ ee Darcet, Th´ eo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Guillermo Owen. Multilinear extensions of games. Management Science , 18(5-part-2):64-79, 1972.

Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing , pp. 1532-1543, 2014.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ' why should i trust you?' explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pp. 1135-1144, 2016.

Patrick Schwab and Walter Karlen. Cxplain: Causal explanations for model interpretation under uncertainty. Advances in Neural Information Processing Systems , 32, 2019.

Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision , 128:336-359, 2017.

Lloyd S. Shapley. Quota solutions of n-person games. Edited by Emil Artin and Marston Morse , pp. 343, 1953.

Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences. CoRR , abs/1605.01713, 2016.

Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning , pp. 3145-3153. PMLR, 2017.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR , abs/1312.6034, 2014.

Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Vi´ egas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. ArXiv , abs/1706.03825, 2017.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing , pp. 1631-1642, 2013.

Elias M Stein and Rami Shakarchi. Real analysis: measure theory, integration, and Hilbert spaces . Princeton University Press, 2009.

Erik ˇ Strumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems , 41(3):647-665, 2014.

Guolong Su, Dennis Wei, Kush R. Varshney, and Dmitry M. Malioutov. Interpretable two-level boolean rule learning for classification. ArXiv , abs/1511.07361, 2015.

Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In International Conference on Machine Learning , pp. 9269-9278. PMLR, 2020.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning , pp. 3319-3328. PMLR, 2017.

Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The shapley taylor interaction index. In International conference on machine learning , pp. 9259-9268. PMLR, 2020.

Che-Ping Tsai, Chih-Kuan Yeh, and Pradeep Ravikumar. Faith-shap: The faithful shapley shapley interaction index. arXiv preprint arXiv:2203.00870 , 2022.

Michael Tsang, Sirisha Rambhatla, and Yan Liu. How does this interaction affect me? interpretable attribution for feature interactions. Advances in Neural Information Processing Systems , 33:6147-6159, 2020.

Berk Ustun and Cynthia Rudin. Methods and models for interpretable linear classification. ArXiv , abs/1405.4047, 2014.

Berk Ustun, Stefano Trac` a, and Cynthia Rudin. Supersparse linear integer models for interpretable classification. ArXiv , abs/1306.6677, 2014.

Jialei Wang, Ryohei Fujimaki, and Yosuke Motohashi. Trading interpretability for accuracy: Oblique treed sparse additive models. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pp. 1245-1254, 2015.

Jiaxuan Wang, Jenna Wiens, and Scott Lundberg. Shapley flow: A graph-based approach to interpreting model predictions. In International Conference on Artificial Intelligence and Statistics , pp. 721-729. PMLR, 2021.

Rui Wang, Xiaoqian Wang, and David I. Inouye. Shapley explanation networks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021.

Tong Wang, Cynthia Rudin, Finale Doshi-Velez, Yimin Liu, Erica Klampfl, and Perry MacNeille. Or's of and's for interpretable classification, with application to context-aware recommender systems. ArXiv , abs/1504.07614, 2015.

Tong Wang, Cynthia Rudin, Finale Doshi-Velez, Yimin Liu, Erica Klampfl, and Perry MacNeille. A bayesian framework for learning rule sets for interpretable classification. The Journal of Machine Learning Research , 18(1):2357-2393, 2017.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 , 2022.

Robert J Weber. Probabilistic values for games, the shapley value. essays in honor of lloyd s. shapley (ae roth, ed.), 1988.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022.

Shawn Xu, Subhashini Venugopalan, and Mukund Sundararajan. Attribution in scale and space. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pp. 9677-9686. Computer Vision Foundation / IEEE, 2020. doi: 10.1109/CVPR42600.2020.00970.

Yongxin Yang, Irene Garcia Morillo, and Timothy M. Hospedales. Deep neural decision trees. ArXiv , abs/1806.06988, 2018.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601 , 2023.

Hao Yuan, Haiyang Yu, Jie Wang, Kang Li, and Shuiwang Ji. On explainability of graph neural networks via subgraph explorations. In International Conference on Machine Learning , pp. 12241-12252. PMLR, 2021.

Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivariate shapley interactions in dnns. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pp. 10877-10886, 2021.

Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. Cumulative reasoning with large language models. arXiv preprint arXiv:2308.04371 , 2023.

Pseudo Code for Harmonica-anchor-constrained

Algorithm 3: Harmonica-anchor-constrained

Input: anchor number k , model f , a distance metric function d ( · , · ) : ( χ S , χ S ) → R which calculates distance between two anchors (e.g., Hamming distance or L p norm), sampling number T , loss balance coefficient λ 1 , λ 2 , update epochs E , update rate η Output: interpretation models g i with coefficients α i , i = 1 , 2 , 3 , ..., k .

1: Fix random bases b i , i = 1 , 2 , 3 , ..., k as interpretation anchors.

2: Randomly initialize α i ∈ R | C | , i = 1 , 2 , 3 , ..., k .

3: Randomly sample T bases b 1 , b 2 , ..., b T and accordingly, we calculate f ( x 1 ) , f ( x 2 ) , ..., f ( x T ) .

4: Assign each basis to the anchor with minimal distance and index with d i = min { argmin j d ( b i , b x j ) } , i = 1 , 2 , 3 , ...T and j = 1 , 2 , 3 , ..., k .

5: for e = 1 , 2 , 3 , ..., E do

7: L r = 1 k ∑ k i =1 ∥ α i ∥ 1 .

8: L c = 1 k ∑ k i =1 ∥ α i -1 k ∑ k j =1 α j ∥ 2 .

9: Update α i with α i ← α i -η ∂ ( L f + λ 1 L r + λ 2 L c )

10: end for

A Fourier Analysis of Boolean Function

Fourier analysis of Boolean function is a fascinating field, and we refer the reader to [43] for a more comprehensive introduction. We define the inner product as follows.

Definition A.1 (Inner product) . Given two functions f, g ∈ {-1 , 1 } n → R , their inner product is:

In addition, we define the induced norm ∥ f ∥ 2 ≜ √ ⟨ f, f ⟩ , and more generally, for p > 0 ,

The inner product defines one kind of similarity between two functions and is invariant under different basis. Specifically, we have the following Theorem.

Definition A.2 (Plancherel's Theorem) . Given two functions f, g ∈ {-1 , 1 } n → R ,

When setting f = g , we get the Parseval's identity: E [ f 2 ] = ∑ S ˆ f 2 S .

A distribution over a discrete domain S is often represented as a non-negative function f : S → R + which is normalized in L 1 , i.e., ∑ x ∈ S f ( x ) = 1 .

Definition A.3 (Fourier p -norm) . For any Boolean function f ∈ {-1 , 1 } n → R , we define the Fourier (or spectral) p -norm of f as

Definition A.4 (Fourier p -distance) . For any Boolean function f, g ∈ {-1 , 1 } n → R , we define the Fourier (or spectral) p -distance between f and g as

∂α i .

Using Fourier p -norm, we could rephrase Parseval's identity as: ∥ f ∥ 2 = ˆ ∥ f ˆ ∥ 2 .

B Proofs

Proof of Theorem 1.

Proof. Pick f ∈ F \ G . If A is consistent with respect to f , let g = A ( f, x ) ∈ G for any x ∈ X . If for every x ∈ X , g ( x ) = f ( x ) , we know g = f ̸∈ G , this is a contradiction. Therefore, there exists x ∈ X such that g ( x ) = f ( x ) .

Proof of Lemma 3.4.

Proof. Denote the complement space as V ¯ C . We may expand f, g, v on both bases and get:

where the last equality holds because we can set the Fourier coefficients ˆ v S = ˆ f S for every S ∈ C , which further gives ⟨ f -v, χ S ⟩ = 0 .

Proof of Theorem 3.

Proof. Taking µ as the uniform distribution, then I 0 ( f, g ) = ∑ x ∈{-1 , 1 } n 1 f ( x ) = g ( x ) 2 n = | supp( f -g ) | 2 n . Note D 0 ( f, g ) = ∑ S ⊆ [ n ] 1 ˆ f ( S ) =ˆ g ( S ) = | supp( ˆ f -ˆ g ) | . The conclusion follows from Proposition C.1.

Proof of Proposition 3.6.

Proof of Theorem 4.

Proof. From Proposition 3.6 (Parseval's identity), we have (notice that our functions g i are defined on X = {-1 , 1 } n globally and then restricted to X i ⊆ X ).

Let us employ the triangle inequality for norms. Specifically, for each i = 1 , 2 , . . . , N , we apply the triangle inequality as follows:

Summing this inequality over all i yields:

̸

̸

̸

By the definitions of I total and D total, we arrive at:

This completes the proof.

B.1 Harmonica Algorithm

To present the theoretical guarantees of the Harmonica algorithm, we introduce the following definition, which is slightly different from its original version in [27].

Definition B.1 (Approximately sparse function) . We say a function f ∈ {-1 , 1 } n → R is ( ϵ, s, C ) -bounded, if E [( f -∑ χ S ∈ C ˆ f ( S ) χ S ) 2 ] ≤ ϵ and ∑ S | ˆ f ( S ) | ≤ s .

Here f is ( ϵ, s, C ) -bounded means it is almost readable and has bounded ℓ 1 norm. Our algorithm is slightly different from the original algorithm proposed by [27], but similar theoretical guarantees still hold, as stated below.

Proof of Theorem 2. Our proof is similar to the one in the original paper [27], with changes in the readable notion, which is now more flexible than being low order. First, recall the classical Chebyshev inequality.

Theorem B.2 (Multidimensional Chebyshev inequality) . Let X be an m dimensional random vector, with expected value µ = E [ X ] , and covariance matrix V = E [ ( X -µ )( X -µ ) T ] . If V is a positive definite matrix, for any real number δ > 0 :

Proof of Theorem 2. Let f be an ( ε/ 4 , s, C ) -bounded function written in the orthonormal basis as ∑ S ˆ f ( S ) χ S . We can equivalently write f as f = h + g , where h is supported on C that only includes coefficients of magnitude at least ε/ 4 s and the constant term of the polynomial expansion of f .

Since L 1 ( f ) = ∑ S ∣ ∣ ∣ ˆ f S ∣ ∣ ∣ ≤ s , we know h is 4 s 2 /ε +1 sparse. The function g is thus the sum of the remaining ˆ f ( S ) χ S terms not included in h . Denote the set of bases that appear in C but not in g as R , so we know the coefficient of f on the bases in R is at most ϵ/ 4 s .

Draw m (to be chosen later) random labeled examples {( z 1 , y 1 ) , . . . , ( z m , y m ) } and enumerate all N = | C | basis functions χ S ∈ C as { χ 1 , . . . , χ N } . Form matrix A such that A ij = χ j ( z i ) and consider the problem of recovering 4 s 2 /ε +1 sparse x given Ax + e = y where x is the vector of coefficients of h , the i th entry of y equals y i , and e i = g ( z i ) .

We will prove that with constant probability over the choice m random examples, ∥ e ∥ 2 ≤ √ εm . Applying Theorem 5 in [27] by setting η = √ ε and observing that σ 4 s 2 /ε +1 ( x ) 1 = 0 (see definition in the theorem), we will recover x ′ such that ∥ x -x ′ ∥ 2 2 ≤ c 2 2 ε for some constant c 2 . As such, for the function ˜ f = ∑ N i =1 x ′ i χ i we will have E [ ∥ h -˜ f ∥ 2 ] ≤ c 2 2 ε by Parseval's identity. Note, however, that we may rescale ε by a constant factor 1 / ( 2 c 2 2 ) to obtain error ε/ 2 and only incur an additional constant (multiplicative) factor in the sample complexity bound. By the definition of g , we have

where each ˆ f ( S ) for S ∈ R is of magnitude at most ε/ 4 s . By Fact 4 in [27] and Parseval's identity we have ∑ R ˆ f ( R ) 2 ≤ ε/ 4 . Since f is ( ε/ 4 , s, C ) -concentrated we have ∑ S,χ S ̸∈ C ˆ f ( S ) 2 ≤ ε/ 4 . Thus, ∥ g ∥ 2 is at most ε/ 2 . Therefore, by triangle inequality E [ ∥ f -˜ f ∥ 2 ] ≤ E [ ∥ h -˜ f ∥ 2 ] + E [ ∥ g ∥ 2 ] ≤ ε . It remains to bound ∥ e ∥ 2 . Note that since the examples are chosen independently, the entries

̸

e i = g ( z i ) are independent random variables. Since g is a linear combination of orthonormal monomials (not including the constant term), we have E z ∼ D [ g ( z )] = 0 . Here we can apply linearity of variance (the covariance of χ i and χ j is zero for all i = j ) and calculate the variance

With the same calculation as Eqn. (13), we know Var ( g ( z i )) is at most ε/ 2 . Now consider the covariance matrix V of the vector e which equals E [ ee ⊤ ] (recall every entry of e has mean 0). Then V is a diagonal matrix (covariance between two independent samples is zero), and every diagonal entry is at most ε/ 2 . Applying Theorem B.2 we have

Setting δ = √ 2 m , we conclude that P ( ∥ e ∥ 2 > √ εm ) ≤ 1 2 . Hence with probability at least 1 / 2 , we have that ∥ e ∥ 2 ≤ √ εm . From Theorem 5 in [27], we may choose m = ˜ O ( s 2 /ε · log n d ) . This completes the proof. Note that the probability 1 / 2 above can be boosted to any constant probability with a constant factor loss in sample complexity.

For the running time complexity, we refer to [4] for optimizing linear regression with ℓ 1 regularization. The running time is O (( T log 1 ϵ + L/ϵ ) · | C | ) , where L is the smoothness of each summand in the objective. Since each χ S takes value in {-1 , 1 } , the smoothness is bounded by the number of entries in each summand, which is | C | . Therefore, the running time is bounded by O (( T log 1 ϵ + | C | /ϵ ) · | C | ) .

C Uncertainty Principle for Boolean functions

In the field of modern physics, the state of a particle on a given domain S is represented by a complex function on that domain, and the probability of finding the particle in a specific position x on S is given by the square of the modulus of the function evaluated at x . To ensure that the function is normalized, in the case where S is continuous, the function must satisfy ∫ x ∈ R | f ( x ) | 2 dx = 1 .

The Fourier transform of the function, denoted by ˆ f , is also normalized in L 2 under a unitary transformation, and | ˆ f ( x ) | 2 represents the probability density function of the distribution of the particle's momentum.

The Heisenberg uncertainty principle states that the product of the variances of a particle's position and momentum is at least one, with an appropriate choice of units. This principle has physical significance and relates a function on R to its Fourier transform.

In 1957, [29] proposed an entropic form of the uncertainty principle, which was later proven nearly two decades later by Beckner [9]. The inequality states that H e [ f ] + H e [ ˆ f ] ≥ 1 -ln 2 , where H e [ f ] is the differential entropy of f , defined as -∫ x ∈ R | f ( x ) | 2 ln | f ( x ) | 2 dx .

When the domain is F n 2 (equivalently, {-1 , 1 } n ), a similar inequality [25] holds with a different constant. Let f : F n 2 → C have a Fourier transform ˆ f : F n 2 → C . Then,

Now we present the discrete uncertainty principle for Z n 2 as the following. It can be proved using Theorem 23 in Dembo, Cover, and Thomas [19].

All the Fourier coefficients together are called the Fourier spectrum of f . We can define the following distance between two functions based on their spectrums.

As there may be many candidates p for evaluating interpretation error and spectrum distance. In the following, we will investigate the most natural choice for p . As the Fourier expansion uniquely determined a Boolean function, we would like the distance between their expansion to have the same tendency as the interpretation error, thus the spectrum can encode enough information to reflect the accuracy of the interpretation. We shall first discuss the most 'strict' case, i.e. the L 0 norm.

̸

We shall introduce some notations for better understanding. Denote the support of an Boolean function f as supp( f ) ≜ { x ∈ {-1 , 1 } n | f ( x ) = 0 } . The support of Fourier coefficients (Fourier spectrum), is defined by supp( ˆ f ) ≜ { S ⊆ [ n ] | ˆ f ( S ) = 0 } . The uncertainty principle for Boolean functions is stated as follows.

̸

Proposition C.1 (Uncertainty Principle for Boolean functions [25, 43]) . For every nonzero function f : {-1 , 1 } n → R ,

D Low-degree Algorithm

The low-degree algorithm is based on the concentration inequality, and it estimates the coefficient of each axis individually.

Algorithm 4: Low-degree

Given uniformly randomly sampled x 1 , · · · , x T , evaluate them on f : { f ( x 1 ) , ...., f ( x T ) } .

For any χ S ∈ C , let ˆ g S = ∑ T i =1 f ( x i ) χ S ( x i ) T .

Output the polynomial g ( x ) = ∑ S,χ S ∈ C ˆ g S χ S ( x ) .

Theorem D.1 ([36]) . Given any ϵ, δ > 0 , assuming that function f is bounded by B , when T ≥ 2 B 2 ϵ 2 log 2 | C | δ , we have

Theorem D.1 was proved using the Hoeffding bound, and we included the proof here for completeness.

Proof. Since we are given T samples to estimate ˆ f ( S ) for every S , we can directly apply the Hoeffding bound (notice that the function is bounded by B ):

Notice that T ≥ 2 B 2 ϵ 2 log 2 | C | δ , we know the right hand size is bounded by δ | C | , so Theorem D.1 is proved.

Remarks . The theoretical guarantee of Harmonica assumes the target function f is approximately sparse in the Fourier space, which means most of the energy of the function (Fourier coefficients) is concentrated in the bases in C . This is not a strong assumption, because if f is not approximately sparse, it means f has energy in many different bases, or more specifically, the bases with higher orders. In other words, f has a large variance and is difficult to interpret. In this case, no existing algorithms will be able to give consistent and meaningful interpretations.

Likewise, although the Low-degree algorithm does not assume sparsity for f , it cannot learn all possible functions accurately as well. There are 2 n different bases, and if we want to learn the coefficients for all of them, the cumulative error for g is at the order of Ω(2 n ϵ ) , which is exponentially large. This is not surprising due to the no free lunch theorem in the generalization theory, as we do not expect to be able to learn 'any functions' without exponentially many samples.

E Discussion on the Existing Algorithms

In this section, we compare our approaches with the existing techniques from different perspectives of interpretation error (efficiency), truthfulness, and consistency.

LIME [47]. Given an input x , Lime samples the neighborhood points based on a sampling distribution Π x , and optimizes the following program:

where L is the loss function describing the distance between f and g on the sampled data points, G is the set of readable functions (e.g. the set of linear functions), Ω( · ) is a function that characterizes the complexity of g . In other words, LIME tries to minimize the fitting error and simultaneously minimizes the complexity of g (which is usually the sparsity of the linear function). By minimizing L , LIME also works towards minimizing the interpretation error, but their approach is purely heuristic, without any theoretical guarantees. Although their readable function set can easily generalize to the set with higher order terms, the sampling distribution Π x is not uniform, so it is difficult to incorporate the orthonormal basis into their framework. In other words, the model they compute is not truthful.

Attribution methods. As we discussed in the introduction, attribution methods mainly focus on individual inputs, instead of the neighboring points. Therefore, it is difficult for the attribution methods to achieve low inconsistency, especially for first-order methods like SHAP [38] and IG [60].

Consider SHAP [38] as a motivating example, illustrated in Figure 1 for the task of sentiment analysis of movie reviews. In this example, the interpretations of two slightly different sentences are inconsistent. This inconsistency arises not only because the weights of each word differ significantly, but also because after removing the word 'very' with a weight of 15 . 0% , the network's output only drops by 6 . 6% . In other words, the interpretation does not explain the network's behavior even in a small neighborhood of the input .

For higher-order attribution methods, consistency can potentially be improved due to their enhanced representation power. The classical Shapley interaction index has the problem of not precisely fitting the underlying function, as observed by [61], who proposed Shapley Taylor interaction index [61] with better empirical performance. Shapley Taylor interaction index satisfies the generalized efficiency axiom, which says for all f ∈ {-1 , 1 } n → R ,

We should remark that both the Shapley interaction index and Shapley Taylor interaction index were not originally designed for consistent interpretations, so they did not specify how to generalize the interpretation for the neighboring points. To this end, we make a global extension to the Shapley value based interpretation, that is, using Shapley interaction indices or Shapley Taylor interaction indices as the coefficients of corresponding terms of the polynomial surrogate function.

However, these higher-order Shapley value based methods all focus on the original Shapley value framework, so their interpretations are not truthful, i.e., not getting the exact coefficient of f even on the 'simple bases'. Moreover, as we will show in our experiments, higher-order methods still incur high interpretation errors compared with our methods.

When applying Shapley value techniques for visual search, [26] proposed an interesting and novel sampling + Lasso regression algorithm for efficiently computing higher order Shapley Taylor index in their experiments. However, their methods are based on a sampling probability distribution generated from permutation numbers, which is far from the uniform distribution. Additionally, their algorithm is based on the Shapley Taylor index, so their method is not truthful as well.

Discussion on universal consistency. When discussing consistency, there are two distinct settings: 'global consistency' and 'universal consistency.' This paper mainly focuses on global consistency (Definition 2.2), where the interpretation pertains only to input features and not to others. This scenario falls under the category of 'removal-based explanations,' and most existing interpretation methods belong to this category (26 of them were discussed in [18]). In contrast, universal consistency implies that the interpretation may depend on features different from the input features. An example of interpreting the sentence 'I am happy' could be, 'This sentence does not include [very], so this person is not extremely happy.' Universal consistency is more challenging than global consistency, and we hypothesize that more powerful machinery, such as foundation models, is needed.

Emergence of self-explainability in foundation models. In the groundbreaking work Sparks of AGI [11], the authors conducted comprehensive experiments on GPT-4's self-explainability. Chain-of-thought reasoning [73, 71, 76, 79] can be considered an interpretation provided by the model itself, although current LLMs are not always to be consistent or truthful. In the long run, however, this approach may represent the ultimate path to explainable AI. In the computer vision area, DINO [12, 44] emerges the ability of self-explanation provided by the attention map inside the model, which can be directly utilized as heatmap visualization or even applied to segmentation tasks.

Proposition E.1 (Omniscient LLM is the way) . An (infinitely) large (multi-modal) language model pre-trained using (infinitely) large corpus, with zero pre-training loss and generalization loss, which has been perfectly aligned to be truthful, is interpretable, efficient, and consistent.

Proof. To prove the proposition, let us examine each attribute (interpretable, efficient, and consistent) in the context of an (infinitely) large (multi-modal) language model (LLM):

Interpretable : By assumption, the LLM has zero pre-training loss and generalization loss, implying perfect knowledge representation. Furthermore, it is perfectly aligned to be truthful, satisfying the criteria for interpretability as per Definition 2.4.

Efficient : The efficiency condition g ( x ) = f ( x ) is trivially satisfied because f = g by the condition of self-explainability. Thus, the LLM is efficient as per Definition 2.3.

Consistent : Given that the LLM's responses are generated based on a perfect understanding and alignment, it will generate the same function A ( f, x ) for every x ∈ X , thereby being consistent as per Definition 2.2.

It's crucial to note that self-explainability does not violate the Impossible Trinity (Theorem 1). When f = g , the LLM serves as its own interpreter, inherently satisfying all three criteria. Alternatively, if f ⊆ g , where g has an even larger concept class than f , the interpretability criteria are also met, albeit this is not a practical scenario for standard interpretation algorithms.

Therefore, an (infinitely) large LLM, pre-trained with an (infinitely) large corpus and aligned perfectly to be truthful, satisfies all the conditions to be interpretable, efficient, and consistent.

F Test with Low order Polynomial Functions

F.1 First order polynomial function

To investigate the performance of different interpretation methods, let us take a closer look at a 1st order polynomial function:

For this simple function, we can manually compute the outcome of each algorithm, as illustrated in Table 2. If the algorithm's output is correct, i.e., equal to the output of f 1 , we write a check mark. Otherwise, we write down the actual output of the given interpretation algorithm.

As we can see, all methods are consistent and efficient for all cases. In fact, all variants of Shapley indices degraded to 1st order Shapley values.

Algorithms

(

-

1

,

-

1

,

-

1)

(

-

1

,

-

1

,

+1)

(

-

1

,

+1

,

-

1)

(

-

1

,

+1

,

+1)

(+1

,

-

1

,

-

1)

(+1

,

-

1

,

+1)

(+1

,

+1

,

-

1)

(+1

,

+1

,

+1)

Ground Truth

-

0

.

417

+0

.

083

-

1

.

083

-

0

.

583

+0

.

583

+1

.

083

-

0

.

083

+0

.

417

LIME

"

"

"

"

"

"

"

"

SHAP

"

"

"

"

"

"

"

"

Shapley Interaction Index (1st order)

"

"

"

"

"

"

"

"

"

"

"

"

"

"

"

"

Faith-Shap (1st order)

"

"

"

"

"

"

"

"

Low-degree (1st order)

"

"

"

"

"

"

"

"

Harmonica (1st order)

"

"

"

"

"

"

"

"

F.2 Second order polynomial function

In addition, let us take a closer look at a 2nd order polynomial function:

For this simple function, we can manually compute the outcome of each algorithm, as illustrated in Table 3. If the algorithm's output is correct, i.e., equal to the output of f 2 , we write a check mark. Otherwise, we write down the actual output of the given interpretation algorithm.

As we can see, 2nd order interpretation algorithms, including Shapley Taylor index, Faithful Shapley, Low-degree, and Harmonica are consistent and efficient for all cases. Other methods can only fit a few inputs, the 2nd order Shapley interaction index misses all the cases because it is not efficient, and LIME misses all the cases because f 2 is not a linear function.

Algorithms

(

-

1

,

-

1

,

-

1)

(

-

1

,

-

1

,

+1)

(

-

1

,

+1

,

-

1)

(

-

1

,

+1

,

+1)

(+1

,

-

1

,

-

1)

(+1

,

-

1

,

+1)

(+1

,

+1

,

-

1)

(+1

,

+1

,

+1)

Ground Truth

-

0

.

593

-

0

.

140

-

0

.

574

-

0

.

693

+0

.

474

+1

.

593

-

0

.

307

+0

.

240

LIME

-

0

.

417

+0

.

083

-

1

.

083

-

0

.

583

+0

.

583

+1

.

083

-

0

.

083

+0

.

417

SHAP

-

0

.

240

+0

.

283

-

1

.

250

-

0

.

726

+0

.

726

+1

.

250

-

0

.

283

"

Shapley Interaction Index (2nd order)

-

0

.

329

+0

.

171

-

0

.

995

-

0

.

781

+0

.

671

+1

.

505

-

0

.

395

+0

.

152

"

"

"

"

"

"

"

"

Faith-Shap (2nd order)

"

"

"

"

"

"

"

"

Low-degree (2nd order)

"

"

"

"

"

"

"

"

Harmonica (2nd order)

"

"

"

"

"

"

"

"

F.3 Third order polynomial function

Finally, we investigate the following 3rd order polynomial and present the result in Table 4.

As we can see, Faith-Shap, Low-degree, and Harmonica are consistent and efficient for all cases. Other methods can only fit a few inputs, the 3rd order Shapley interaction index misses all the cases because it is not efficient, and LIME misses all the cases because f 3 is not a linear function.

Algorithms

(

-

1

,

-

1

,

-

1)

(

-

1

,

-

1

,

+1)

(

-

1

,

+1

,

-

1)

(

-

1

,

+1

,

+1)

(+1

,

-

1

,

-

1)

(+1

,

-

1

,

+1)

(+1

,

+1

,

-

1)

(+1

,

+1

,

+1)

Ground Truth

-

0

.

718

-

0

.

015

+0

.

449

-

0

.

818

+0

.

599

+1

.

468

-

0

.

432

+0

.

365

LIME

-

0

.

417

+0

.

083

-

1

.

083

-

0

.

583

+0

.

583

+1

.

083

-

0

.

083

+0

.

417

SHAP

-

0

.

365

+0

.

242

-

1

.

292

-

0

.

685

+0

.

685

+1

.

292

-

0

.

242

"

Shapley Interaction Index (3rd order)

-

0

.

485

+0

.

224

-

0

.

943

-

0

.

896

+0

.

724

+1

.

390

-

0

.

510

+0

.

496

"

+0

.

194

-

0

.

606

-

0

.

970

+0

.

751

+1

.

625

-

0

.

642

"

Faith-Shap (3rd order)

"

"

"

"

"

"

"

"

Low-degree (3rd order)

"

"

"

"

"

"

"

"

Harmonica (3rd order)

"

"

"

"

"

"

"

"

Faith-SHAP has an intricate representation theorem, assigning coefficients to different terms under the M¨ obius transform. Since the basis induced by the M¨ obius transform is not orthonormal, it is unclear to us whether Faith-SHAP can theoretically compute the accurate coefficients for higherorder functions. However, the running time of Faith-SHAP has an exponential dependency on n , so empirically weighted sampling on subsets of features is needed [62]. This might be the main reason why our algorithms outperform Faith-SHAP in experiments on real-world datasets.

G More on Experiment Details

In this section, we will first describe the detailed experimental settings. For the two language tasks, i.e., SST-2 and IMDb, we use the same CNN neural network. The model has a test accuracy of 85 . 6% . For the readability of results, we treat sentences as units instead of words - masking several

words in a sentence may render the entire paragraph difficult to understand and even meaningless while masking a critical sentence has meaningful semantic effects. Therefore, the radius is defined as the maximum number of masked sentences for IMDb. The word embedding layer is pre-trained by GloVe [46] and the maximum word number is set to 25, 000. Besides the embedding layer, the network consists of several convolutional kernels with different kernel sizes (3, 4, and 5). After that, we use several fully connected layers, non-linear layers, and pooling layers to process the features. A Sigmoid function is attached to the tail of the network to ensure that the output can be seen as a probability distribution. Our networks are trained with an Adam [32] optimizer with a learning rate of 1e-2 for 5 epochs. For the vision task, we choose the official ResNet [28] architecture, which is available on PyTorch and we do not discuss the architecture details here. All the experiments are run on a server with 4 Nvidia 2080 Ti GPUs. A running time comparison can be found in Table 5. More information about the run-time Python environment and implementation details can be found in our code.

The accurate representation of the high non-linearity of deep neural networks (DNNs) can be quite complex, often leading to computationally expensive or practically infeasible interpretability methods. To strike a balance between computational efficiency and faithful representation, we employ higherorder polynomial approximations to capture the non-linear aspects of the target DNNs. Specifically, we leverage Fourier basis (or polynomial basis) to represent Boolean functions over feature subsets, thus providing a compact and computationally efficient way to approximate DNNs. This is particularly useful for removal-based explanations, which inherently deal with Boolean functions representing the presence or absence of features.

It is crucial to point out that our choice of using higher-order polynomials is not arbitrary but is guided by the principle of 'truthfulness.' We introduce a metric called the 'truthful gap,' mathematically defined as T V ( f, g ) (as per Equation 1), to quantify how well our polynomial approximations capture the information in a specific subspace V of the original function f . This notion of truthfulness serves as a rigorous measure to validate the efficacy of our higher-order polynomial approximations.

Our empirical results, particularly those outlined in Section 5, indicate that higher-order polynomials, specifically of orders 2 and 3, yield a favorable trade-off. They offer a substantial reduction in interpretation error and a low truthful gap compared to other techniques, substantiating the reasonableness and effectiveness of our approach.

Notice that our algorithm is a post-hoc model-agnostic interpretation algorithm, we only need to use the original neural network f to infer on given input as an oracle. This means that one can easily change the network architecture without any additional changes.

H More Experimental Results

In this section, we present ablation studies that examine the impact of various factors such as image segmentation format, neural network architecture, and choice of baselines. Our results demonstrate the robustness and efficacy of our approach in different experimental settings.

H.1 Image Segmentation Format

We employ the SLIC algorithm to generate segmentation superpixels on the ImageNet dataset. We initially divide each 224 × 224 image into 16 superpixels for a balance between human readability and computational efficiency. The results, displayed in Table 6 and Figure 4, validate the effectiveness of our approach. Further experiments with 10 superpixels are reported in Table 7, confirming the robustness of our method.

H.2 Neural Network Architectures

We also conduct experiments using diverse neural network architectures such as ResNet-18 and VGG16. Tables 8 and 9 respectively present the interpretation errors on ImageNet for these architectures.

H.3 Choice of Baselines

The choice of baseline is critical for removal-based interpretation methods. We investigate the interpretation errors when using the average pixel value and a blurred image as baselines. The results are shown in Tables 10 and 11.

I Detailed Results on Interpretation Error

In this section, we provide interpretation error results under other norms (Figure 6, 7, and 8) and the corresponding numerical results (Table 12, 13 and 14).

(a) Laptop

(b) Flatworm

x10-2

x10-1

x10-1

x10-1

laptop

Harmonica

LIME

IG

SHAP

2

5

2

2

1

1

1

0

0

0

-1

-1

-1

-2

-2

-2

x10-1

x10-1

x10-1

x10-1

flatworm

Harmonica

LIME

4

IG

SHAP

1

2

0

2

1

1

0

0

0

0

0

-2

-1

-1

-2

-1

-2

0

5

10

15

20

25

30

35

Number of inserted top important words

0.1

0.2

0.3

0.4

0.5

0.6

Probability error

Harmonica-1st

LIME

SHAP

IG

(a) Insertion

0

5

10

15

20

25

30

35

Number of deleted bottom important words

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Probability error

Harmonica-1st

LIME

SHAP

IG

(b) Deletion

J Detailed Results on Harmonica-local

J.1 Additional Experiments

We further explore Harmonica's performance when limited to a local space instead of the whole space. It is worth noting that L r is a regularization loss for Harmonica, ensuring the sparseness of each interpretation model g i . Additionally, L c is a regularization loss penalizing the difference between interpretation models. The value of balance coefficients depends on the application scenario and can be determined by the end user.

1

2

4

8

16

32

Subspace radius

10

1

4×10 2

6×10 2

2×10 1

3×10 1

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

LIME

SHAP

Integrated Gradients

Integrated Hessians

Faith-Shap (2nd order)

(a)

L

2

norm

1

2

4

8

16

32

Subspace radius

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

LIME

SHAP

Integrated Gradients

Integrated Hessians

Faith-Shap (2nd order)

(b)

L

1

norm

1

2

4

8

16

32

Subspace radius

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

LIME

SHAP

Integrated Gradients

Integrated Hessians

Faith-Shap (2nd order)

(c)

L

0

norm

1

2

4

8

16

32

Subspace radius

10

1

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

LIME

SHAP

Integrated Gradients

Integrated Hessians

Faith-Shap (2nd order)

(a)

L

2

norm

1

2

4

8

16

32

Subspace radius

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

LIME

SHAP

Integrated Gradients

Integrated Hessians

Faith-Shap (2nd order)

(b)

L

1

norm

1

2

4

8

16

32

Subspace radius

10

2

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

LIME

SHAP

Integrated Gradients

Integrated Hessians

Faith-Shap (2nd order)

(c)

L

0

norm

1

2

4

8

Subspace radius

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

LIME

SHAP

Integrated Gradients

Integrated Hessians

Faith-Shap (2nd order)

(a)

L

2

norm

1

2

4

8

Subspace radius

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

LIME

SHAP

Integrated Gradients

Integrated Hessians

Faith-Shap (2nd order)

(b)

L

1

norm

1

2

4

8

Subspace radius

10 0

3×10 1

4×10 1

6×10 1

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

LIME

SHAP

Integrated Gradients

Integrated Hessians

Faith-Shap (2nd order)

(c)

L

0

norm

1

2

4

8

16

32

Subspace radius

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-local (2nd order)

Harmonica-local (3rd order)

(a)

L

2

norm

1

2

4

8

16

32

Subspace radius

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-local (2nd order)

Harmonica-local (3rd order)

(b)

L

1

norm

1

2

4

8

16

32

Subspace radius

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-local (2nd order)

Harmonica-local (3rd order)

(c)

L

0

norm

1

2

4

8

16

32

Subspace radius

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-local (2nd order)

Harmonica-local (3rd order)

(a)

L

2

norm

1

2

4

8

16

32

Subspace radius

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-local (2nd order)

Harmonica-local (3rd order)

(b)

L

1

norm

1

2

4

8

16

32

Subspace radius

10

2

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-local (2nd order)

Harmonica-local (3rd order)

(c)

L

0

norm

1

2

4

8

Subspace radius

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-local (2nd order)

Harmonica-local (3rd order)

(a)

L

2

norm

1

2

4

8

Subspace radius

10

1

10 0

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-local (2nd order)

Harmonica-local (3rd order)

(b)

L

1

norm

1

2

4

8

Subspace radius

10 0

2×10 1

3×10 1

4×10 1

6×10 1

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-local (2nd order)

Harmonica-local (3rd order)

(c)

L

0

norm

K Detailed Results on Truthful Gap

Estimating truthful gap For convenience, we define the set of bases C d up to degree d as C d = χ S | S ⊆ [ n ] , | S | ≤ d . We evaluate the truthful gap on the set of bases C 3 , C 2 , and C 1 . By definition in Eqn. (3), we have T V C ( f, g ) = ∑ χ S ∈ C ⟨ f -g, χ S ⟩ 2 =

Then we perform a sampling-based estimation of the truthful gap. Worth mentioning that since the size of set C d satisfies | C d | = ∑ d i =0 ( n i ) and the max number of words, sentences, or superpixels

n ∗ ≤ 50 , ∑ χ S ∈ C χ S ( x ) , as the summation function of orthonormal basis, is easy to compute on every sample x ∈ {-1 , 1 } n (for n ∗ very large, we will perform another sampling step on this function).

Table 1 shows the C 3 truthful gap evaluated on different datasets. We can see that Harmonica outperforms all the other baseline algorithms. Figure 12 shows the truthful gap evaluated on the SST-2 dataset. We can see that Harmonica achieves the best performance for C 2 and C 3 . For the simple linear case C 1 , Harmonica is almost as good as LIME. Figure 13 shows the truthful gap evaluated on the IMDb dataset under the same settings as the SST-2 dataset. Figure 14 shows the truthful gap evaluated on the ImageNet dataset. We can see that Harmonica outperforms all the other baselines consistently.

Harmonica

(2nd order)

Harmonica

(3rd order)

LIME

SHAP

Integrated

Gradients

Integrated

Hessians

(2nd order)

Faith-Shap

(2nd order)

0

1

2

3

4

5

6

7

Truthful gap

1e

5

(a) bases

C

3

Harmonica

(2nd order)

Harmonica

(3rd order)

LIME

SHAP

Integrated

Gradients

Integrated

Hessians

(2nd order)

Faith-Shap

(2nd order)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Truthful gap

1e

4

(b) bases

C

2

Harmonica

(2nd order)

Harmonica

(3rd order)

LIME

SHAP

Integrated

Gradients

Integrated

Hessians

(2nd order)

Faith-Shap

(2nd order)

0

1

2

3

4

5

6

7

Truthful gap

1e

4

(c) bases

C

1

Harmonica

(2nd order)

Harmonica

(3rd order)

LIME

SHAP

Integrated

Gradients

Integrated

Hessians

(2nd order)

Faith-Shap

(2nd order)

0

2

4

6

8

10

12

14

16

Truthful gap

1e

4

(a) bases

C

3

Harmonica

(2nd order)

Harmonica

(3rd order)

LIME

SHAP

Integrated

Gradients

Integrated

Hessians

(2nd order)

Faith-Shap

(2nd order)

0

1

2

3

4

5

6

7

8

Truthful gap

1e

4

(b) bases

C

2

Harmonica

(2nd order)

Harmonica

(3rd order)

LIME

SHAP

Integrated

Gradients

Integrated

Hessians

(2nd order)

Faith-Shap

(2nd order)

0

1

2

3

4

Truthful gap

1e

3

(c) bases

C

1

Harmonica

(2nd order)

Harmonica

(3rd order)

LIME

SHAP

Integrated

Gradients

Integrated

Hessians

(2nd order)

Faith-Shap

(2nd order)

10

4

10

3

10

2

Truthful gap

(a) bases

C

3

Harmonica

(2nd order)

Harmonica

(3rd order)

LIME

SHAP

Integrated

Gradients

Integrated

Hessians

(2nd order)

Faith-Shap

(2nd order)

10

4

10

3

10

2

Truthful gap

(b) bases

C

2

Harmonica

(2nd order)

Harmonica

(3rd order)

LIME

SHAP

Integrated

Gradients

Integrated

Hessians

(2nd order)

Faith-Shap

(2nd order)

10

4

10

3

10

2

10

1

Truthful gap

(c) bases

C

1

L Detailed Results on Harmonica-anchor

L.1 Harmonica-anchor

Figure 15, 16, and 17 show the interpretation error results evaluated on three datasets. All Harmonicaanchor algorithms further reduce the interpretation error compared to Harmonica. As we increase the number of anchors, the interpretation error slightly reduces, which is consistent with our intuition. Table 15, 16, and 17 show the numerical results.

1

2

4

8

16

32

Subspace radius

10

1

2×10 2

3×10 2

4×10 2

6×10 2

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-anchor (k=3)

Harmonica-anchor (k=5)

Harmonica-anchor (k=7)

Harmonica-anchor (k=9)

(a)

L

2

norm

1

2

4

8

16

32

Subspace radius

2×10 2

3×10 2

4×10 2

6×10 2

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-anchor (k=3)

Harmonica-anchor (k=5)

Harmonica-anchor (k=7)

Harmonica-anchor (k=9)

(b)

L

1

norm

1

2

4

8

16

32

Subspace radius

10

2

10

1

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-anchor (k=3)

Harmonica-anchor (k=5)

Harmonica-anchor (k=7)

Harmonica-anchor (k=9)

(c)

L

0

norm

1

2

4

8

16

32

Subspace radius

10

2

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-anchor (k=3)

Harmonica-anchor (k=5)

Harmonica-anchor (k=7)

Harmonica-anchor (k=9)

(a)

L

2

norm

1

2

4

8

16

32

Subspace radius

10

2

4×10 3

6×10 3

2×10 2

3×10 2

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-anchor (k=3)

Harmonica-anchor (k=5)

Harmonica-anchor (k=7)

Harmonica-anchor (k=9)

(b)

L

1

norm

1

2

4

8

16

32

Subspace radius

10

3

10

2

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-anchor (k=3)

Harmonica-anchor (k=5)

Harmonica-anchor (k=7)

Harmonica-anchor (k=9)

(c)

L

0

norm

1

2

4

8

Subspace radius

10

1

6×10 2

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-anchor (k=3)

Harmonica-anchor (k=5)

Harmonica-anchor (k=7)

Harmonica-anchor (k=9)

(a)

L

2

norm

1

2

4

8

Subspace radius

10

1

4×10 2

6×10 2

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-anchor (k=3)

Harmonica-anchor (k=5)

Harmonica-anchor (k=7)

Harmonica-anchor (k=9)

(b)

L

1

norm

1

2

4

8

Subspace radius

10

1

2×10 1

3×10 1

4×10 1

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-anchor (k=3)

Harmonica-anchor (k=5)

Harmonica-anchor (k=7)

Harmonica-anchor (k=9)

(c)

L

0

norm

L.2 Harmonica-anchor-constrained

Here we provide the interpretation error of Harmonica-anchor-constrained evaluated on SST-2 dataset in Figure 18. As we increase the constraint coefficient λ 2 , the interpretation error increases, but still smaller than that of Harmonica.

Harmonica-anchor (

)

Harmonica-anchor (

)

1

2

4

8

16

32

Subspace radius

10

1

3×10 2

4×10 2

6×10 2

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-anchor (

2=0.0)

Harmonica-anchor (

2=1.0)

Harmonica-anchor (

2=5.0)

(a)

L

2

norm

1

2

4

8

16

32

Subspace radius

2×10 2

3×10 2

4×10 2

6×10 2

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-anchor (

2=0.0)

Harmonica-anchor (

2=1.0)

Harmonica-anchor (

2=5.0)

(b)

L

1

norm

1

2

4

8

16

32

Subspace radius

10

2

10

1

Interpretation error

Harmonica (2nd order)

Harmonica (3rd order)

Harmonica-anchor (

2=0.0)

Harmonica-anchor (

2=1.0)

Harmonica-anchor (

2=5.0)

(c)

L

0

norm

M Discussion on the Low-degree algorithm

We also investigate the sample complexity of Harmonica and Low-degree algorithms, which demonstrates that Harmonica achieves better performance with the same sample size.

1

2

4

8

16

32

Subspace radius

10

2

10

1

Interpretation error

Harmonica (2 order, 2000 samples)

Low-degree (2 order, 2000 samples)

Low-degree (2 order, 4000 samples)

Low-degree (2 order, 6000 samples)

Low-degree (2 order, 8000 samples)

Low-degree (2 order, 10000 samples)

(a)

L

2

norm

1

2

4

8

16

32

Subspace radius

10

1

6×10 2

2×10 1

3×10 1

4×10 1

Interpretation error

Harmonica (2 order, 2000 samples)

Low-degree (2 order, 2000 samples)

Low-degree (2 order, 4000 samples)

Low-degree (2 order, 6000 samples)

Low-degree (2 order, 8000 samples)

Low-degree (2 order, 10000 samples)

(b)

L

1

norm

1

2

4

8

16

32

Subspace radius

10 0

Interpretation error

Harmonica (2 order, 2000 samples)

Low-degree (2 order, 2000 samples)

Low-degree (2 order, 4000 samples)

Low-degree (2 order, 6000 samples)

Low-degree (2 order, 8000 samples)

Low-degree (2 order, 10000 samples)

(c)

L

0

norm

From Theorem 2 and Theorem D.1, we know that the sample complexity of the Harmonica algorithm ( ˜ O ( 1 ϵ ) ) is much more efficient than the Low-degree algorithm ( ˜ O ( 1 ϵ 2 ) ). Figure 19 shows that when evaluating the interpretation error on SST-2 dataset, with the same sample size, the Harmonica

algorithm outperforms the Low-degree algorithm by a large margin. We further increase the sample size for the Low-degree algorithm and see that its interpretation error gradually approaches that of Harmonica. However, even with 5x sample size, the Low-degree algorithm still gives a larger interpretation error compared with Harmonica. However, since exactly computing the Low-degree algorithm is extremely time-consuming, here we present the results using five times the sample size to show the calculation difficulty of the Low-degree algorithm.

Peer reviews:
comments: SUMMARY:
This paper studies the consistency and efficiency of removal-based explanation methods. By establishing the impossible trinity theorem, the authors show that consistency and efficiency cannot be achieved simutaneously. They further propose the interpretation error as a metric to gauge the trade-off between consistency and efficiency.  Then, two algorithms based on standard polynomial basis are developed to minimize the interpretation error for more consistent explanations, when relaxing the efficiency to truthfulness.

STRENGTHS:
1. Consistency and efficiency are important properties for evaluating explanation methods.
2. The finding of trade-off between efficiency and consistency is interesting and leads to some insights.

WEAKNESSES:
1. My major concern is that this paper is quite hard to follow and the presentation needs significant improvement. The authors present a lot of definitions and claims but not sufficient explanations and motivations for them.

2. The definition for consistency and efficiency is not rigorous and seems to be different from that in previous literature. The authors should justify the correctness of definitions and give more explanations here.

QUESTIONS:
1. At line 136, if the consistency should be conditioned on a local input region which shares the same ground truth label?

2. At line 139, how to match the definition of efficiency with that at line 194?

3. How to approximately calculate the equation 2 and 3 in experiments?

4. Apart from the proposed metrics, truthful gap and interpretation error, are there any existing metrics in current literature for consistency and efficiency? The existing ones should be compared with the proposed ones.

LIMITATIONS:
Please see the weakness and questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of tradeoffs between interpretability, consistency, and efficiency of AI models. The authors present “the Impossible Trinity Theorem”, which posits that interpretability, efficiency and consistency cannot hold simultaneously. To quantify the tradeoff, the authors define the utilization of interpretation error as a metric to gauge inconsistencies and inefficiencies of AI models in order to minimize an interpretation error. With such definitions of terms and metrics, the authors achieve a reduction in interpretation error, up to 31.8 times lower when compared to alternative techniques.\n\nSTRENGTHS:\nThe strength of the paper lies in the effort to formulate a mathematical framework including efficiency, consistency, and interpretability of AI models. The framework separates linear and non-linear parts of input-output functions represented by trained AI models by stating “only linear functions are readable” and therefore interpretable.\n\nWEAKNESSES:\nThe weaknesses of the paper lie in \n(1)\tforcing reviewers to read Appendices because the main text does not contain sufficient results supporting the claims. This is unfair to other submissions.\n(2)\tobfuscating many simple concepts with too many theorems and definitions to formulate relationships.\n(3)\tmissing discussion why 2nd order Harmonica results are worse than Shapley in Table 1 which implies that the 3rd order Harmonica is needed. BTW, Standard deviation would help to understand statistical significance of the values when comparing the results.\n\nQUESTIONS:\nIn reference to “Note that all of these methods are applied to the same image segmentation using the SLIC superpixels [61]. “ it is not clear why the simple linear iterative clustering algorithm applied to 224 x 224 images split into 16 superpixels is used as ground truth for ImageNet rather ground truth segmentation obtained by manual segmentation.\n\nLIMITATIONS:\nOverall, the framework has introduced too much complexity for a problem that can be described very clearly. For example, Theorem 1(Impossible Trinity for Interpretation) could just state that the input-output relationship in a trained AI model is either accurate and non-linear or inaccurate and linear.\n\nThe focus should be rather on Fourier spectrum and the order of Harmonica algorithm (i.e., LASSO regression over the coefficients on the polynomial basis) needed to meet a given epsilon error.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the interplay between two properties of removal based explanations: efficiency and consistency. It established that these two properties cannot coexist when the set of predictive models (requiring explanation), and the set of interpreting models (doing the explaining) are disjoint. It proposes a new property: “truthfulness”, which is a relaxation of efficiency, and measures the information-capture within a functional sub-space (a chosen subset of the interpretable models). It proposes several algorithms for learning truthful interpreting models, grounded in methods from Boolean functional analysis, which allow for different trade-offs between consistency and efficiency. It compares these algorithms to existing methods by looking at the interpretation error on a few practical NLP and Vision tasks.\n\nSTRENGTHS:\nThe authors demonstrate a solid understanding of existing removal-based (and other) explainability methods. The paper surveys a large number of these related works.\nThe paper explains a tension in post-hoc explanations: efficiency vs. consistency in a concise and easy to understand way. \nThe front half of the paper (sections 1 to 4.1), where most of the concepts are presented, is clearly written and easy to understand.\nThe paper draws insightful connections between producing interpreting models and work in Boolean functional analysis.\nBetween the theoretical discussion, the methods introduced, and the subsequent experiments, the paper makes numerous useful contributions.\n\nWEAKNESSES:\nThe paper defines consistency for explanations (Definition 3.2). It is unclear whether the authors are proposing this particular definition, or if it has been previously defined in this way. This should be clarified. Furthermore, the term consistency has been used differently in the explainability and ML literature previously, albeit more casually. It would be helpful to contrast this definition to other ways the term has been used, and to relate/contrast it to other previously defined properties of explanations, such as fidelity.\n\nThe motivation for consistency could be improved. The extent to which consistency is desirable, and the size of the neighborhood in which it would be desired, both seem to be highly dependent on how the explanation is going to be used. As it stands, there is a single (simple) motivating example in the main text. Since the primary reason one would use the methods proposed in the paper is to improve consistency, it seems reasonable to further motivate this property.\n\nI question the choice of the words “truthful” and “truthfulness” for Definition 3.4 .When T_v = 0, g is a sort of best approximation to f within a space of interpretable models. The property is interesting, but it does not intuitively align with “truthfulness” for me. With the number of overloaded terms that already exist in the explainability space (many of which invoke their own “layman” concepts), I caution against introducing another, especially a term as loaded as “truth”. I strongly encourage the use of a more verbose and precise term. \n\nIt seems unfair to call Theorem 1 the “Impossible Trinity for Interpretability” when the set of interpretable models is separated from the set of predictive models in order to get the result. Interpretable models are a very important class of predictive models. If the authors wish to name the result, I would suggest something along the lines of “Impossible Trinity for Post-Hoc Explanation”.\n\nThere is something of a conceptual gap in the flow of the paper. Section 3, and all definitions consider an arbitrary input space X. While Section 4 onwards deals with Boolean functions. What is missing is any real discussion of the process of binarizing real-valued (or other) inputs and how that affects or relates to consistency. From what I understand, ensuring the consistency of an explanation in a neighborhood of removed features (the binarized input space) is different from ensuring the consistency of that explanation in some ball around the original real-valued input. This is simply because moving to another region of the original input space (for example by changing a word/token in an input sentence or by perturbing the input image) will require a different binarization, because what it means to include/exclude one (or more) of the features is different. This seems like it should be better explored and discussed.\n\nThe paper discusses “amounts” of inconsistency, but only defines what it means to be consistent, not how to measure deviation from consistency. This seems like a considerable omission, especially since Algorithm 3 aims to reduce inconsistency. \n\nThe second half of the paper (section 4.2 onwards) struggles with clarity. \n\nThe paper should include concrete examples of output explanations. Especially for Harmonica-anchor and the constrained variant. As it stands, I do not understand how one is meant to take a set of coefficients for some polynomial bases and produce a heatmap over an image or textual input. I can guess, but this should be made clear. \n\nThe algorithm boxes include some notation that is not explained elsewhere in the main text. For instance, C is introduced in the appendix. Algorithm 1 should define input and outputs.\n\nFor clarity I would also consider using different symbols when referring to Boolean models vs. models having arbitrary inputs, instead of f(x) for both. \n\nFinally, the paper relies heavily on the Appendix, and in that sense the main text does not truly seem self-contained. For instance, in the opening paragraph of the introduction the authors ask the reader to refer to a figure in the Appendix (Figure 3). I suggest shuffling some content around. For instance, the definition of the Shapley value (which is well-known in the field) and even Figure 1 (which isn’t really discussed), would both seem like more appropriate Appendix content than Figure 3, which helps to motivate consistency. The current introduction could also be compressed somewhat. \n\nSome typos:\nLine 72: “the algorithm only retain”\nLine 131 “We call A model g is interpretable”\n\nQUESTIONS:\nThe paragraph starting on line 218 quantifies inconsistency, but does not explicitly explain how. What is an “amount” of inconsistency? It seems like what is being reported is the number of different explanation functions required in the neighbourhood.\n\nIs Figure 2 correctly labeled? Should the subspace radius for Imagenet not be 1-16 and 1-\\inf be for IMDB the \n\nShould the C in Harmonica-anchor be indexed by i? \n\nIn Table 1 and Section 6.4, it is not clear to me why we would expect (or even want) any of the other methods (other than Harm. 3rd) to have a small “truthful gap” in C^3. The other methods do not aim to explain with 3rd order polynomials. Why should we assess them in this way?\n\nIt would seem to me that Harmonica’s “truthful gap” should be a function of T and \\lambda, is this explored somewhere?\n\nLIMITATIONS:\nThe paper would benefit from an explicit discussion of its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyses the trade-off between efficiency and consistency in the context of removal-based post-hoc explanations. The authors demonstrate that having an interpretability method that is both efficient and consistent is impossible. Further, on the problem of finding consistent explanations, the authors propose to relax the efficiency desiderata and instead propose a new desiderata called truthfulness (efficiency within a subspace). The authors argue that this maps to the problem of learning Boolean functions that can be tackled by using Fourier analysis.\n\nSTRENGTHS:\n1. The paper is well-written. The motivation, definitions and key contributions are clearly explained.\n\n2. The core idea of leveraging Fourier analysis in the context of removal-based explanations is interesting.\n\n3. The experiments and bibliography sections are extensive.\n\nWEAKNESSES:\n1. I do not understand why is consistency of explanations a more important desiderata than efficiency. Instead, one may argue that efficiency is critical since it acts as a proxy to the problem of lack of ground-truth. Specifically, one of the main advantages of using SHAP is that supports some mathematically properties which come handy in the absence of ground-truth. Since, due to impossibility, we can't achieve both, I would be keen to hear when is consistency more important than efficiency. Further, here we talk about consistency in Boolean sense which I guess is different in a non-Boolean sense (similar inputs should have similar explanations) as there this desiderata is referred to as robustness.\n\n2. Using Fourier analysis for Boolean functions is a well-explored topic. Hence, although interesting, I see limited novelty in applying this to post-hoc removal-based methods.\n\n3. Some experimental details seem to be missing. For example, all removal based methods use a baseline and the explanations are quite sensitive to the choice of the baseline [1]. I wonder what is baseline for SHAP/IG/LIME etc. and if the authors analysed the effect of choosing a different baseline.\n\n[1]  https://distill.pub/2020/attribution-baselines/\n\nQUESTIONS:\n1.  In experiments mentioned in section 6.1, how did the authors choose the feature values to compute ground-truth?\n\n2. What version of the SHAP algorithm was used? Further, as LIME is very sensitive to the chosen hyperparameters[2], I wonder what are the hyperparameters for the LIME algorithm.\n\n\n[2]  https://arxiv.org/pdf/2005.07788.pdf\n\nLIMITATIONS:\nThe authors do not mention any  limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes using interpretation error as a metric to gauge inconsistencies and inefficiencies. The authors proposed three algorithms founded on the standard polynomial basis to minimize the interpretation errors. Authors perform experiments using both image and text classification to demonstrate the superior performance in terms of lowest interpretation error to the baseline methods.\n\nSTRENGTHS:\nA major issue for the removal-based explanation is that the change of input distribution with the removal of inputs. This makes the performance evaluation problematic. The trade-off between efficiency and consistency for removal-based explanations is well known. The novelty of this paper is to propose using a single metric, i.e., interpretation error, for evaluating the quality of model interpretation, followed by two algorithms for finding the best interpretation models assuming the interpretable model is polynomial. Authors also conducted comprehensive experiments to evaluate both interpretation error and truthful gap of Harmonica and other baseline algorithms.\n\nWEAKNESSES:\nTo establish your approach is superior, comparison in terms of prediction accuracy and time complexity across multiple DNN architectures would be helpful. Indeed, the trade-off between prediction performance and interpretability can also be important. It would be nice to see your approach achieves lower interpretation errors without compromising prediction performance.\n\nQUESTIONS:\nBesides efficiency and consistency, there are other factors that would influence the evaluation of model interpretability. Can you also show the prediction accuracy together with interpretability error for your experiments? What about computational complexity?\n\nLIMITATIONS:\nNot really a limitation but rather an assumption: using higher order polynomial to approximate the nonlinearity of DNN.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-12"}