SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left. \left\| g _ {i} \right| _ {x \in \mathcal {X} _ {i}} - \bar {g} \right| _ {x \in \mathcal {X} _ {i}} \left\| _ {2} = \left\| g _ {i} \right| _ {x \in \mathcal {X}} - \bar {g} \right| _ {x \in \mathcal {X}} \left\| _ {2} = \left\| g _ {i} \right| _ {x \in \mathcal {X}} - \bar {g} \right| _ {x \in \mathcal {X}} \| _ {2} \geq \| g _ {i} | _ {x \in \mathcal {X} _ {i}} - \bar {g} | _ {x \in \mathcal {X} _ {i}} \| _ {2}.
$$
Let us employ the triangle inequality for norms. Specifically, for each  $i = 1,2,\dots ,N$ , we apply the triangle inequality as follows:
$$
\left\| f \right| _ {x \in \mathcal {X} _ {i}} - g _ {i} \left| _ {x \in \mathcal {X} _ {i}} \right| _ {2} + \left\| g _ {i} \right| _ {x \in \mathcal {X} _ {i}} - \bar {g} \left| _ {x \in \mathcal {X} _ {i}} \right| _ {2} \geq \left\| f \right| _ {x \in \mathcal {X} _ {i}} - \bar {g} \left| _ {x \in \mathcal {X} _ {i}} \right| _ {2}.
$$
Summing this inequality over all  $i$  yields:
$$
\sum_ {i = 1} ^ {N} \| f | _ {x \in \mathcal {X} _ {i}} - g _ {i} | _ {x \in \mathcal {X} _ {i}} \| + \sum_ {i = 1} ^ {N} \| g _ {i} | _ {x \in \mathcal {X} _ {i}} - \bar {g} | _ {x \in \mathcal {X} _ {i}} \| \geq \sum_ {i = 1} ^ {N} \| f | _ {x \in \mathcal {X} _ {i}} - \bar {g} | _ {x \in \mathcal {X} _ {i}} \| _ {2}.
$$
By the definitions of  $\mathbb{I}_{\mathrm{total}}$  and  $\mathbb{D}_{\mathrm{total}}$ , we arrive at:
B.1 Harmonica Algorithm
To present the theoretical guarantees of the Harmonica algorithm, we introduce the following definition, which is slightly different from its original version in [27].
Definition B.1 (Approximately sparse function). We say a function  $f \in \{-1, 1\}^n \to \mathbb{R}$  is  $(\epsilon, s, C)$ -bounded, if  $\mathbb{E}[(f - \sum_{\chi_S \in C} \hat{f}(S) \chi_S)^2] \leq \epsilon$  and  $\sum_S |\hat{f}(S)| \leq s$ .
Here  $f$  is  $(\epsilon, s, C)$ -bounded means it is almost readable and has bounded  $\ell_1$  norm. Our algorithm is slightly different from the original algorithm proposed by [27], but similar theoretical guarantees still hold, as stated below.
Proof of Theorem 2. Our proof is similar to the one in the original paper [27], with changes in the readable notion, which is now more flexible than being low order. First, recall the classical Chebyshev inequality.
Theorem B.2 (Multidimensional Chebyshev inequality). Let  $X$  be an  $m$  dimensional random vector, with expected value  $\mu = \mathbb{E}[X]$ , and covariance matrix  $V = \mathbb{E}\left[(X - \mu)(X - \mu)^T\right]$ . If  $V$  is a positive definite matrix, for any real number  $\delta > 0$ :
$$
\mathbb {P} \left(\sqrt {(X - \mu) ^ {T} V ^ {- 1} (X - \mu)} > \delta\right) \leq \frac {m}{\delta^ {2}}.
$$
Proof of Theorem 2. Let  $f$  be an  $(\varepsilon /4,s,C)$ -bounded function written in the orthonormal basis as  $\sum_{S}\hat{f} (S)\chi_{S}$ . We can equivalently write  $f$  as  $f = h + g$ , where  $h$  is supported on  $C$  that only includes coefficients of magnitude at least  $\varepsilon /4s$  and the constant term of the polynomial expansion of  $f$ .
Since  $L_{1}(f) = \sum_{S} \left| \hat{f}_{S} \right| \leq s$ , we know  $h$  is  $4s^{2} / \varepsilon + 1$  sparse. The function  $g$  is thus the sum of the remaining  $\hat{f}(S) \chi_{S}$  terms not included in  $h$ . Denote the set of bases that appear in  $C$  but not in  $g$  as  $R$ , so we know the coefficient of  $f$  on the bases in  $R$  is at most  $\epsilon / 4s$ .
Draw  $m$  (to be chosen later) random labeled examples  $\left\{\left(z^{1},y^{1}\right),\ldots ,\left(z^{m},y^{m}\right)\right\}$  and enumerate all  $N = |C|$  basis functions  $\chi_S\in C$  as  $\{\chi_1,\dots ,\chi_N\}$ . Form matrix  $A$  such that  $A_{ij} = \chi_j\left(z^i\right)$  and consider the problem of recovering  $4s^{2} / \varepsilon +1$  sparse  $x$  given  $Ax + e = y$  where  $x$  is the vector of coefficients of  $h$ , the  $i$  th entry of  $y$  equals  $y^{i}$ , and  $e_i = g\left(z^i\right)$ .
We will prove that with constant probability over the choice  $m$  random examples,  $\| e\| _2\leq \sqrt{\varepsilon m}$ . Applying Theorem 5 in [27] by setting  $\eta = \sqrt{\varepsilon}$  and observing that  $\sigma_{4s^2 /\varepsilon +1}(x)_1 = 0$  (see definition in the theorem), we will recover  $x^{\prime}$  such that  $\| x - x^{\prime}\| _2^2\leq c_2^2\varepsilon$  for some constant  $c_{2}$ . As such, for the function  $\tilde{f} = \sum_{i = 1}^{N}x_{i}^{\prime}\chi_{i}$  we will have  $\mathbb{E}\left[\| h - \tilde{f}\| ^2\right]\leq c_2^2\varepsilon$  by Parseval's identity. Note, however, that we may rescale  $\varepsilon$  by a constant factor  $1 / (2c_2^2)$  to obtain error  $\varepsilon /2$  and only incur an additional constant (multiplicative) factor in the sample complexity bound. By the definition of  $g$ , we have
$$
\left\| g \right\| ^ {2} = \left(\sum_ {S, \chi S \notin C} \hat {f} (S) ^ {2} + \sum_ {S \in R} \hat {f} (S) ^ {2}\right). \tag {13}
$$
where each  $\hat{f}(S)$  for  $S \in R$  is of magnitude at most  $\varepsilon/4s$ . By Fact 4 in [27] and Parseval's identity we have  $\sum_{R} \hat{f}(R)^2 \leq \varepsilon/4$ . Since  $f$  is  $(\varepsilon/4, s, C)$ -concentrated we have  $\sum_{S, \chi_S \notin C} \hat{f}(S)^2 \leq \varepsilon/4$ . Thus,  $\|g\|^2$  is at most  $\varepsilon/2$ . Therefore, by triangle inequality  $\mathbb{E}\left[\|f - \tilde{f}\|^2\right] \leq \mathbb{E}\left[\|h - \tilde{f}\|^2\right] + \mathbb{E}\left[\|g\|^2\right] \leq \varepsilon$ . It remains to bound  $\|e\|_2$ . Note that since the examples are chosen independently, the entries
$e_i = g(z^i)$  are independent random variables. Since  $g$  is a linear combination of orthonormal monomials (not including the constant term), we have  $\mathbb{E}_{z\sim D}[g(z)] = 0$ . Here we can apply linearity of variance (the covariance of  $\chi_{i}$  and  $\chi_{j}$  is zero for all  $i\neq j$ ) and calculate the variance
$$
\operatorname {V a r} \left(g \left(z ^ {i}\right)\right) = \left(\sum_ {S, \chi_ {S} \notin C} \hat {f} (S) ^ {2} + \sum_ {S \in R} \hat {f} (S) ^ {2}\right).
$$
With the same calculation as Eqn. (13), we know  $\operatorname{Var}\left(g\left(z^i\right)\right)$  is at most  $\varepsilon /2$ . Now consider the covariance matrix  $V$  of the vector  $e$  which equals  $\mathbb{E}\left[ee^\top\right]$  (recall every entry of  $e$  has mean 0). Then  $V$  is a diagonal matrix (covariance between two independent samples is zero), and every diagonal entry is at most  $\varepsilon /2$ . Applying Theorem B.2 we have
$$
\mathbb {P} \left(\| e \| _ {2} > \sqrt {\frac {\varepsilon}{2}} \delta\right) \leq \frac {m}{\delta^ {2}}.
$$
Setting  $\delta = \sqrt{2m}$ , we conclude that  $\mathbb{P}(\| e\|_2 > \sqrt{\varepsilon m}) \leq \frac{1}{2}$ . Hence with probability at least  $1/2$ , we have that  $\| e\|_2 \leq \sqrt{\varepsilon m}$ . From Theorem 5 in [27], we may choose  $m = \tilde{O}\left(s^2/\varepsilon \cdot \log n^d\right)$ . This completes the proof. Note that the probability  $1/2$  above can be boosted to any constant probability with a constant factor loss in sample complexity.
For the running time complexity, we refer to [4] for optimizing linear regression with  $\ell_1$  regularization. The running time is  $O((T\log \frac{1}{\epsilon} + L / \epsilon) \cdot |C|)$ , where  $L$  is the smoothness of each summand in the objective. Since each  $\chi_S$  takes value in  $\{-1, 1\}$ , the smoothness is bounded by the number of entries in each summand, which is  $|C|$ . Therefore, the running time is bounded by  $O((T\log \frac{1}{\epsilon} + |C| / \epsilon) \cdot |C|)$ .
C Uncertainty Principle for Boolean functions
In the field of modern physics, the state of a particle on a given domain  $S$  is represented by a complex function on that domain, and the probability of finding the particle in a specific position  $x$  on  $S$  is given by the square of the modulus of the function evaluated at  $x$ . To ensure that the function is normalized, in the case where  $S$  is continuous, the function must satisfy  $\int_{x\in \mathbb{R}}|f(x)|^2 dx = 1$ .
The Fourier transform of the function, denoted by  $\hat{f}$ , is also normalized in  $L_{2}$  under a unitary transformation, and  $|\hat{f}(x)|^{2}$  represents the probability density function of the distribution of the particle's momentum.
The Heisenberg uncertainty principle states that the product of the variances of a particle's position and momentum is at least one, with an appropriate choice of units. This principle has physical significance and relates a function on  $\mathbb{R}$  to its Fourier transform.
In 1957, [29] proposed an entropic form of the uncertainty principle, which was later proven nearly two decades later by Beckner [9]. The inequality states that  $H_{e}[f] + H_{e}[\hat{f}] \geq 1 - \ln 2$ , where  $H_{e}[f]$  is the differential entropy of  $f$ , defined as  $-\int_{x \in \mathbb{R}} |f(x)|^{2} \ln |f(x)|^{2} dx$ .
When the domain is  $\mathbb{F}_2^n$  (equivalently,  $\{-1,1\}^n$ ), a similar inequality [25] holds with a different constant. Let  $f:\mathbb{F}_2^n\to \mathbb{C}$  have a Fourier transform  $\hat{f}:\mathbb{F}_2^n\to \mathbb{C}$ . Then,
$$
H \left[ \frac {f}{\| f \|} \right] + H \left[ \frac {\hat {f}}{\| \hat {f} \|} \right] \geq n,
$$
where  $H[f] = -\sum_{x \in \mathbb{F}_2^n} |f(x)|^2 \log_2 |f(x)|^2$ , and  $\|f\| = \sqrt{\sum_{x \in \mathbb{F}_2^n} f(x)^2}$ .
Now we present the discrete uncertainty principle for  $\mathbb{Z}_2^n$  as the following. It can be proved using Theorem 23 in Dembo, Cover, and Thomas [19].
All the Fourier coefficients together are called the Fourier spectrum of  $f$ . We can define the following distance between two functions based on their spectrums.
As there may be many candidates  $p$  for evaluating interpretation error and spectrum distance. In the following, we will investigate the most natural choice for  $p$ . As the Fourier expansion uniquely determined a Boolean function, we would like the distance between their expansion to have the same tendency as the interpretation error, thus the spectrum can encode enough information to reflect the accuracy of the interpretation. We shall first discuss the most "strict" case, i.e. the  $L_0$  norm.
We shall introduce some notations for better understanding. Denote the support of an Boolean function  $f$  as  $\operatorname{supp}(f) \triangleq \{x \in \{-1, 1\}^n \mid f(x) \neq 0\}$ . The support of Fourier coefficients (Fourier spectrum), is defined by  $\operatorname{supp}(\hat{f}) \triangleq \{S \subseteq [n] \mid \hat{f}(S) \neq 0\}$ . The uncertainty principle for Boolean functions is stated as follows.
Proposition C.1 (Uncertainty Principle for Boolean functions [25, 43]). For every nonzero function  $f: \{-1, 1\}^n \to \mathbb{R}$ ,
$$
| \operatorname {s u p p} (f) | \cdot | \operatorname {s u p p} (\hat {f}) | \geq 2 ^ {n}.
$$
D Low-degree Algorithm
The low-degree algorithm is based on the concentration inequality, and it estimates the coefficient of each axis individually.
Algorithm 4: Low-degree
1. Given uniformly randomly sampled  $x_{1},\dots ,x_{T}$  , evaluate them on  $f$  ..  $\{f(x_1),\ldots ,f(x_T)\} .$
2. For any  $\chi_S\in C$  , let  $\hat{g}_S = \frac{\sum_{i = 1}^Tf(x_i)\chi_S(x_i)}{T}$
3. Output the polynomial  $g(x) = \sum_{S, \chi_S \in C} \hat{g}_S \chi_S(x)$ .
Theorem D.1 ([36]). Given any  $\epsilon, \delta > 0$ , assuming that function  $f$  is bounded by  $B$ , when  $T \geq \frac{2B^2}{\epsilon^2} \log \frac{2|C|}{\delta}$ , we have
$$
\Pr \left[ \forall \chi_ {S} \in C, s. t., | \hat {g} _ {S} - \hat {f} _ {S} | \leq \epsilon \right] \geq 1 - \delta .
$$
Theorem D.1 was proved using the Hoeffding bound, and we included the proof here for completeness.
Proof. Since we are given  $T$  samples to estimate  $\hat{f}(S)$  for every  $S$ , we can directly apply the Hoeffding bound (notice that the function is bounded by  $B$ ):
$$
\Pr \left(| \alpha_ {S} - \hat {f} (S) | \geq \epsilon\right) \cdot | C | \leq 2 \exp \left(- \frac {2 T \epsilon^ {2}}{4 B ^ {2}}\right) = 2 \exp \left(- \frac {T \epsilon^ {2}}{2 B ^ {2}}\right).
$$
Notice that  $T \geq \frac{2B^2}{\epsilon^2} \log \frac{2|C|}{\delta}$ , we know the right hand size is bounded by  $\frac{\delta}{|C|}$ , so Theorem D.1 is proved.
Remarks. The theoretical guarantee of Harmonica assumes the target function  $f$  is approximately sparse in the Fourier space, which means most of the energy of the function (Fourier coefficients) is concentrated in the bases in  $C$ . This is not a strong assumption, because if  $f$  is not approximately sparse, it means  $f$  has energy in many different bases, or more specifically, the bases with higher orders. In other words,  $f$  has a large variance and is difficult to interpret. In this case, no existing algorithms will be able to give consistent and meaningful interpretations.
Likewise, although the Low-degree algorithm does not assume sparsity for  $f$ , it cannot learn all possible functions accurately as well. There are  $2^n$  different bases, and if we want to learn the coefficients for all of them, the cumulative error for  $g$  is at the order of  $\Omega(2^n\epsilon)$ , which is exponentially large. This is not surprising due to the no free lunch theorem in the generalization theory, as we do not expect to be able to learn "any functions" without exponentially many samples.
E Discussion on the Existing Algorithms
In this section, we compare our approaches with the existing techniques from different perspectives of interpretation error (efficiency), truthfulness, and consistency.
LIME [47]. Given an input  $x$ , Lime samples the neighborhood points based on a sampling distribution  $\Pi_x$ , and optimizes the following program:
$$
\min _ {g \in \mathcal {G}} L (f, g, \Pi_ {x}) + \Omega (g).
$$
where  $L$  is the loss function describing the distance between  $f$  and  $g$  on the sampled data points,  $\mathcal{G}$  is the set of readable functions (e.g. the set of linear functions),  $\Omega(\cdot)$  is a function that characterizes the complexity of  $g$ . In other words, LIME tries to minimize the fitting error and simultaneously minimizes the complexity of  $g$  (which is usually the sparsity of the linear function). By minimizing  $L$ , LIME also works towards minimizing the interpretation error, but their approach is purely heuristic, without any theoretical guarantees. Although their readable function set can easily generalize to the set with higher order terms, the sampling distribution  $\Pi_x$  is not uniform, so it is difficult to incorporate the orthonormal basis into their framework. In other words, the model they compute is not truthful.
Attribution methods. As we discussed in the introduction, attribution methods mainly focus on individual inputs, instead of the neighboring points. Therefore, it is difficult for the attribution methods to achieve low inconsistency, especially for first-order methods like SHAP [38] and IG [60].
Consider SHAP [38] as a motivating example, illustrated in Figure 1 for the task of sentiment analysis of movie reviews. In this example, the interpretations of two slightly different sentences are inconsistent. This inconsistency arises not only because the weights of each word differ significantly, but also because after removing the word "very" with a weight of  $15.0\%$ , the network's output only drops by  $6.6\%$ . In other words, the interpretation does not explain the network's behavior even in a small neighborhood of the input.
For higher-order attribution methods, consistency can potentially be improved due to their enhanced representation power. The classical Shapley interaction index has the problem of not precisely fitting the underlying function, as observed by [61], who proposed Shapley Taylor interaction index [61] with better empirical performance. Shapley Taylor interaction index satisfies the generalized efficiency axiom, which says for all  $f \in \{-1, 1\}^n \to \mathbb{R}$ ,
$$
\sum_ {S \subseteq [ n ], | S | \leq k} \mathcal {I} _ {S} ^ {k} (f) = f ([ n ]) - f (\emptyset).
$$
We should remark that both the Shapley interaction index and Shapley Taylor interaction index were not originally designed for consistent interpretations, so they did not specify how to generalize the interpretation for the neighboring points. To this end, we make a global extension to the Shapley value based interpretation, that is, using Shapley interaction indices or Shapley Taylor interaction indices as the coefficients of corresponding terms of the polynomial surrogate function.
$$
g \left(x _ {1}, x _ {2}, \dots , x _ {n}\right) = f (\emptyset) + \sum_ {x _ {i} \in S, S \subseteq [ n ]} \mathcal {I} (f, S).
$$
However, these higher-order Shapley value based methods all focus on the original Shapley value framework, so their interpretations are not truthful, i.e., not getting the exact coefficient of  $f$  even on the "simple bases". Moreover, as we will show in our experiments, higher-order methods still incur high interpretation errors compared with our methods.
When applying Shapley value techniques for visual search, [26] proposed an interesting and novel sampling  $+$  Lasso regression algorithm for efficiently computing higher order Shapley Taylor index in their experiments. However, their methods are based on a sampling probability distribution generated from permutation numbers, which is far from the uniform distribution. Additionally, their algorithm is based on the Shapley Taylor index, so their method is not truthful as well.
Discussion on universal consistency. When discussing consistency, there are two distinct settings: "global consistency" and "universal consistency." This paper mainly focuses on global consistency (Definition 2.2), where the interpretation pertains only to input features and not to others. This scenario falls under the category of "removal-based explanations," and most existing interpretation methods belong to this category (26 of them were discussed in [18]). In contrast, universal consistency implies that the interpretation may depend on features different from the input features. An example of interpreting the sentence "I am happy" could be, "This sentence does not include [very], so this person is not extremely happy." Universal consistency is more challenging than global consistency, and we hypothesize that more powerful machinery, such as foundation models, is needed.
Emergence of self-explainability in foundation models. In the groundbreaking work Sparks of AGI [11], the authors conducted comprehensive experiments on GPT-4's self-explainability. Chain-of-thought reasoning [73, 71, 76, 79] can be considered an interpretation provided by the model itself, although current LLMs are not always to be consistent or truthful. In the long run, however, this approach may represent the ultimate path to explainable AI. In the computer vision area, DINO [12, 44] emerges the ability of self-explanation provided by the attention map inside the model, which can be directly utilized as heatmap visualization or even applied to segmentation tasks.
Proposition E.1 (Omniscient LLM is the way). An (infinitely) large (multi-modal) language model pre-trained using (infinitely) large corpus, with zero pre-training loss and generalization loss, which has been perfectly aligned to be truthful, is interpretable, efficient, and consistent.
Proof. To prove the proposition, let us examine each attribute (interpretable, efficient, and consistent) in the context of an (infinitely) large (multi-modal) language model (LLM):
1. Interpretable: By assumption, the LLM has zero pre-training loss and generalization loss, implying perfect knowledge representation. Furthermore, it is perfectly aligned to be truthful, satisfying the criteria for interpretability as per Definition 2.4.
2. Efficient: The efficiency condition  $g(x) = f(x)$  is trivially satisfied because  $f = g$  by the condition of self-explainability. Thus, the LLM is efficient as per Definition 2.3.
3. Consistent: Given that the LLM's responses are generated based on a perfect understanding and alignment, it will generate the same function  $\mathcal{A}(f,x)$  for every  $x\in \mathcal{X}$ , thereby being consistent as per Definition 2.2.
It's crucial to note that self-explainability does not violate the Impossible Trinity (Theorem 1). When  $f = g$ , the LLM serves as its own interpreter, inherently satisfying all three criteria. Alternatively, if  $f \subseteq g$ , where  $g$  has an even larger concept class than  $f$ , the interpretability criteria are also met, albeit this is not a practical scenario for standard interpretation algorithms.
Therefore, an (infinitely) large LLM, pre-trained with an (infinitely) large corpus and aligned perfectly to be truthful, satisfies all the conditions to be interpretable, efficient, and consistent.
F Test with Low order Polynomial Functions
F.1 First order polynomial function
To investigate the performance of different interpretation methods, let us take a closer look at a 1st order polynomial function:
$$
f _ {1} \left(x _ {1}, x _ {2}, x _ {3}\right) = \frac {1}{2} x _ {1} - \frac {1}{3} x _ {2} + \frac {1}{4} x _ {3},
$$
For this simple function, we can manually compute the outcome of each algorithm, as illustrated in Table 2. If the algorithm's output is correct, i.e., equal to the output of  $f_{1}$ , we write a check mark. Otherwise, we write down the actual output of the given interpretation algorithm.
As we can see, all methods are consistent and efficient for all cases. In fact, all variants of Shapley indices degraded to 1st order Shapley values.
Algorithms	(-1,-1,-1)	(-1,-1,+1)	(-1,+1,-1)	(-1,+1,+1)	(+1,-1,-1)	(+1,-1,+1)	(+1,+1,-1)	(+1,+1,+1)
Ground Truth	-0.417	+0.083	-1.083	-0.583	+0.583	+1.083	-0.083	+0.417
LIME	✓	✓	✓	✓	✓	✓	✓	✓
SHAP	✓	✓	✓	✓	✓	✓	✓	✓
Shapley Interaction Index (1st order)	✓	✓	✓	✓	✓	✓	✓	✓
Shapley Taylor Index (1st order)	✓	✓	✓	✓	✓	✓	✓	✓
Faith-Shap (1st order)	✓	✓	✓	✓	✓	✓	✓	✓
Low-degree (1st order)	✓	✓	✓	✓	✓	✓	✓	✓
Harmonica (1st order)	✓	✓	✓	✓	✓	✓	✓	✓
Table 2: Interpretations by LIME, SHAP, Shapley Interaction Index, Shapley Taylor Index, Faith-Shap, Low-degree, and Harmonica on the 1st order polynomial function  $f_{1}$ .
F.2 Second order polynomial function
In addition, let us take a closer look at a 2nd order polynomial function:
$$
f _ {2} \left(x _ {1}, x _ {2}, x _ {3}\right) = \frac {1}{2} x _ {1} - \frac {1}{3} x _ {2} + \frac {1}{4} x _ {3} - \frac {1}{5} x _ {1} x _ {2} + \frac {1}{6} x _ {1} x _ {3} - \frac {1}{7} x _ {2} x _ {3},
$$
For this simple function, we can manually compute the outcome of each algorithm, as illustrated in Table 3. If the algorithm's output is correct, i.e., equal to the output of  $f_{2}$ , we write a check mark. Otherwise, we write down the actual output of the given interpretation algorithm.
As we can see, 2nd order interpretation algorithms, including Shapley Taylor index, Faithful Shapley, Low-degree, and Harmonica are consistent and efficient for all cases. Other methods can only fit a few inputs, the 2nd order Shapley interaction index misses all the cases because it is not efficient, and LIME misses all the cases because  $f_{2}$  is not a linear function.
Algorithms	(-1,-1,-1)	(-1,-1,+1)	(-1,+1,-1)	(-1,+1,+1)	(+1,-1,-1)	(+1,-1,+1)	(+1,+1,-1)	(+1,+1,+1)
Ground Truth	-0.593	-0.140	-0.574	-0.693	+0.474	+1.593	-0.307	+0.240
LIME	-0.417	+0.083	-1.083	-0.583	+0.583	+1.083	-0.083	+0.417
SHAP	-0.240	+0.283	-1.250	-0.726	+0.726	+1.250	-0.283	✓
Shapley Interaction Index (2nd order)	-0.329	+0.171	-0.995	-0.781	+0.671	+1.505	-0.395	+0.152
Shapley Taylor Index (2nd order)	✓	✓	✓	✓	✓	✓	✓	✓
Faith-Shap (2nd order)	✓	✓	✓	✓	✓	✓	✓	✓
Low-degree (2nd order)	✓	✓	✓	✓	✓	✓	✓	✓
Harmonica (2nd order)	✓	✓	✓	✓	✓	✓	✓	✓
F.3 Third order polynomial function
Finally, we investigate the following 3rd order polynomial and present the result in Table 4.
$$
f _ {3} \left(x _ {1}, x _ {2}, x _ {3}\right) := \frac {1}{2} x _ {1} - \frac {1}{3} x _ {2} + \frac {1}{4} x _ {3} - \frac {1}{5} x _ {1} x _ {2} + \frac {1}{6} x _ {1} x _ {3} - \frac {1}{7} x _ {2} x _ {3} + \frac {1}{8} x _ {1} x _ {2} x _ {3},
$$