SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yifan Zhang*1
Haowei He\*1
Zhiquan Tan
Yang Yuan $^{1,3,4\dagger}$
$^{1}$ IIIS, Tsinghua University
$^{2}$ Department of Math, Tsinghua University
$^{3}$ Shanghai Artificial Intelligence Laboratory
$^{4}$ Shanghai Qizhi Institute
{zhangyif21,hhw19,tanzq21}@mails.tsinghua.edu.cn
In the current landscape of explanation methodologies, most predominant approaches, such as SHAP and LIME, employ removal-based techniques to evaluate the impact of individual features by simulating various scenarios with specific features omitted. Nonetheless, these methods primarily emphasize efficiency in the original context, often resulting in general inconsistencies. In this paper, we demonstrate that such inconsistency is an inherent aspect of these approaches by establishing the Impossible Trinity Theorem, which posits that interpretability, efficiency, and consistency cannot hold simultaneously. Recognizing that the attainment of an ideal explanation remains elusive, we propose the utilization of interpretation error as a metric to gauge inefficiencies and inconsistencies. To this end, we present two novel algorithms founded on the standard polynomial basis, aimed at minimizing interpretation error. Our empirical findings indicate that the proposed methods achieve a substantial reduction in interpretation error, up to 31.8 times lower when compared to alternative techniques $^{\ddagger}$ .
Most existing explanation approaches are removal-based [18], which involve the sequential process of eliminating certain input features, examining the subsequent alterations in the model's behavior, and ascertaining each feature's impact through observation. However, most of these methods [60, 38] primarily focus on the original input with all features, often yielding inconsistent outcomes in alternative scenarios. Consequently, the resulting interpretations may not explain the network's behavior consistently even in a small neighborhood of the input, as demonstrated in Figure 1.
Inconsistency is a non-trivial concern. Imagine a doctor treating a diabetic patient with the help of an AI system. The patient has features A, B, and C, representing three positive signals from various tests. The AI recommends administering 4 units of insulin with the following explanation: A, B, and C have weights 1, 1, and 2, respectively, amounting to 4 units in total. The doctor might then ask the AI: what if the patient only has A and B, but not C? One might expect the answer to be close to 2, as A+B has a weight of 2. However, the network, being highly non-linear, might output a different suggestion like 3 units, explaining that both A and B have a weight of 1.5. Such inconsistent
$\ddagger$ Code is available at https://github.com/trusty-ai/efficient-consistent-explanations.
behaviors can significantly reduce the doctor's confidence in the interpretations, limiting the practical value of the AI system.
Figure 1: Interpretations generated by SHAP on movie review.
Consistency (see Definition 2.2) is certainly not the only objective for interpretability. Being equally important, efficiency is a commonly used axiom in the attribution methods [72, 22, 59], also called local accuracy [38] or completeness [60], stating that the model's output should be equal to the network's output for the given input (see Definition 2.3). Naturally, one may ask the following question:
Q1: Can we generate an interpreting model that is both consistent and efficient?
Unfortunately, this is generally unattainable. We have proved the following theorem in Section 2:
Theorem 1(Impossible trinity, informal version). For post-hoc interpreting models, interpretability, efficiency and consistency cannot hold simultaneously.
A few examples following Theorem 1:
(a) Most attribution methods are interpretable and efficient, but not consistent.
(b) The original (deep) network is consistent and efficient, but not interpretable.
(c) If one model is interpretable and consistent, it cannot be efficient.
However, consistency is necessary for many scenarios, leading to the follow-up question:
Q2: For consistent interpreting models, can they be approximately efficient?
The answer depends on the definition of "approximately efficient". We introduce a new notion called truthfulness, which serves as a natural relaxation of efficiency, or partial efficiency. We divide the functional space of a network  $f$  into two subspaces: the readable part and the unreadable part. We call the interpreting model  $g$  truthful if it can accurately represent the readable part of  $f$ , denoted as  $V$  (see Definition 2.4). The unreadable part is not merely a "fitting error"; it truthfully represents the higher-order non-linearities in the network that our interpretation model  $g$ , even at its best, cannot cover. In short, what  $g$  conveys is true, though it may not encompass the entire truth. Due to Theorem 1, this is essentially the best that consistent algorithms can achieve.
Truthfulness is a parameterized notion, depending on the choice of the readable subspace. While there are theoretically infinite possible choices, we follow previous researchers on interpretability with non-linearities [61, 41, 63], using the basis that induces interpretable terms like  $x_{i}x_{j}$  or  $x_{i}x_{j}x_{k}$ . These capture higher-order correlations and are easy to understand. The resulting subspace has the standard polynomial basis (or equivalently, the Fourier basis).
When truthfulness is parameterized with the polynomial basis, designing consistent and truthful interpreting models is equivalent to learning the Fourier spectrum (see Algorithm 1 and Lemma 3.4). However, exact consistency is not always necessary for most real-world applications, as approximate consistency is usually sufficient. We formally use the number of different interpreting models (in log scale) as a metric for inconsistency. Our last question is:
Q3: When a little inconsistency is allowed, can we get better interpreting models?
We affirmatively answer this question in Section 4 by introducing a new algorithm called Harmonical-local. We then apply it multiple times to develop Harmonica-anchor and Harmonica-anchor-constrained, which offer smaller interpretation errors with a small degree of inconsistency.
In this paper, we focus on removal-based explanations [18], which implies that  $f$  and  $g$  are Boolean functions. We remark that empirically most networks are not Boolean functions, i.e., the input variables are real valued. However, for every explanation algorithm (including LIME [47] and SHAP [38]) in the removal-based framework, the input  $x$  is not modified to an arbitrary real value. Instead,  $x$  is fixed, and the algorithm only retain or remove each feature of  $x$  for generating the explanations. When  $x$  is fixed with  $n$  features, it becomes natural to use Boolean functions to represent both the network  $f$  and the interpreting model  $g$  for their outputs in all  $2^{n}$  feature-removal scenarios. In other words, the algorithms in the framework only consider a fixed  $x$  each time, and given this  $x$ , there are only  $2^{n}$  possible outcomes, even for the models with real inputs. Therefore, treating  $f$  and  $g$  as Boolean functions is not a simplification, but an accurate characterization of the removal-based framework (see Figure 1 in [18] for an illustration).
Our new algorithms, including Harmonica-local, Harmonica-anchor, and Harmonica-anchor-constrained, are all based on the Harmonica algorithm from Boolean functional analysis [27], which has rigorous theoretical guarantees on recovery performance and sampling complexities. In Section 5, we demonstrate that on datasets like IMDb and ImageNet, our algorithms achieve up to  $31.8\mathrm{x}$  lower interpretation error compared with other methods.
- We prove the impossible trinity theorem for removal-based explanations, demonstrating that interpretable algorithms cannot be consistent and efficient simultaneously.
- When a small inconsistency is allowed, we propose new algorithms using Harmonica-local and empirically demonstrate that these algorithms achieve significantly lower interpretation errors compared with other methods.
- For interpretable algorithms that are consistent but not efficient, we introduce a new notion called truthfulness, which can be regarded as partial efficiency. Due to the impossible trinity theorem, this is the best achievable outcome when consistency is required.
2 Our Framework on Interpretability
We consider a Hilbert space  $\mathcal{H}$  equipped with inner product  $\langle \cdot, \cdot \rangle$ , and induced norm  $\|\cdot\|$ . We denote the input space by  $\mathcal{X}$ , the output space by  $\mathcal{Y}$ , which means  $\mathcal{H} \subseteq \mathcal{X} \to \mathcal{Y}$ . We use  $\mathcal{G} \subset \mathcal{H}$  to denote the set of interpretable functions, and  $\mathcal{F} \subset \mathcal{H}$  to denote the set of machine learning models that need interpretation. In this paper, if not mentioned otherwise we focus on models that are not self-interpretable, i.e.,  $f \in \mathcal{F} \setminus \mathcal{G}$ .
Definition 2.1 (Interpretable and Interpretation Algorithm). We call A model  $g$  is interpretable, if  $g \in \mathcal{G}$ . An interpretation algorithm  $\mathcal{A}$  takes  $f \in \mathcal{H}, x \in \mathcal{X}$  as inputs, and outputs  $\mathcal{A}(f, x) \in \mathcal{G}$  for interpreting  $f$  on  $x$ .
As we mentioned previously, for many interdisciplinary fields, the interpretation algorithm should be consistent.
Definition 2.2 (Consistent). Given  $f \in \mathcal{H}$ , an interpretation algorithm  $\mathcal{A}$  is consistent with respect to  $f$ , if  $\mathcal{A}(f, x)$  remains the same (function) for every  $x \in \mathcal{X}$ .
Efficiency is an important property of the attribution methods.
Definition 2.3 (Efficient). A model  $g \in \mathcal{H}$  is efficient with respect to  $f \in \mathcal{F}$  on  $x \in \mathcal{X}$ , if  $g(x) = f(x)$ .
The following theorem states that one cannot expect to achieve the best of all three worlds.
Theorem 1 (Impossible Trinity for Removal-based Explanations). For any interpretation algorithm  $\mathcal{A}$  and function sets  $\mathcal{G} \subset \mathcal{F} \subseteq \mathcal{H}$ , there exists  $f \in \mathcal{F}$  such that with respect to  $f$ , either  $\mathcal{A}$  is not consistent, or  $\mathcal{A}(f, x)$  is not efficient on  $x$  for some  $x \in \mathcal{X}$ .
Proof. Please refer to Appendix B for all the proofs.
Theorem 1 says efficiency is too restrictive for consistent interpretations. However, being inefficient does not mean the interpretation is wrong, it can still be truthful. Recall a subspace  $V \subset \mathcal{H}$  is closed if whenever  $\{f_n\} \subset V$  converges to some  $f \in \mathcal{H}$ , then  $f \in V$ . We have:
Definition 2.4 (Truthful gap and truthful). Given a closed subspace  $V \subseteq \mathcal{H}$ ,  $g \in \mathcal{G} \subseteq V$  and  $f \in \mathcal{F}$ , the truthful gap of  $g$  to  $f$  for  $V$  is:
$$
\mathbb {T} _ {V} (f, g) \triangleq \| f - g \| ^ {2} - \inf  _ {v \in V} \| f - v \| ^ {2}. \tag {1}
$$
When  $\mathbb{T}_V(f,g) = 0$ , we say  $g$  is truthful for subspace  $V$  with respect to  $f$ , and we know (see e.g. Lemma 4.1 in [56])  $\forall v \in V, \langle f - g, v \rangle = 0$ .
Truthfulness means  $g$  fully captures the information in the subspace  $V$  of  $f$ , therefore it can be seen as a natural relaxation of efficiency. To characterize the interpretation quality, we introduce the following notion.
Definition 2.5 (Interpretation error). Given functions  $f, g \in \mathcal{X} \to \mathcal{Y}$ , the interpretation error between  $f$  and  $g$  with respect to measure  $\mu$  is
$$
\mathbb {I} _ {p, \mu} (f, g) \triangleq \left(\int_ {\mathcal {X}} | f (x) - g (x) | ^ {p} d \mu (x)\right) ^ {1 / p}. \tag {2}
$$
Notice that interpretation error is only a loss function that measures the quality of the interpretation, instead of a metric in  $\ell_p$  space. Therefore,  $\mu$  can be a non-uniform weight distribution following the data distribution. If  $\mu$  is uniform distribution over  $\mathcal{X}$ , we abbreviate  $\mathbb{I}_{p,\mu}(f,g)$  as  $\mathbb{I}_p(f,g)$ . For real-world applications, interpreting the model over the whole  $\mathcal{X}$  is unnecessary, so  $\mu$  is usually defined as a uniform distribution on the neighborhood of input  $x$  (under a certain metric), in which case we denote the distribution as  $\mathcal{N}_x$ .
3 Applying Our Framework to Removal-based Explanations
Now we focus on interpreting removal-based explanations [38, 18]. Removal-based explanations are post-hoc, which means they are generated based on a target network for a fixed input, by removing features from that input. There are three choices affecting the removal-based explanations: how features are removed, what behavior is analyzed after feature removal, and how to summarize the feature influence. For example, SHAP [38] considers all possible subsets of the features, and analyzes how holding out different features affects functional value, and finally summarizes the differences based on the Shapley value calculation.
Therefore, removal-based explanations can be represented as Boolean functions, as feature subset  $S \subseteq [n]$  can be represented as Boolean input:  $f \in \{-1, 1\}^n \to \mathbb{R}$ , where  $-1$  means the specific feature  $i \in S$  is removed, 1 means the specific feature is retained. We use  $-1/1$  instead of  $0/1$  to represent the binary variables for ease of exposition using the Fourier basis.
3.1 Fourier Basis and Truthful Gap
Fourier analysis is a handy tool for analyzing Boolean functions. Due to the space limit, we defer the more comprehensive introduction on Fourier analysis for Boolean functions to Appendix A, and only present the necessary notions here.
Definition 3.1 (Fourier basis). For any subset of variables  $S \subseteq [n]$ , we define the corresponding Fourier basis as  $\chi_S(x) \triangleq \Pi_{i \in S} x_i \in \{-1, 1\}^n \to \{-1, 1\}$ .
The Fourier basis is also called polynomial basis in the literature. It is a complete orthonormal basis for Boolean functions, under the uniform distribution on  $\{-1, 1\}^n$ . We remark that this uniform distribution is used for theoretical analysis and algorithm design, and is different from the measure  $\mu$  for interpretation quality assessment in Definition 2.5.
Definition 3.2 (Fourier expansion). Any Boolean function  $f \in \{-1, 1\}^n \to \mathbb{R}$  can be expanded as
$$
f (x) = \sum_ {S \subseteq [ n ]} \hat {f} _ {S} \chi_ {S} (x),
$$
where  $\hat{f}_S = \langle f, \chi_S \rangle$  is the Fourier coefficient on  $S$ .
Now we define the notion of  $C$ -Readable function.
Definition 3.3 ( $C$ -Readable function). Given a set of Fourier bases  $C$ , a function  $f$  is  $C$ -readable if it is supported on  $C$ . That is, for any  $\chi_S \notin C$ ,  $\langle f, \chi_S \rangle = 0$ . Denote the corresponding subspace as  $V_C$ .
The Readable notion is parameterized with  $C$ , because it may differ case by case. If we set  $C$  to be all the single variable bases, only linear functions are readable; if we set  $C$  to be all the bases with the degree at most 2, functions with pairwise interactions are also readable. Moreover, if we further add one higher order term to  $C$ , e.g.,  $\chi_{\{x_1,x_2,x_3,x_4\}}$ , it means we can also reason about the factor  $x_1x_2x_3x_4$  in the interpretation, which might be an important empirical factor that people can easily understand. Starting from the bases set  $C$ , we have the following formula for computing the truthful gap.
Lemma 3.4 (Truthful gap for Boolean functions). Given a set of Fourier bases  $C$ , two functions  $f, g \in \{-1,1\}^n \to \mathbb{R}$ , the truthful gap of  $g$  to  $f$  for  $C$  is
$$
\mathbb {T} _ {V _ {C}} (f, g) = \sum_ {\chi_ {S} \in C} \left\langle f - g, \chi_ {S} \right\rangle^ {2}. \tag {3}
$$
With the previous definitions, it becomes clear that finding a truthful interpretation  $g$  is equivalent to accurately learning a Boolean function with respect to the readable bases set  $C$ . Intuitively, it means we want to find algorithms that can compute the coefficients for the bases in  $C$ . In other words, we want to find the importance of the bases like  $x_{1}, x_{2}x_{5}, x_{2}x_{6}x_{7}$ , etc.
3.2 Representative Algorithms
Applying the impossible trinity theorem to removal-based explanations, there are two notable algorithms on the extremes.
Shapley values: efficient but not consistent. Let  $N = \{1,2,\dots,n\}$  represent a set of  $n$  players, and let  $v:2^{N}\to \mathbb{R}$  be a characteristic function that assigns a real value to each coalition of players. The Shapley value of player  $i$  is given by:
$$
\phi_ {i} (v) = \sum_ {S \subseteq N \backslash i} \frac {(n - | S | - 1) ! \cdot | S | !}{n !} [ v (S \cup i) - v (S) ], \tag {4}
$$
where  $|S|$  is the number of players in coalition  $S$ , and the sum is taken over all possible coalitions  $S$  that do not include player  $i$ . One of the most important properties of Shapley values is efficiency, i.e.  $\sum_{i\in N}\phi_i(v) = v(N) - v(\emptyset)$  (or denoting explanation function  $g\triangleq \sum_{i\in N}\phi_i(v) + v(\emptyset)$ ). As Shapley value based explanations do not focus on consistency, its explanation is efficient only for the original input, but not for other scenarios when certain features are removed.
Harmonica: consistent but not efficient. What can we do if we want to address consistency? As mentioned in Definition 2.4, we do not expect the algorithm to be efficient, but it can still be truthful with respect to a subspace  $V$ , which is naturally represented with the polynomial basis. Therefore, in order to learn truthful explanations for given subspace  $V$ , it is natural to consider LASSO regression over the coefficients on the polynomial basis. Harmonica (Algorithm 1) fulfills this requirement [27] and has superior complexity shown in Theorem 2.
Algorithm 1: Harmonica [27]
1. Given uniformly randomly sampled  $x_{1},\dots ,x_{T}$  , evaluate them on  $f$  ..  $\{f(x_1),\ldots ,f(x_T)\} .$
2. Solve the following regularized regression problem.
$$
\underset {\alpha \in \mathbb {R} ^ {| C |}} {\operatorname {a r g m i n}} \left\{\sum_ {i = 1} ^ {T} \left(\sum_ {S, \chi_ {S} \in C} \alpha_ {S} \chi_ {S} \left(x _ {i}\right) - f \left(x _ {i}\right)\right) ^ {2} + \lambda \| \alpha \| _ {1} \right\} \tag {5}
$$
3. Output the polynomial  $g(x) = \sum_{S, \chi_S \in C} \alpha_S \chi_S(x)$ .
Theorem 2 (Complexity of Harmonica). Given  $f \in \{-1,1\}^n \to \mathbb{R}$ , a  $(\epsilon /4,s,C)$ -bounded function, Algorithm 1 finds a function  $g$  with interpretation error at most  $\epsilon$  in time  $O((T\log \frac{1}{\epsilon} +|C| / \epsilon)\cdot |C|)$  and sample complexity  $T = \tilde{O} (s^{2} / \epsilon \cdot \log |C|)$ .
We defer detailed descriptions and theoretical guarantees of these algorithms to Appendix B.1, C and D, discussion on the comparison of our algorithms with the existing algorithms to Appendix E.
3.3 Quantifying Inconsistency
All the Fourier coefficients together are called the Fourier spectrum of  $f$ . We can define the following distance between two functions based on their spectrums as the quantitative measure of the inconsistency between different explanations.
Definition 3.5 (Spectrum (Fourier) distance for Boolean functions). Given two interpretations  $g(\cdot), h(\cdot) \in \{-1,1\}^n \to \mathbb{R}$ , we define the spectrum  $p$ -distance between them as:
$$
\mathbb {D} _ {p} (g (\cdot), h (\cdot)) \triangleq \left\| g - h \right\| _ {p} = \left(\sum_ {S \subseteq [ n ]} \left| \hat {g} _ {S} - \hat {h} _ {S} \right| ^ {p}\right) ^ {1 / p}. \tag {6}
$$
As there may be many candidates  $p$  for evaluating interpretation error and spectrum distance. In the following, we will investigate the most natural choice for  $p$ . As the Fourier expansion uniquely determined a Boolean function. We would like the distance between their expansion to have the same tendency as the interpretation error, thus the spectrum can encode enough information to reflect the accuracy of the interpretation. We shall first discuss the most "strict" case, i.e. the  $L_0$  norm.
Denote the support of an Boolean function  $f$  as  $\operatorname{supp}(f) \triangleq \{x \in \{-1, 1\}^n \mid f(x) \neq 0\}$ . The support of Fourier coefficients (Fourier spectrum), is defined by  $\operatorname{supp}(\hat{f}) \triangleq \{S \subseteq [n] \mid \hat{f}(S) \neq 0\}$ . Based on Proposition C.1, we have the following uncertainty principle for removal-based explanations:
Theorem 3 (Uncertainty Principle for Removal-based Explanations). Assume  $f, g \in \{-1, 1\}^n \to \mathbb{R}$  and  $f \neq g$ . When evaluated under  $L_0$  norm, the interpretation error and spectrum distance (inconsistency value) have the following uncertainty principle (can be seen as a quantitative version of Impossible Trinity Theorem 1):
$$
\mathbb {I} _ {0} (f, g) \mathbb {D} _ {0} (f, g) \geq 1. \tag {7}
$$
We can see that the  $L_{0}$  norm is too strict for evaluating the differences between interpretations (inconsistency). Fortunately, the following proposition shows that the  $L_{2}$  relaxed version is much better, or to say, natural, due to Parseval's identity.
Proposition 3.6 ( $L_{2}$  norm is natural). When taking  $\mu$  as a uniform distribution, for  $f,g\in \{-1,1\} ^n\to \mathbb{R}$ , we have  $\mathbb{I}_2(f,g) = \mathbb{D}_2(f,g)$ .
Now we present a theorem on the trade-off between efficiency and consistency, quantified by total interpretation error and total inconsistency value (spectrum distance).
Theorem 4 (Trade-off between Efficiency and Consistency). Given a function  $f: \{-1,1\}^n \to \mathbb{R}$  and a set of  $N$  interpretable functions  $\{g_1, g_2, \ldots, g_N\}$  defined on disjointed sets  $\mathcal{X}_i$  ( $\mathcal{X} = \bigcup \mathcal{X}_i$ ), each  $g_i: \{-1,1\}^n \to \mathbb{R}$ , let  $\bar{g}$  denote the average (spectrum) of  $g_i$  across the space  $\mathcal{X} = \{-1,1\}^n$ . Define (we use  $L_2$  norm as default):
$$
\mathbb {I} _ {\text {t o t a l}} = \sum_ {i = 1} ^ {N} \| f | _ {x \in \mathcal {X} _ {i}} - g _ {i} | _ {x \in \mathcal {X} _ {i}} \| _ {2},
$$
$$
\mathbb {D} _ {\text {t o t a l}} = \sum_ {i = 1} ^ {N} \left\| g _ {i} \right| _ {x \in \mathcal {X} _ {i}} - \bar {g} | _ {x \in \mathcal {X} _ {i}} \left\| _ {2}, \right.
$$
Then, we have the following inequality:
$$
\mathbb {I} _ {\text {t o t a l}} + \mathbb {D} _ {\text {t o t a l}} \geq \sum_ {i = 1} ^ {N} \| f | _ {x \in \mathcal {X} _ {i}} - \bar {g} | _ {x \in \mathcal {X} _ {i}} \| _ {2}.
$$
In addition, denote  $g^{\dagger}$  as the globally consistent interpretation  $g^{\dagger}$  defined on  $\{-1, 1\}^{n}$  with minimal interpretation error (can be obtained by Harmonica), we have:
$$
\sum_ {i = 1} ^ {N} \| f | _ {x \in \mathcal {X} _ {i}} - \bar {g} | _ {x \in \mathcal {X} _ {i}} \| _ {2} \geq \sum_ {i = 1} ^ {N} \| f | _ {x \in \mathcal {X} _ {i}} - g ^ {\dagger} | _ {x \in \mathcal {X} _ {i}} \| _ {2} = \mathbb {I} _ {2} (f, g ^ {\dagger}) = \mathbb {D} _ {2} (f, g ^ {\dagger}).
$$
Algorithm 2: Harmonica-anchor
Input: anchor number  $k$ , model  $f$ , a distance metric function  $d(\cdot, \cdot): (\chi_S, \chi_S) \to \mathbb{R}$  which calculates distance between two anchors (e.g.,  $L_p$  norm or Hamming distance), sampling number  $T$ , regularization coefficient  $\lambda_1$ .
Output: interpretation models  $g_{i}$  with coefficients  $\alpha_{i}\in \mathbb{R}^{|C|}, i = 1,2,3,\dots,k.$
1: Fix random bases  $b_{i}, i = 1,2,3,\dots,k$  as anchors.
2: Randomly sample  $T$  bases  $b_{1}, b_{2}, \ldots, b_{T}$  and accordingly, we calculate  $f(x_{1}), f(x_{2}), \ldots, f(x_{T})$ .
3: Assign each basis to the anchor with minimal distance and index using  $d_{i} = \min \{\arg \min_{j}d(b_{i},b_{j}^{x})\} ,i = 1,2,3,\ldots T$  and  $j = 1,2,3,\dots,k$ .
4: for  $i = 1,2,3,\dots,\hat{k}$  do
5: Solve the following regularized regression problem: