SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sample based Explanations via Generalized Representatives
Che-Ping Tsai
chepingt@andrew.cmu.edu
Google Deepmind *
jason6582@gmail.com
We propose a general class of sample based explanations of machine learning models, which we term generalized representatives. To measure the effect of a training sample on a model's test prediction, generalized representatives use two components: a global sample importance that quantifies the importance of the training point to the model and is invariant to test samples, and a local sample importance that measures similarity between the training sample and the test point with a kernel. A key contribution of the paper is to show that generalized representatives are the only class of sample based explanations satisfying a natural set of axiomatic properties. We discuss approaches to extract global importances given a kernel, and also natural choices of kernels given modern non-linear models. As we show, many popular existing sample based explanations could be cast as generalized representatives with particular choices of kernels and approaches to extract global importances. Additionally, we conduct empirical comparisons of different generalized representatives on two image and two text classification datasets.
As machine learning becomes increasingly integrated into various aspects of human life, the demand for understanding, interpreting, and explaining the decisions made by complex AI and machine learning models has grown. Consequently, numerous approaches have been proposed in the field of Explainable AI (XAI). Feature based explanations interpret models by identifying the most relevant input features [1-4], while sample based explanations do so via the most relevant training samples [5-8]. Although different methods emphasize different aspects of the model, some may even have conflicting philosophies [9]. To address this issue, there have been growing calls within the XAI community for more objective or normative approaches [10-12], which could help align XAI techniques more effectively with human needs.
One of the most straightforward approaches to assess the effectiveness of explanations is by evaluating their utility in downstream applications [13, 14]. However, such evaluations can be costly, particularly during the development stages of explanations, as they often necessitate the involvement of real human users. As a result, a well-grounded, axiom-based evaluation can be beneficial for designing and selecting explanations for implementation. Axioms can be viewed as theoretical constraints that dictate how explanations should behave in response to specific inputs. A notable example is the Shapley value [15], which originated in cooperative game theory and has gained popularity in XAI due to its appealing axiomatic properties. Nonetheless, while axiomatic approaches have been widely applied in identifying significant features [4, 16, 17], feature interactions [18, 19], and high-level concepts [20], they have not been extensively discussed in sample based explanations.
*Work done in Carnegie Mellon University.
In this work, we propose a set of desirable axioms for sample based explanations. We then show that any sample based attributions that satisfy (a subset of) these axioms are necessarily the product of two components: a global sample importance, and a local sample importance that is a kernel similarity between a training sample and the test point. We term the explanations in this form generalized representatives.
We note that the efficiency axiom (detailed in the sequel) can only be satisfied if the model function lies in the RKHS subspace spanned by the training kernel representatives, which is indeed typically the case. Otherwise, we could ask for the smallest error in satisfying the efficiency axiom which can be cast as an RKHS regression problem. Thus, given a kernel, extracting global importances can be cast as solving an RKHS regression problem, by recourse to RKHS representative theorems [21]. We additionally also propose tracking representatives that scalably compute the global importance by tracking kernel gradient descent trajectories.
The class of generalized representatives allow for the user to specify a natural kernel given the model they wish to explain, perhaps drawn from domain knowledge. We discuss some natural automated choices given modern non-linear models. Specifically, we discuss the kernel with feature maps specified by last-layer embeddings, neural tangent kernels [22], and influence function kernels [5]. Many existing sample-based explanation methods such as representer point selection [7], and influence functions [5] can be viewed as specific instances of the broad class of generalized representatives. As we show, TracIn [6] could be viewed as a natural extension of generalized representatives that uses multiple kernels, and computes multiple corresponding global and local importances. We empirically compare different choices of generalized representatives for neural networks on two image and two text classification datasets.
Axiomatic attribution in XAI: The most common line of work that incorporates axioms in the design of explanations is the family of Shapley values [15], that originates from cooperative game theory. This line of work first tackles the attribution problem by converting it into a set value function and then applying the Shapley value. The Shapley value is widely used in feature-based explanations [4, 16, 23], feature-interaction explanations [18, 19], concept-based explanations [20] and sample-based explanations [8].
We note that the axiomatic framework for sample-based explanations in data Shapley [8] has distinct purposes and interpretations to ours. Generalized representatives assess the significance of training samples with respect to a specific test sample's prediction. In contrast, data Shapley assesses training sample importance via training loss and can be directly adapted to original Shapley value axioms. Consequently, while there are shared axiomatic principles between the two frameworks, the generalized representatives require a different approach due to their additional focus on a specific test sample. We will delve into a detailed comparison of these distinctions in Section 3.
Sample based explanations: Existing sample-based explanation approaches can be separated into retraining-based and gradient-based approaches [24]. Retraining-based approaches are based on the measurement of the difference between a model prediction with and without a group of training samples [8, 25-34]. Gradient-based methods estimate data influence based on similarities between gradients. The majority of methods build upon the three theoretical frameworks, namely: (1) Representer theorems [7, 35-37], (2) Hessian-based influence functions [5, 38-45], and (3) Decomposing the training loss trajectory [6, 46-48]. In this work, we show that most gradient-based methods can be viewed as generalized representatives.
We consider the task of explaining a supervised machine learning model  $f: \mathbb{R}^d \mapsto \mathbb{R}$ , given inputs  $\mathbf{x} \in \mathbb{R}^d$ , where  $d$  is the input dimension. We are interested in explaining such models  $f(\cdot)$  in terms of the training points  $\mathcal{D} := \{(\mathbf{x}_i, y_i)\}_{i=1}^n$  with each training sample  $(\mathbf{x}_i, y_i) \in \mathbb{R}^d \times \mathbb{R}$ . We denote a sample explanation functional  $\phi_{\mathcal{D}}: \mathcal{F} \times \mathcal{D} \times \mathbb{R}^d \mapsto \mathbb{R}^n$ , as a mapping that takes a real-valued
Note that we assume a single-dimensional output for notational convenience. In the appendix, we show that our development can be extended to vector-valued outputs by using vector-output kernel functions.
model  $f \in \mathcal{F}$ , training data points  $\mathcal{D}$ , and an arbitrary test data point  $\mathbf{x} \in \mathbb{R}^d$  as input, and outputs a vector of explanation weights  $[\phi_{\mathcal{D}}(f, (\mathbf{x}_1, y_1), \mathbf{x}), \dots, \phi_{\mathcal{D}}(f, (\mathbf{x}_n, y_n), \mathbf{x})] \in \mathbb{R}^n$  with each value corresponding to an importance score of each training sample to the test point. In the sequel, we will suppress the explicit dependence on the entire set of training points in the notation for the explanation functional and the dependence on the training label  $y_i$ . Also, to make clear that the first data point argument is the training sample, and the second is the test sample, we will use  $\phi(f, \mathbf{x}_i \to \mathbf{x}) \in \mathbb{R}$  to denote the sample explanation weight for  $\mathbf{x}_i$  to explain the prediction of the model  $f(\cdot)$  for the test point  $\mathbf{x}$ .
3 Axioms for Sample based Explanations
In this section, we begin by presenting a collection of axioms that describe various desirable characteristics of sample based explanations.
Definition 1 (Efficiency Axiom). For any model  $f$ , and test point  $x \in \mathbb{R}^d$ , a sample based explanation  $\phi(\cdot)$  satisfies the efficiency axiom iff:
$$
\sum_ {i = 1} ^ {n} \phi (f, \mathbf {x} _ {i} \rightarrow \mathbf {x}) = f (\mathbf {x}).
$$
The efficiency axiom entails that the sum of the attributions to each training sample together adds up to the model prediction at the test point. This is a natural counterpart of the efficiency axioms used in the Shapley values [49].
Definition 2 (Self-Explanation Axiom). A sample based explanation  $\phi(\cdot)$  satisfies the self-explanation axiom iff there exists any training point  $\mathbf{x}_i$  having no effect on itself, i.e.  $\phi(f, \mathbf{x}_i \to \mathbf{x}_i) = 0$ , the training point should not impact any other points, i.e.  $\phi(f, \mathbf{x}_i \to \mathbf{x}) = 0$  for all  $\mathbf{x} \in \mathbb{R}^d$ .
The self-explanation axiom states that if the label  $y_{i}$  does not even have an impact on the model's prediction for  $\mathbf{x}_i$ , it should not impact other test predictions. This axiom shares a similar intuition as the dummy axiom in the Shapley values [15] since both axioms dictate that explanations should be zero if a training sample has no impact on the model. However, the self-explanation axiom requires a different theoretical treatments due to the additional focus in generalized representatives of explaining a model prediction on a particular test sample.
Definition 3 (Symmetric Zero Axiom). A sample explanation  $\phi(\cdot)$  satisfies the symmetric zero axiom iff any two training points  $\mathbf{x}_i, \mathbf{x}_j$  such that if  $\phi(f, \mathbf{x}_i \to \mathbf{x}_i) \neq 0$  and  $\phi(f, \mathbf{x}_j \to \mathbf{x}_j) \neq 0$ , then
$$
\phi (f, \mathbf {x} _ {i} \rightarrow \mathbf {x} _ {j}) = 0 \Longrightarrow \phi (f, \mathbf {x} _ {j} \rightarrow \mathbf {x} _ {i}) = 0.
$$
The symmetric-zero axiom underscores the bidirectional nature of "orthogonality". It emphasizes that if a sample has no impact on another sample, this lack of correlation is mutual and implies that they are orthogonal.
Definition 4 (Symmetric Cycle Axiom). A sample explanation  $\phi(\cdot)$  satisfies the symmetric cycle axiom iff for any set of training points  $\mathbf{x}_{t_1}, \ldots, \mathbf{x}_{t_k}$ , with possible duplicates, and  $\mathbf{x}_{t_{k+1}} = \mathbf{x}_{t_1}$ , it holds that:
$$
\prod_ {i = 1} ^ {k} \phi (f, \mathbf {x} _ {t _ {i}} \rightarrow \mathbf {x} _ {t _ {i + 1}}) = \prod_ {i = 1} ^ {k} \phi (f, \mathbf {x} _ {t _ {i + 1}} \rightarrow \mathbf {x} _ {t _ {i}}).
$$
Let us first consider the vacuous case of two points:  $\mathbf{x}_1, \mathbf{x}_2$ , for which the axiom is the tautology that:  $\phi(f, \mathbf{x}_1 \to \mathbf{x}_2) \phi(f, \mathbf{x}_2 \to \mathbf{x}_1) = \phi(f, \mathbf{x}_2 \to \mathbf{x}_1) \phi(f, \mathbf{x}_1 \to \mathbf{x}_2)$ . Let us next look at the case with three points:  $\mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_3$ , for which the axiom entails:
$$
\phi (f, \mathbf {x} _ {1} \to \mathbf {x} _ {2}) \phi (f, \mathbf {x} _ {2} \to \mathbf {x} _ {3}) \phi (f, \mathbf {x} _ {3} \to \mathbf {x} _ {1}) = \phi (f, \mathbf {x} _ {3} \to \mathbf {x} _ {2}) \phi (f, \mathbf {x} _ {2} \to \mathbf {x} _ {1}) \phi (f, \mathbf {x} _ {1} \to \mathbf {x} _ {3}).
$$
It can be seen that this is a generalization of simply requiring that the explanations be symmetric as in the symmetry axiom in the Shapley values. In fact, the unique explanation satisfying this and other listed axioms is in general not symmetric. The axiom could also be viewed as a conservation or path independence law, in that the flow of explanation based information from a point  $\mathbf{x}_i$  to itself in a cycle is invariant to the path taken.
Definition 5 (Continuity Axiom). A sample based explanation  $\phi(\cdot)$  satisfies the continuity axiom iff it is continuous wrt the test data point  $\mathbf{x}$ , for any fixed training point  $\mathbf{x}_i$ :
$$
\lim _ {\mathbf {x} ^ {\prime} \to \mathbf {x}} \phi (f, \mathbf {x} _ {i} \to \mathbf {x} ^ {\prime}) = \phi (f, \mathbf {x} _ {i} \to \mathbf {x}).
$$
Such continuity is a minimal requirement on the regularity of the explanation functional, and which ensures that infinitesimal changes to the test point would not incur large changes to the explanation functional.
Definition 6 (Irreducibility Axiom). A sample explanation  $\phi(\cdot)$  satisfies the irreducibility axiom iff for any number of training points  $\mathbf{x}_1, \ldots, \mathbf{x}_k$ ,
$$
d e t \left( \begin{array}{c c c c} \phi (f, \mathbf {x} _ {1}, \mathbf {x} _ {1}) & \phi (f, \mathbf {x} _ {1}, \mathbf {x} _ {2}) & ... & \phi (f, \mathbf {x} _ {1}, \mathbf {x} _ {k}) \\ \phi (f, \mathbf {x} _ {2}, \mathbf {x} _ {1}) & \phi (f, \mathbf {x} _ {2}, \mathbf {x} _ {2}) & ... & \phi (f, \mathbf {x} _ {2}, \mathbf {x} _ {k}) \\ ... & ... & ... & ... \\ \phi (f, \mathbf {x} _ {k}, \mathbf {x} _ {1}) & \phi (f, \mathbf {x} _ {k}, \mathbf {x} _ {2}) & ... & \phi (f, \mathbf {x} _ {k}, \mathbf {x} _ {k}) \end{array} \right) \geq 0.
$$
A sufficient condition for an explanation  $\phi (\cdot)$  to satisfy the irreducibility axiom is for
$$
\left| \phi \left(f, \mathbf {x} _ {i} \rightarrow \mathbf {x} _ {i}\right)\right| > \sum_ {j \neq i} \left| \phi \left(f, \mathbf {x} _ {i} \rightarrow \mathbf {x} _ {j}\right)\right|, \tag {1}
$$
since this makes the matrix above strictly diagonally dominant, and since the diagonal entries are non-negative, by the Gershgorin circle theorem, the eigenvalues are all non-negative as well, so that the determinant in turn is non-negative.
The continuity and irreducibility axiom primarily serves a function-analytic purpose by providing sufficient and necessary conditions of a kernel being a Mercer kernel, which requires that the kernel function be continuous and positive semi-definite.
We are now ready to investigate the class of explanations that satisfy the axioms introduced above.
Theorem 7. An explanation functional  $\phi(f, \cdot, \cdot)$  satisfies the continuity, self-explanation, symmetric zero, symmetric cycle, and irreducibility axioms for any training samples  $\mathcal{D}$  containing  $n$  training samples  $(\mathbf{x}_i, y_i) \in \mathbb{R}^d \times \mathbb{R}$  for all  $i \in [n]$  iff
$$
\phi (f, \mathbf {x} _ {i} \rightarrow \mathbf {x}) = \alpha_ {i} K (\mathbf {x} _ {i}, \mathbf {x}) \quad \forall i \in [ n ], \tag {2}
$$
for some  $\alpha \in \mathbb{R}^n$  and some continuous positive-definite kernel  $K:\mathbb{R}^d\times \mathbb{R}^d\mapsto \mathbb{R}$
This suggests that a sample explanation  $\phi(f, \mathbf{x}_i \to \mathbf{x}) = \alpha_i K(\mathbf{x}_i, \mathbf{x})$  has two components: a weight  $\alpha_i$  associated with just the training point  $\mathbf{x}_i$  independent of test points, and a similarity  $K(\mathbf{x}_i, \mathbf{x})$  between the training and test points specified by a Mercer kernel. Following Yeh et al. [7], we term the first component the global importance of the training sample  $\mathbf{x}_i$  and the second component the local importance that measures similarities between training and test samples.
Once we couple this observation together with the efficiency axiom, one explanation that satisfies these properties is:
$$
f (\mathbf {x}) = \sum_ {j = 1} ^ {n} \phi (f, \mathbf {x} _ {i} \rightarrow \mathbf {x}) = \sum_ {j = 1} ^ {n} \alpha_ {i} K (\mathbf {x} _ {i}, \mathbf {x}), \text {f o r a n y} x \in \mathbb {R} ^ {p}. \tag {3}
$$
This can be seen to hold only if the target function  $f$  lies in the RKHS subspace spanned by the kernel evaluations of training points. When this is not necessarily the case, then the efficiency axiom (where the sum of training sample importances equals the function value) exactly, cannot be satisfied exactly. We can however satisfy the efficiency axiom approximately with the approximation error arising from projecting the target function  $f$  onto the RKHS subspace spanned by training representatives.
This thus provides a very simple and natural framework for specifying sample explanations: (1) specify a Mercer kernel  $K(\cdot, \cdot)$  so that the target function can be well approximated by the corresponding kernel machine, and (2) project the given model onto the RKHS subspace spanned by kernel evaluations on the training points. Each of the sample explanation weights then has a natural specification in terms of global importance associated with each training point (arising from the projection of the function onto the RKHS subspace, which naturally does not depend on any test points), as well as a localized component that is precisely the kernel similarity between the training and test points.
4 Deriving Global Importance Given Kernel Functions
The previous section showed that the class of sample explanations that satisfy a set of key axioms naturally correspond to an RKHS subspace. Thus, all one needs, in order to specify the sample explanations, is to specify a Mercer kernel function  $K$  and solve for the corresponding global importance weights  $\alpha$ . In this section, we focus on the latter problem, and present three methods to compute the global importance weights given some kernel  $K$ .
4.1 Method 1: Projecting Target Function onto RKHS Subspace
The first method is to project the target function onto the RKHS subspace spanned by kernel evaluations on the training points. Given the target function  $f$ , loss function  $\mathcal{L} : \mathbb{R} \times \mathbb{R} \mapsto \mathbb{R}$  and training dataset  $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^n$  (potentially, though not necessarily used to train  $f$ ), and a user-specified Mercer kernel  $K$ , our goal is to find a projection  $\hat{f}_K$  of the target model onto the RKHS subspace defined by  $\mathcal{H}_n = \operatorname{span}(\{K(\mathbf{x}_i, \cdot)\}_{i=1}^n)$ . To accomplish this, we formulate it as a RKHS regression problem:
$$
\hat {f} _ {K} = \underset {f _ {K} \in \mathcal {H} _ {K}} {\operatorname {a r g m i n}} \left\{\frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} \left(f _ {K} \left(\mathbf {x} _ {i}\right), f \left(\mathbf {x} _ {i}\right)\right) + \frac {\lambda}{2} \| f _ {K} \| _ {\mathcal {H} _ {K}} ^ {2} \right\}, \tag {4}
$$
where  $\mathcal{H}_K$  as the RKHS defined by kernel  $K$ ,  $\| \cdot \|_{\mathcal{H}_K} : \mathcal{H}_K \mapsto \mathbb{R}$  is the RKHS norm, and  $\lambda$  is a regularization parameter that controls the faithfulness and complexity of the function  $\hat{f}_K$ . The loss function  $\mathcal{L}$  can be chosen as the objective function used to train the target function  $f$  to closely emulate the behavior of target function  $f$  and its dependence on the training samples  $\mathcal{D}$ . By the representer theorem [21], the regularization term  $\| f_K\|_{\mathcal{H}_K}^2$  added here ensures that the solution lies in the RKHS subspace  $\mathcal{H}_n$  spanned by kernel evaluations. Indeed, by the representer theorem [21], the minimizer of Eqn.(4) can be represented as  $\hat{f}_K(\cdot) = \sum_{i=1}^{n} \alpha_i K(\mathbf{x}_i, \cdot)$  for some  $\alpha \in \mathbb{R}^n$ , which allows us to reparameterize Eqn.(4):
$$
\hat {\alpha} = \underset {\alpha \in \mathbb {R} ^ {n}} {\operatorname {a r g m i n}} \left\{\frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} \left(\sum_ {j = 1} ^ {n} \alpha_ {j} K \left(\mathbf {x} _ {i}, \mathbf {x} _ {j}\right), f \left(\mathbf {x} _ {i}\right)\right) + \frac {\lambda}{2} \alpha^ {\top} \mathbf {K} \alpha \right\}, \tag {5}
$$
where  $\mathbf{K} \in \mathbb{R}^{n \times n}$  is the kernel gram matrix defined as  $K_{ij} = K(\mathbf{x}_i, \mathbf{x}_j)$  for  $i, j \in [n]$ , and we use the fact that  $\|f_K\|_{\mathcal{H}_K} = \langle \sum_{i=1}^n \alpha_i K(\mathbf{x}_i, \cdot), \sum_{i=1}^n \alpha_i K(\cdot, \mathbf{x}_i) \rangle^{\frac{1}{2}} = \sqrt{\alpha^\top \mathbf{K} \alpha}$ . By solving the first-order optimality condition, the global importance  $\alpha$  must be in the following form:
Proposition 8. (Surrogate derivative) The minimizer of Eqn.(4) can be represented as  $\hat{f}_K = \sum_{i=1}^{n} \hat{\alpha}_i K(\mathbf{x}_i, \cdot)$ , where
$$
\hat {\alpha} \in \left\{\alpha^ {*} + v \mid v \in n u l l (\mathbf {K}) \right\} a n d \alpha_ {i} ^ {*} = - \frac {1}{n \lambda} \frac {\partial \mathcal {L} \left(\hat {f} _ {K} \left(\mathbf {x} _ {i}\right) , f \left(\mathbf {x} _ {i}\right)\right)}{\partial \hat {f} _ {K} \left(\mathbf {x} _ {i}\right)}, \forall i \in [ n ]. \tag {6}
$$
We call  $\alpha_{i}^{*}$  the surrogate derivative since it is the derivative of the loss function with respect to the surrogate function prediction.
$\alpha_{i}^{*}$  can be interpreted as the measure of how sensitive  $\hat{f}_K(\mathbf{x}_i)$  is to changes in the loss function. Although the global importance  $\alpha$  solved via Eqn.(5) may not be unique as indicated by the above results, the following proposition ensures that all  $\hat{\alpha} \in \{\alpha^* + v \mid v \in \mathrm{null}(\mathbf{K})\}$  result in the same surrogate function  $\hat{f}_K = \sum_{i=1}^{n} \alpha_i^* K(\mathbf{x}_i, \cdot)$ .
Proposition 9. For any  $v \in \mathrm{null}(\mathbf{K})$ , the function  $f_v = \sum_{i=1}^n v_i K(\mathbf{x}_i, \cdot)$  specified by the span of kernel evaluations with weights  $v$  is a zero fucntion, such that  $f_v(\mathbf{x}) = 0$  for all  $\mathbf{x} \in \mathbb{R}^d$ .
The proposition posits that adding any  $v \in \mathrm{null}(\mathbf{K})$  to  $\alpha^*$  has no effect on the function  $\hat{f}_K$ . Therefore, we use  $\alpha^*$  to represent the global importance as it captures the sensitivity of the loss function to the prediction of the surrogate function.
4.2 Method 2: Approximation Using the Target Function
Given the derivation of global importance weights  $\alpha^{*}$  in Eqn.(6), we next consider a variant replacing the surrogate function prediction  $\hat{f}_K(\mathbf{x}_i)$  with the target function prediction  $f(\mathbf{x}_i)$ :
Definition 10 (Target derivative). The global importance computed with derivatives of the loss function with respect to the target function prediction is defined as:
$$
\alpha_ {i} ^ {*} = - \frac {\partial \mathcal {L} (f (x _ {i}) , y _ {i})}{\partial f (x _ {i})}, \forall i \in [ n ], \tag {7}
$$
where  $\mathcal{L}(\cdot ,\cdot)$  is the loss function used to train the target function.
A crucial advantage of this variant is that we no longer need solve for an RKHS regression problem. There are several reasons why this approximation is reasonable. Firstly, the loss function in Eqn.(4) encourages the surrogate function to produce similar outputs as the target function, so that  $\hat{f}_K(\mathbf{x}_i)$  is approximately equal to  $f(x_{i})$ . Secondly, when the target function exhibits low training error, which is often the case for overparameterized neural networks that are typically in an interpolation regime, we can assume that  $f(x_{i})$  is close to  $y_{i}$ . Consequently, the target derivative can serve as an approximation of the surrogate derivative in Eqn.(6). As we will show below, the influence function approach [5] is indeed as the product between the target derivative and the influence function kernel.
4.3 Method 3: Tracking Gradient Descent Trajectories