Data-centric Prediction Explanation via Kernelized Stein Discrepancy
Authors: Mahtab Sarvmaili
Hassan Sajjad
Ga Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Prediction Explanation
Kernelized Stein Discrepency
Primary area: interpretability and explainable AI

Abstract: Existing example-based prediction explanation methods often bridge test and training data points through the model’s parameters or latent representations. While these methods offer clues to the causes of model predictions, they often exhibit innate shortcomings, such as incurring significant computational overhead or producing coarse-grained explanations. This paper presents a Highly-precise and Data-centric Explanation (HD-Explain) prediction explanation method that exploits properties of Kernelized Stein Discrepancy (KSD). Specifically, the KSD uniquely defines a parameterized kernel function for a trained model that encodes model-dependent data correlation. By leveraging the kernel function, one can identify training samples that provide the best predictive support to a test point efficiently. We conducted thorough analyses and experiments across multiple classification domains, where we show that HD-Explain outperforms existing methods from various aspects, including 1) preciseness (fine-grained explanation), 2) consistency, and 3) computation efficiency, leading to a surprisingly simple, effective, and robust prediction explanation solution.

Published as a conference paper at ICLR 2025
DATA- PREDICTION EXPLANATION VIA KERNELIZED STEIN DISCREPANCY
Mahtab Sarvmaili, Hassan Sajjad, Ga Wu Department of Computer Science Dalhousie University
(mahtab. sar sarvmailo i 11, hsa j jjad gu. @dal.ca
ABSTRACT
Existing example-based prediction explanation methods often bridge test and training data points through the model's parameters latent representations. While these methods offer clues to the causes of model predictions, they often exhibit innate shortcomings, such as incurring significant computational overhead or producing coarse-grained explanations. This paper presents Eighly-precist and Data-centric Explanation (HD -Xpplainn prediction explanation method that exploits properties of Kernelized Stein Discrepancy (KSD). Specifically, the KSD uniquely defines a parameterized kernel function for a trained model that encodes model-dependent data correlation. By leveraging the kernel function, identify training samples that provide the best predictive support to a test point efficiently. We conducted thorough analyses and experiments across multiple classification domains, where we show that HD-Explain outperforms existing methods from various aspects, including 1) preciseness (fine-graine eeeeeggaindd explanation), 2) consistency, and 3) computation efficiency, leading to a surprisingly simple, effective, and robust prediction explanation solution. Source code is available at https: : //t::://gttttaa com http wwt.aaaaaaaaaiiaaa.aalaaa lain
INTRODUCTION
As one of the decisive factors affecting the performance of a Machine Learning (ML) model, training data points are of great value in promoting the model's transparency and trustworthiness, including explaining prediction results, tracing sources of errors, summarizing the characteristics of the model (Cai et al. 2019 Anik & Bunt 2021 Nam et al. 2022). The challenges example-based prediction explanation mainly come from retrieving relevant data points from a vast pool of training samples or justifying the rationale such explanations (Lim et al. 2019 Zhou et al. 2021
Modern example-base based prediction explanation methods commonly approach the above challenges by constructing an influence chain between training and test data points (Li ct al. 2020 Nam et al. model
2022 Tsai et al. 2023). The influence chain could be either data points' co-intluence parameters or their similarity in terms of latent representations. In particular, Influence Function (Koh) Liang 2017), one of the representative, model-aware explanation methods, looks for the shift of the model parameters (due to up-weighting each training sample) as the sample's 's influence score. Since computing the inverse Hessian matrix is challenging, the approach adapts Conjugate Gradients Stochastic Estimation and the Perlmutter trick to reduce its computation cost. Representer Point Selection (RPS) YYhh et al. 2018). as another example, reproduces the representer theorem by refining the trained neural network model with 7.2 regularization, such that the influence score of each training sample can represented as the gradient of the predictive layer. While computationally efficient, RPS is criticized for producing coarse grained explanations that more class-level rather than instance-level explanations (Sui et al., (022)) (In this paper we use the instance level explanation and example-based explanation interchangeably.) Multiple later variants (Pruthi al. 20200 Sui et al. 2021) attempted to mitigate the drawbacks above, but their improvements were often limited by the cause of their shared theoretical scalability bounds.
This paper presents Highly-precist and Data-centric Explanation (HD-Explain), post-hoc, model- aware, example-based explanation solution for neural classifiers. Instead of relying data co-

---

Published as a conference paper at ICLR 2025
Table 1: Summary of existing Post-hoc Example-based Prediction Explanation Methods that work with deep neural networks. Practicality of the whole model explanation is measured by the feasibility of explaining the prediction of ResNet-18 18 trained on CIFAR-10 10 with single A100 GPU machine. CIFAR- 10 is small benchmark data with 50000 training samples.
Need optimizationn
Whole model explanation
Inference computation
Memorrycache (of training sample) bounded by
Method
Explanation of
ss sub--nutinee
Theoretical
Proctical
complesiry bounded by
Influence Function
(terative HVP
1.fg Vol(x, 8) approximationn
Original Model
upproximation)
Yes
No
< VoLla, 0), H6 Vel(x,, >
Size of model parameters
Yes (1.2 regularized last layer retrain)
Llast layer representationn f.
Size of model paramesers of the last layer
RPS
Fine-tuned Model
No
No
2 a.f.
1. 9) apprxximation 2. < VoLix, 0i, VeL(x. 0)
Trackn*
Original Model
No
Yes
No
Size of model parameters
I.,,,, 2. Closed-Fomm Fu x,) defined by KSD
HDX-xxploi
Original Model
No
Yes
Yes
Size of data dimension
TracIn typically requires to access the training process. Here, TracIn refers to a special case that only use the
last training checkpoint.
influence on model parameters or feature representation similarity, HD-Explain retains the influence chain between training and test data points by exploiting the underrated properties of Kemeliized Stein Diserepancy (KSD) (Liu et al. 2016) between the trained predictive model and its training dataset. Specifically, we note that the Stein operator augmented kernel uniquely defines a pairwise data correlation (in the context of a trained model) whose expectation on the training dataset results in the minimum KSD (as discrete approximation) compared to that of the dataset sampled from different distributions. By exploiting this property, we can 1) reveal a subset of training data points that provides the best predictive support to the test point and 2) identify the potential distribution mismatch among training data points. Jointly leveraging these advantages, HID-Explain can produce explanations faithful to the original trained model.
The contributions of our work are summarized as follows: 1) We propose a novel example- based explanation method. 2) We propose several quantitative evaluation metrics measure the correctness and effectiveness of generated explanations. 3) We perform thorough evaluation comparing several existing explanation methods across wide set of classification tasks.
Our findings conclude that HD-Explain offers fine-grained, instance- explanations with re- markable computational efficiency, compared to well-known example-based prediction explanation methods. Its algorithmic simplicity, effective performance and scalability enables its deployment to real world scenarios where transparency and trustworthiness is essential.
2 PRELIMINARY AND RELATED WORK
2.1 POST-HOC CLASSIFIER EXPLANATION BY EXAMPLES
Post-hoc Classifier Explanation by Examples (a.k.a prototypes, representers) refers category of classifier explanation approaches that pick a subset of training data points as prediction explanations without accessing the model training process. Its research history spans from the inttiniit to the recent impact-based approach (Li et al. 2020).
approach (see survey Molnar (222
Model-inherent approaches Molnar 2020) refer to machine learning models that are considered interpretable such as k-nearest neighbor (Peterson 2009) or decision tree; For given test data point, similar data points on the raw feature space can be efficiently selected explanations through the inherent decision making mechanism of the self-explanatory machine learning models. In fact, attracted by their inherent explanatory power, multiple well-known works attempted to compile complex black-box models into ceff-explanatory models for enabling prediction explanation (Frosst & Hinton 2017). while computationally inefficient.
To unlock the general explanatory power applicable to black-box models, multiple later studies suggested to fall back to statistics-basd based solutions, looking for prototype samples that represent data points that either are common in the dataset play critical roles in data distribution. MMD-critic (Kim ctt al.) 2016) and Normative and Comparative explanations (Cai et al., 2019) are the well-known examples in this category. Unfortunately, those approaches are often with strong assumption of good
2

---

Published as a conference paper at ICLR 2025
prototypes (which often overlook the characteristics of trained models) (Li et al. 2020 , making their prediction explanations general to the training dataset rather than a trained model instance.
Recently, influence-baset methods have emerged the prevailing technique in model explanation (Li ct al. 2020 Nam et al., 2022 Bae et al., 2022 Park et al. 2023a). Influence function (Koh Liang 2017).
of the earliest influence based solutions, bridges the outcome of a prediction task to training data points by, first, evaluating training data's 's influence on the model parameters and, then, estimating how model parameter changes affect prediction. Similarly, Representative Point Selection (RPS) (Yeh et al. 2018) builds such an influence chain by fitting the representation theorem, where the weighted product between the representations of test and training samples comes into play, Concerning the computational overhead of previous work, the later solution TracIn (Pruthi et al.) 2020) proposed a simple approximation of the influence function via a first-order Taylor expansion (essentially Neural Tangent Kernel (Jacot et al. 2018)), successfully discarding the inverse Hessian matrix from the influence chain formulation. BoostIn (Brophy et al.] 2023) further extends TracIn and is dedicated to interpreting the predictions of gradient gradient-boosted decision trees. RPS-LJE (Sui cct al. 202T), on the other hand, alleviated the inconsistent explanation problem of RPS through Local Jacobian Expansion. In the latest publication (Tsai et al. 2023), all of the above methods described in this paragraph are identified as special cases Generalized Representers but with different chosen kernels.
One limitation of the current influence-based methods is that they attribute the influence of each training data point the parameters of the trained model essential intermediate step. Indeed, as the nature of stochastic gradient descent (the dominating training strategy of neural networks), isolating such contribution is barely possible without 1) relying on approximations or 2) accessing the training process, Unfortunately, either solution would result in performance degradation or heavy computational overhead (Schioppa al. (022). Hence, this work delves into the exploration of an alternative influence connection between training and test data points without exploiting the
perturbation of model parameters.
2.2 KERNELIZED STEIN DISCREPANCY
The idea of Kernelized Stein Discrepancy (KSD) (Liu et al. 2016) be traced back to theorem called Stein Identity (Kattumannil 2009) that states, if smooth distribution p(x) and a function Φ(x) satisfy | p(z)o(z) O,
(
- v=o(z)] 0, Vá.
The identity characterize distribution p(æ) such that it is often served to assess the goodness-ol -f-fee (Kubokawa) 2024) of the model. The above expression could be further abstracted to use function operator Ap (a.k.a Stein operator) such that
Apo(z) = ((x) V logp(z) + V2Φ(z),
where the operator encodes distribution p(x) in the form of derivative to input (a.k.a score function).
Stein's identity offers a mechanism measure the gap between two distributions by assuming the variable is sampled from different distribution 4 + p such that
8(g.p) = max R Eez (Ap)))) dEF
where the expression takes the most discriminant & that maximizes the violation of Stein's 's identity to quantify the distribution discrepancy. This discrepancy is, accordingly, referred as Stein Discrepancy.
The challenge of computing Stein Discrepancy comes from the selection of function F, which motivates the later innovation of KSD that takes F to be the unit ball of a reproducing kernel Hilbert space (RKHS). By leveraging the reproducing property of RKHS, the KSD could be eventually transformed into
S(q. p) [e,,''"!/"p(t,,r'' x')]
where Kp (z, x') = A k(1, x') that can work with arbitrary kernel function k( x'). See Ap- pendix for expanded derivations.
In the literature. KSD has been adopted for tackling three types of application tasks 1) parameter inference (Barp et al. 2019), Goodness-of-fr tests (Chwialkowski et al. 2016 Liu al. 2016
3

---

Published as a conference paper at ICLR 2025
Duverra eeree 25
375
230-
Traniing
2
280
-306
300
64
04
03
20
Glss Tukio: 1:9 Lless Ratio: 57
Trairing Data
Class Bate: 7:3 Class Racke Vi1
Degres Rotation
Data Rets ThowWWel
Figure 1: Varying of Kemeliiedd Stein Discrepancy given the shift of training data distribution on Two Moon dataset.
Yang al., 2018), and 3) particle filtering (sampling) (Gorham al., 2020 Korba et al., 2021), However, to the best of our knowledge, its innate property that uniquely defines model-dependen data correlation has never been exploited, which, we note, is valuable to interpret model behaviour from various aspects, including instance-level prediction explanation and global prototypical explanations.
3 HIGHLYY-PRECISE AND DATA-CENTRIC EXPLANATION
HD-Explain is an example-based prediction explanation method based on Kernelized Stein Discrep- ancy. Consider trained classifier as the outcome of training process with Maximum Likelihood Estimation (MLE)
argmax ((x Po (y(xx)]
Theoretically, maximizing observation likelihood is equivalent to minimizing KI. divergence between data distribution Pp and the parameterized distribution Po such that
PD (x(x))) v)
DRL. (Pn, Po) = (xx.)>>
log Po(x,
= --B(x.)) [log (x()x
+ (x. [log Pb(y(x)] + ((xx)) llog Pxx
=
Likelihood
constant
inputs
which, in turn, is proven align with minimizing KSD in the form of gradient descent (Liu & Wang 2016)
VKD (PD,(PP, PP)le SSPP,,
where e is the step size of gradient decent.
The chain of reasoning above shows that a well-trained classifier fo through gradient should lead to minimum discrepancy between the training dataset distribution and the model encoded distribution S( (Pp. Pa) We empirically verify the connection through simple examples as shown in Figure D where the changes in training data distribution would result in larger KSD compared to that of the original training data distribution. Intuitively, the connection shows that there is tie between model and its training data points, encoded in the form of Stein kernel function k- defined on each pair of data points. As the kernel function is conditioned on model fe. we note is encoding of data correlation under the context of trained model, which paves the foundation of the example-bar prediction explanation.
3.1 KSD BETWEEN MODEL AND TRAINING DATA
Recall that KSD, S(Pp, Pa). defines the correlation between pairs of training samples through model 9 dependent kernel function with closed-form decomposition
Ko (x( Va). (xs, yb)) = A3 b)
V. V.k(a, 6) k(a, b) log (o) log (b)
+ Vak(a, b) (b(b) - Vak(a, V log Po(a).
where we denote data point (x,, ya) with for clean notation. The only model-dependent factor in the above decomposition is derivative Vx log Po P y) (for both data or b).
Since Pp is discrete distribution while Pg is continuous, the Discrepancy between the two distributions will recap Stein Identity (= with limited number of training data points.
4

---

Published as a conference paper at ICLR 2025
However, the KSD only models the discrepancy between joint distributions rather than conditional distributions, it is challenging to estimate discrepancy between predictive models Po(y|x) and its training set (x.y) y) PD without including (x(x) into consideration, even though the marginal distribution Po (x) is not estimated by the predictive model at all.
Inspired by the previous study on Goodness of Fit (Jitkrittum al. 2020), unlock KSD support on predictive models, we propose to set (x(x) = (x(x) such that the identical marginal distribution would not contribute to the discrepancy between the joint distributions PD(x, y) and (x In addition, while original distribution P(x) that generates data could be arbitrary complex distribution (that is out of modelling scope of the predictive model), we may simply set data point distribution PD (not original distribution P) as an Uniform distribution over data points in the dataset under the particular context, given a data point is sampled from generated dataset uniformly. Although the relaxation appears hasty, we believe that the relaxation is valid in the prediction explanation context (and probably only valid in this particular context) in terms of measuring correlation between a pair of data points, where all train/test data points' input valid observation (e.g. images) rather than random continuous valued sample the same space form an unknown distribution.
With the above relaxations, the score function Xxyy log (x, u) in the Stein operator Ag could be derived as a concatenation of the gradient of model fo(x), to its input x and its probabilistic prediction fe (x) in logarithm form, since
V x.y log y) =
Vx.yllog Po(y + log PD(x)]
Xxy log Po x) + V log Po (x) (xb)))
=[Vxy' log fo V log [x)] + [
=IV log fx(x), log fe(x)]
where [I]-] denotes concatenation operation. We use one-hot vector representation to represent data label y here. Since (x) follows uniform distribution, gradient the inputs is 0 vector.
In the above derivation, we treat discrete label y as- without specialized discrete distribution treatments (see (Yang et al., 2018)) to avoid significant computation overhead. In fact, data space in practice is unlikely dense even if group of features are continuous (e.g. images). Treating the label as a sparse continuous feature can be viewed as an approximation.
Combining Equation]] I and 2
estimate the correlation of any pairs of training data points conditioned on the trained machine learning model. Computationally, since a score function single data point, its outputs of the training set could be pre-computed
Vx,y log (x, y) depends and cached to accelerate the kernel computation. In particular, the output dimension of the score function is simply m k for data with dimensional features and k class labels. Compared the existing solutions, whose training data cache (or influence) are bounded by the dimension of model parameters (such Influence function, TracIn, RPS, RPS-JL.),, the explanation method built KSD would come with a significant advantage in terms of scalability (see comparison in Table []. This statement is generally true for neural network based classifiers, whenever the size of model parameters is far larger than the data dimension.
3.2 PREDICTION EXPLANATION
The computation of kernel function Equation T requires access to features and labels of data point. While the ground-trudt label information is available for the training set, it is inaccessible for test data point. We consider the predicted class ÿ, of the test data point x; as a label to construct a complete data point (x,, 7i) and apply the KSD kernel function. For test data point x,, we search for top-k training data points that maximize the KSD defined kernel.
Figure 2 demonstrates HD-Explain on 2d synthetic dataset. The distribution of Ne (d, in the right most plot shows that only a small number of training data points have a strong influence on a particular prediction.
4 EVALUATION AND ANALYSIS
In this section, we conduct several qualitative and quantitative experiments to demonstrate various properties of HDD-Explain and compare with the existing example-based solutions.
5

---

Published as a conference paper at ICLR 2025
Week Prediction Support
2
Histogram
200
Strong Prodiction Support
K2
Figure 2: Demonstration of HD-Explain on 2D Rectangular synthetic dataset. Left shows the training dataset with three classes. Middle figure shows the explanation support of training data points given test point (as black cross), where green shows higher KSD kernel value. Right shows the distribution of KSD kernel values (over the training set) the test point, where only small number of training data points provide strong support to this prediction.
Datasets: We consider multiple disease classification tasks where diagnosis explanation is highly desired. We also introduced synthetic and benchmark classification datasets deliver the main idea without the need for medical background knowledge. Concretely, we use CIFAR-10 (32 x 32 x 3), Brain Tumor (Magnetic Resonance Imaging. 128 128 3), Ovarian Cancer (Histopathology Images, 128 x 128 x 3) datasets, and SVHN (32 x 32 x 3). More details are listed in the Appendix
The baseline explainers used in experiments include Influence Function, Representer Point Selection, and TracIn. While other variants of these baseline explainers exist (Barshan et al.] 2020 Sui et al. 2021: Chen et al.) 202I), we note they don't offer fundamental performance improvements over the classic ones. In addition, as Influence Function and TracIn face scalability issues, we limit the influence of parameters to the last layer of the model so that they can work with models that contain large number of parameters. Our experiments use ResNet-18 18 as the backbone model architecture (with around 11 million trainable parameters) for all image datasets (see Appendix H for detail on our hardware setup).
Baselines:
Finally, we also introduce an HD-Explain variant (HD-Expllan*)) to match the last layer setting of other baseline models, even though HD-Explain can scale up to the whole model without computation pressure. The HD-Explain* a simple change of HD-Explain in terms of using data representations (the output of the last non-predictive layer of the neural classifier) rather than the raw features. Specifically, we assume a neural network model fe could be decomposed into two components fer fe,, where fe, is representation encoder and fe, is linear model for prediction. With this decomposition, we define the KSD) kernel function for HD-Explain** as
Ko((Fe, (xa), ya), (Se, (xx), yb))
= Ab, Ab, k(a,b)
= VaVak(a, b) + k(a (o) log Po. (b)
b) log Po, (b) Vok(a, log Po, (a),
where we define (So, (xx). va) and b (So, (x.), yb) for short. This setting reduces the prediction explanation to the last layer of the ncural network in a similar fashion to RPS.
Metrics: In existing example-based explanation works, the experimental results often demon- strated qualitatively, as visualized explanation instances, without quantitative evaluation. This results in subjective evaluation. In this paper, propose several quantitative evaluation metrics to measure the effectiveness of each method.
Hit rate measures how likely an explanation sample hits the desired example cases where the desired examples are guaranteed be undisputed. Specifically, we modify a training data point with minor augmentations (adding noise or flipping horizontally) and use it test data point, such that the best explanation for the generated test data point should be the original data point in the training set.
Hit Rate:
Coverage: Given n test data points, the metric measures the number of unique explanation samples explanation method produces when configuring to return top-k training samples.

---

Published as a conference paper at ICLR 2025
Eqlaaeer
Ten Dets
Top inhaardd Thaninn Data
Segläie
Teel Dera
Top infuaataa Traiing Cota
Tall Okta
Top Bia fut DMd
(a) High-Conf Correct Predict
(b) Low-Conf Correct Predict
(c) Low-Conf Incorrect Predict
Figure 3: Qualitative evaluation of various example-base baasee explanation methods using CIFARIO. We show three scenarios where the target model makes a) highly-confident prediction that matches ground truth label, b) a low-confident prediction that matches ground truth label, c) low-confident prediction that does not match ground truth label (which is a bird). For each sub plot, we show top-3 influential training data points picked by the explanation methods for the test example.
Eqpaaerr
Iett Duta
No Biaat of Houney Date
To kibur Tonnn Data
Cxplaarr
Tel Data
kop Viburtal Data
17
214
51
214
17
51
48
(a) High-Conf Correct Predict
High-Conf Correct Predict
Low-Conf Correct Predict
Figure 4: Qualitative evaluation of various example-bar basele-basedd explanation methods using SVHN. We show two scenarios where the target model makes a-b) highly-confident prediction that matches ground truth label, low-confident prediction that matches ground truth label. For each sub plot, we show top-3 influential training data points picked by the explanation methods for the test example. We include two samples of high-confidence correct predictions to show the overlap of explanations.
Formally,
UU esl
Coverage
X k
where e; is the set of top-k explanations for test data point i. Coverage is motivated measure the diversity of explanations across a test set where a high value reflects higher granularity (per test point) of the explanation.
Run Time: It measures the run time of an explanation method in wall clock time.
4.1 QUALITATIVE EVALUATION
Figure 3 shows three test cases of the CIFARIO classification task that cover different classification outcomes, including high-confidence correct prediction. low-confident correct prediction, and low- confident incorrect prediction. For both correct prediction cases, we are confident that HD-Explain provides better explanation than others in terms of visually matching test data points e.g. brown
7

---

Published as a conference paper at ICLR 2025
HD-E HD-Explain
HD-Explain*
RPS
TracIn
on Time
207
0.4
CFAR10
OCH
CIFAR10
CIFAR10
(a) Hit Rate
(b) Coverage
(c) Execution Time
Figure 5: Quantitative explanation comparison among candidate example-base based explanation methods. Data augmentation strategy used is Noise Injection. Error bar shows 95% confidence interval.
0.8
201
10-4
Besoutinn Time
uncoonda)
30x
20t
10-4
CIMAR10
DCH
DIPAR10
SVWNK
CIFAR1O
DCH
Hit Rate
Coverage
Execution Time
Figure 6: Quantitative explanation comparison among candidate example- based explanation methods. Data augmentation strategy used is Horizontal Flip. Error bar shows 95% % confidence interval. We reuse the legend of Figure
frogs in Figure 3 (a) and deer on the grass in Figure (b). In contrast, for the misclassified prediction case (as shown in Figure 3 (c)), we note the HD-Explain produces an example that does not even belong to the same class the predicted one. pecifically, the predicted class cat (while the ground truth label is bird), and HD-Explain generates an explanation sample from the deer class. This reflects a low confidence model's prediction for the partieular test example and highlight a potential error in the prediction. RPS also shows such inconsistency in explanation, which aligns with its claim (Yeh et al., 2018). The other two baseline methods do not offer such properties and still produce explanations that match the predicted label well. It is hard justify how those training samples support such prediction visually (since no clear shared pattern obvious to us). addition, is interesting to that Influence Function and TracIn produce near identical explanations, reflecting their similarity leveraging the perturbation model parameters.
Figure 4 provides additional insights SVHN dataset. HD-Expii HD-Explain again shows better explanation for producing training samples that appear similar to the test samples. In addition, we notice that RPS produces the same of explanations for different sample cases, shown in Fig. (a-b), which reveals its limitation in providing instance-level explanations. To verify this observation further, we conducted quantitative evaluation as described in the next section.
The qualitative evaluation for OCH and MRI datasets are given in the appendix due to the page limit of the main paper. The overall observations remain consistent with CIFARIO and SVHN.
4.2 QUANTITATIVE EVALUATION
In order to perform quantitative evaluation, we limit our experiments to datasets where ground-truth explanation samples are available. Specifically, given a training data sample (xi, v.), we generate test point by adopting two image data augmentation methods:
Noise Injection: X, = X; + s.t. ~~ (0, 0.01o), where a is the element-wise standard deviation of features the entire training dataset.
Horizontal Flip: x; = flip(x:), where we flip images horizontally that do not compromise the semantic meaning of images.
8

---

Published as a conference paper at ICLR 2025
RBF
IMQ
Linear
1.0
1.0
1.0
0.8
0.8
0.8
Specol Time 1h sesseeeee
0.6
0.6
0.0
CIAR1O
OCH
NRI
SVHN
CIRAR1O
OCH
MRI
SVHN
CIRRRO
OCH
NRI
SVHN
CIRALO OCH
MRI
SVHN
(a) Hit Rate
(b) Coverage (both)
(c) Hit Rate
(d) Execution Time (both)
(Noise Injection)
(Horizontal Flip)
Figure 7: Quantitative explanation comparison among HDD-Epllaiers with different kernll functions on all image classification datasets. Error bar shows 95% confidence interval.
We created 30 augmented test points for each training data point (> 10, 000 data points) in each dataset, resulting in more than 300, 000 independent runs. Since the data augmentation is guaranteed to maintain prediction consistency, the ideally best explanation for the generated test point is the original data point x; itself. Hence, the quantitative evaluation could be sample retrieval evaluation where Hit Rate measures the probability of successful retrieval.
Figure aaa) shows the hit rate comparison among candidate methods on the four image classification datasets under Noise Injection data augmentation. The existing methods face significant difficulty in retrieving the ideal explanatory sample (< 10%), even with such a simple problem setup: only HD-Explain (and its variant) produces a reasonable successful rate (> 80%). We further investigate the diversity of explanations across a testset using the Coverage metric. Here, diversity indirectly reflects the granularity of an explanation when accumulated over the test dataset. Figure shows the Coverage score, the ratio of explanation samples that are unique over many test points. It turns out that existing solutions produce only 10% 50% coverage many test points receive the same set of explanations, disregarding their unique characteristic, We further observed that the explanations of baselines are often dominated by the class labels; data points predicted the same class would receive a similar set of explanations. In contrast, HD- -xplaxxliin shows substantially higher coverage, generating explanations that considers the unique characteristics of each test point.
Regarding computation efficiency. while we have summarized the scalability limitation of the candidate methods in Ta there
computational efficiency evaluation conducted in previous works. We recorded the wall clock execution time of each experiments as shown in Figure ((c)). As expected, the Influence Function takes longer to return its explanation than other methods. HD- Explain TracIn* and RPS, all use the last layer to generate explanation. RPS showed the lowest compute time since it does not require tittodiiffeeentatioon for computing the training data influence. HD-Explain* showed the second best compute time and is efficient than TracIn considers the whole model for explanation and compute time is not directly comparable to others. However, it shows better efficiency than IF across all datasets and is better than TracIn on CIFAR 10.
F and IF. HD-Explain
We observe a similar trend the other data augmentation scenario, Horizontal Flip, where computa- tion time and coverage are roughly the same, as shown in Figure However, we do notice that, the outcome image flipping, the raw feature (pixel) level similarity between x, and x; is destroyed. As features suffers from performance deduction while
an outcome, the HD-Exp Explain that works other methods, including HD-Explain*, are less impacted. This observation suggests that choosing the layer of explanation might be considered in the practical usage of this approach.
4.3 KERNEL OPTIONS
We use the Radial Basis Function (RBF) as our default choice of kernel. However, another kernel may better fit particular application domain. In this experiment, compare three well-known kernels i.e. Linear, RBF, and Inverse Multi-Quadric (IMQ) on the three image classification datasets.
T is configured only to compute the gradient of the prediction layer due to its high memory require- ments.

---

Published as a conference paper at ICLR 2025
Figure 7 presents the results under both data augmentation scenarios. Overall, the IMQ kernel performs better than the RBF kernel regarding explanation quality (Hit Rate). The advantage is significant when the data augmentation scenario is Horizontal Flip (Figure which appears more challenging than Noise Injection. IMQ also showed better performance on Coverage (Figure ( The linear kernel performs worse compared to other kernels. However, it is substantially efficient than the others, as shown in Figure Zü. highlighting its utility on large datasets. Compared to the baselines presented Figure we note that the Linear kernel sufficient for HD-Explain stand out from other methods in both performance and efficiency.
4.4 DISCUSSION: INTCITION ON WIYY HD-EXPLAIN WORKS
After showing HD-Explain's empirical performance, we now present our understanding of how HD-Explain finds the explanations for the prediction of test data point. In particular, we want to understand why the approach is faithful the pre- rre-trinee model.
In Explaixi the key metric on measuring the predictive supports of a test point x; given a training data (x,, ys) is the KSD defined kernel Ne He(( [xx: [y., [x]]|y]]), where ý, denotes the predicted class label by model fo in one-hot encoding. By definition, the kernel Ki (x(( ya), (xo, 9b)) = (o
between two data points be decomposed into four terms
trace Vok(a, b)) + k(a, V. log Pa(a) log Po(6)
Vak(a,b) b) lo Po( + Vok(a, b) V. log (a
We examine the effect of each term as follows:
@: We note that the first term often similarity bias of raw data points given a specified kernel function. In particular, for the RBF kernel k(a, b) exp(->lla - bll²), the first term is simply dd 2x-22 b), where d refers to the sum of input and output (in one-hot) dimensions of data point. Intuitively, the term shows how similar the two data points are given the RBF kernel. For linear kernel k(a, b) a b, another hand, the first term is simply d + I as constant bias term, which does not deliver any similarity information between the two data points.
2: The second term reflects the similarity between two data points in the context of the trained model. In particular, considering the sub-term V. log (a(a) Vv log (b),,, based on our deriva- tion in Equation (in the main paper), we note it is equivalent to
Vx. log fo())) log fo(x.) )1 Xx, log fo(x)) ( pn log fx()))]
Vx, log fo(xa) T. Vx, log fx(x)) vn + log fx())) log (xt),
similarity scntes (input gradients)
similarity of predictions
where both terms could be viewed as similarity between data points in the context of trained model.
3-C: Both of the last two terms examine the aligmment between the score of one data point and the kernel derivative of another data point. We conjecture that this alignment reflects how test prediction would change if there training data point present closer to it than before.
5 CONCLUSION
This paper presents HD-Explain, a Kernel Stein Discrepancy-driven example-based prediction explanation method. We performed comprehensive qualitative and quantitative evaluation comparing three baseline explanation methods using three datasets. The results demonstrated the efficacy of HD-Explain in generating explanations that are accurate and effective in terms of their granularity level. In addition, compared to other methods, HD-Explain is flexible to apply on any layer of interest and can be used to analyze the evolution of a prediction across layers. HD-Explain serves as an important contribution towards improving the transparency of machine learning models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ariful Islam Anik and Andrea Bunt. Data-centric explanations: explaining training data of machine learning systems to promote transparency. In Proceedings of the 2021 CHI Conference Human Factors in Computing Systems, pp. 1-13,200 2021.
Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions the answer, then what is the question? Advances in Neural Information Processing Systems, 35:1793-11796, 2022.
Alessandro Вarp, Francois-Xate Xavier Briol, Andrew Duncan, Mark Girolami, and Lester Mackey. Minimum stein discrepancy estimators. Advances in Neural Information Processing Systems, 32, 2019.
Elnaz Barshan, Mare-EEienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training samples via relative influence. In International Conference Artificial Intelligence and Statistics, pp. 1899-1908 1909. PMLR, 2020.
Jonathan Brophy. Zayd Hammoudeh, and Daniel Lowd. Adapting and evaluating influence-essimatiin methods for gradient-boosted decision trees. Journal of Machine Learning Research, 24(154): 1-48, 2023.
Carrie J Cai, Jonas Jongejan, and Jess Holbrook. The effects of example-based explanations in a machine learning interface. In Proceedings the 24th international conference on intelligent user interfaces, pp. 258-228882 208,262,220 2019.
Yuanyuan Chen, Boyang Li, Han Yu, Pengeheng Wu, and Chunyan Miao. Hydra: Hypergradient data relevance analysis for interpreting deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7081-7089, 2021.
Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton. kernel test of goodness of fit. In International conference machine learning, pp. 2606-2615, PMLR, 2016.
Nicholas Frosst and Geoffrey Hinton. Distilling neural network into soft decision tree. arXiv preprint arXiv:1711.no 1.0077844 2017.
Jackson Gorham, Anant Raj, and Lester Mackey. Stochastic stein discrepancies. Advances Neural Information Processing Systems, 33:17931-17942, 2020.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Understanding predictions with data and data with predictions. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Pro- ceedings of Machine Learning Research, pp. 9525-9587. PMLR, 17-23 Jul 2022. URL sttps://proceeeinngs m www.ppress.ve.... wwt.v////l1lsaa/a yasl2a. html
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Wittawat Jitkrittum, Heishiro Kanagawa, and Bernhard Schölkopf. Testing goodness of fit of conditional density models with kernels. In Conference on Uncertainty in Artificial Intelligence, pp. 221-230. PMLR, 2020.
Sudheesh Kumar Kattumannil. On stein's 's identity and its applications. Statistics & prrbbaility letters. 19(12):1444-11 2009.
Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. Examples enough, learn to criticize! criticism for interpretability. Advances in neural information processing systems, 29, 2016.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Anna Korba, Pierre-Cyril Aubin Frankowki,, Szymon Majewski, and Pierre Ablin. Kernel stein discrepancy descent. In International Conference on Machine Learning, pp. 5719-5730. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2025
Tatsuya Kubokawa. Stein's identities and the related topics: instructive explanation shrinkage. characterization, normal approximation and goodness-of-fit Japanese Journal of Statistics and Data Science, pp. 1-45,200 2024.
Xiao-Hui Li, Caleb Chen Cao, Yuhan Shi, Wei Bai, Han Gao, Luyu Qiu, Cong Wang, Yuanyuan Gao, Shenjia Zhang, Xun Xue, et al. A survey of data-driven and knowledge-aware explainable ai. IEEE Transactions on Knowledge and Data Engineering, 34(1):29-44 49, 2020.
Brian Lim, Qian Yang, Ashraf M Abdul, and Danding Wang. Why these explanations? selecting intelligibility types for explanation goals. In IUI Workshops, 2019.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29, 2016.
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-in tests. In International conference on machine learning, pp. 276-284 284. PMLR, 2016.
Christoph Molnar. Interpreluble machine learning. Lulu. com, 2020.
Chang Nam, Yoon Jung, and Sangwon Lee. Human-Centered Artificial Intelligence: Research and Applications. Academic Press, 2022.
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. In International Conference on Machine Learning (ICML), 2023a.
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. TRAK: Attributing model behavior at scale. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 27074-27113, PMLR, 23-29 Jul 2023b. URL httpps:/ppoced.ngs mlr, m...ppess/ v202/park23c. html
Leif E Peterson. K-nearest neighbor. Scholarpedia, 4(2):1883, 2009.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 1779-1166.. 2022.
Yi Sui, Ga Wu, and Scott Sanner. Representer point selection via local jacobian expansion for post-hoc classifier explanation of deep neural networks and ensemble models. Advances in neural information processing systems, 34:23347-23358, 2021.
Che-Ping Tsai, Chih- Yeh, and Pradeep Ravikumar. Sample based explanations via generalized representers. Advances in Neural Information Processing Systems., 36, 2023.
Jiasen Yang, Qiang Liu, Vinayak Rao, and Jennifer Neville. Goodness-of-ff testing for discrete distributions via stein discrepancy. In International Conference on Machine Learning, pp. 5561- 5570. PMLR, 2018.
Chih- Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31, 2018.
Jianlong Zhou, Amir H Gandomi, Fang Chen, and Andreas Holzinger. Evaluating the quality of machine learning explanations: A survey on methods and metries. Electronics, 10(5):593, 2021.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX / SUPPLEMENTAL MATERIAL
B PROBLEM DEFINITION RECAP
We consider the task of explaining the prediction differentiable classifier IR + IR', given inputs test sample x IR², where d denotes the input dimension and denotes the number of classes. Specifically, we are interested in explaining a prediction of a model f(.) by returning subset of its training samples D {x,, yt )= that has strong predictive support the prediction test point X₁. While not explicitly stated in the main paper. we treat example-based prediction explanation as function (f, D, xx) : F x D x IR - {RR IR' such that it takes a trained model f, training dataset D, and an arbitrary test point x, inputs and output top-k training samples as explanations.
C ADDITIONAL DERIVATION OF KERNELIZED STEIN DISCREPANCY
While Stein's Identity has been well described in many previous works (Liu et al. 2016 Liu & Wang 2016; Chwialkowski et al. 2016), we briefly recap some key derivations in this paper to seek for self-contained.
As mentioned in the main paper, Stein's Identity states that, if smooth distribution p(z) and function (ιx) satisfy [imp (()) 0, we have
E V. log p(x) v()) Ex-p (An((())))))
Ad.
Intuitively, by using integration by part rules, we can reveal the original assumption from the derived expression such that
[ (V))2>logl( + v
Stein Discrepancy measures the difference between two distributions & and by replacing the expectation of distribution term in Stein's 's Identity expression with distribution q, which reveals the difference between two distributions by projecting their score functions (gradients) with the function ((x)
max
BaxlEra Exwn (nxxEx((App(()] max Ex #EF
[Apo(z)] Exzg (x)))A,
#EF
max Egooy FEF
((x)
V. llogg(() V log (x)))
projection coefficients
score function difference
Clearly, the choice of projection coefficients (function ((z)) term is critical to measure the distribution difference.
Kernelized Stein Discrepancy (KSD) addresses the task of searching function & by treating the above challenge an optimization task where it decomposes the target function e with linear decomposition such that
max E (A,()())] = max E Enr.nn (Ap Σ wr))(x)) dEF
Σ w. Burn (epBennnAAsCCCC (x)) dEf
max
dEF
with linear property of Stein operator Ap. The linear decomposition path is the way to reduce the optimization task into looking for a finite number of the base functions Φ; E F whose coefficient norm is constraint space (RKHS) and leverages its reproducing property such that ((x) = (o(+). k(x, :)), which in turn transforms the maximization objective of the Stein Discrepancy into
([]w
KSD takes F be the unit ball of reproducing kernel Hilbert
max (..())),,-uu0 Ez B [Apk(, ( s.t. lo/l| S 1.
The optimal & is therefore a normalized version of Enpa [Apkl [A,k(,,2) optimal between the distribution and with the optimal solution of
:)). Hence, KSD defined as the
S(g. p) By ~ (x,,x')))
where (x)xxx))) AT k(xx
13

---

Published as a conference paper at ICLR 2025
D DISCUSSION: RELAXATIONS OF KSD ESTIMATION
In section 3.1 introduced multiple relaxations such that KSD estimation can support predictive model (as conditional distribution model) with discrete labels. However, we want to clarify that the relaxations introduced are not generally applicable to other context (e.g.. Goodness of Fit) given their selective conditions on the data input distribution. In fact, the data distribution P that generated the datasets {(x,, yt ] could be complex given the potentially intractable marginal distribution of P(x). Our solution avoids the modelling of such complexity by only limiting the KSD discrepancy computation between model distribution Pe and the sampled data distribution PD instead of touch original distribution P.
In particular, in the data-centric prediction explanation context, we only aim to extract training data points that are similar to the test sample, such that all inputs in the framework are valid (e.g.. images) rather than random continuous inputs sampled from an unknown distribution that might be inevitable in other contexts. To apply the similar relaxation to other context, thorough theoretical proof is needed, which out of the scope of this research.
It is worth noting that the both relaxations introduced in this paper has corresponding theoretically rigid solutions (Yngg et al. 2018 Jitkrittum et al. 2020) with the cost of computational overhead. While they are much more elegant, for explanation prediction purpose, we may prefer faster approxi- mations. How to better incorporate the theoretically rigid solution in a more efficient way will be in our future research.
E DISCUSSION: PREDICTION EXPLAIN QUALITY ON HEALTHCARE DATASET
Teet Crea
Interrraa Training Date
Teet uambaa Traiiin Dats
Tart Data
Top Intuueettti Treainn
Teat Dart
usental Traiiin Dats
Irtenss Rundtion
inannoe Fuector
RPS
FD DD-aann
HD bolar
HD tookat
Overian Cancer Histopathology (OCH)
(b) Brain Tumor MRI
Figure 8: Qualitative evaluation of example-base based explanation methods on Overian Cancer histopathology and Brain Tumor MRI datasets. We show two test data points that are predicted to belong to the same class in each dataset. Red triangle in the top right corner of image shows the duplicate explanations across test samples.
Figure 8 provides additional insights into Ovarian Cancer histopathology and Brain Tumor MRI datasets. HD-Explain again shows a better explanation for producing training samples that appear similar to the test samples (note for the semantic similarity, these explanations should be referred to medical practitioner). For instance, the explanation of HD-Explain follows the scanning orientation of test points in MRI as shown in Figure g (b). We note all baseline approaches tend to produce similar explanations to test samples belonging to the same classes. Rather than providing individual prediction explanations, those approaches act closer to per-class interpreters that look for class prototypes. To verify this observation further, we conducted a quantitative evaluation as described in the next section.
There is potential concern on interpreting our experiments on the two healthcare datasets provided, where domain experts' evaluation contained in this work.
14

---

Published as a conference paper at ICLR 2025
In the quantitative evaluation (Figures and b we highlight that HD-Explain demonstrates bet- ter explanation performance in terms of retrieving original data points that were used for data augmentation-base based retrieval tests. This study is objective, requiring domain knowledge for result validation.
Regarding the qualitative evaluation (as part of the visualization in Figure 4), agree that the explanation quality of healthcare datasets needs insights from domain experts, whereas machine learning research in general often lacks corresponding authority to justify specific domain's results. This challenge extends across the entire model explanation literature. Our objective is to facilitate the general population's 's understanding. even in the absence of domain expert evaluation. It is important to recap that the provided explanations by HD-Explain are visually similar to the test data points. However, their deeper pathological interpretation requires further investigation by healthcare practitioners, which we encourage for future studies. Furthermore, experimental evaluation of our approach on CIFAR demonstrates that the effectiveness of our method is not limited to medical datasets and can be easily applied across other domains. Overall, we believe that qualitative evaluation aims to enhance understanding the model's behavior rather than serving as a performance justification.
F
DATASET DETAILS
Table 2: Summary of datasets used in the paper.
Dutaset
Application
Type
# Size
# Feature Dinetsion
Number of Clases
Duplicated Samples
Public Dutuact
Two Moens
Syathetie
2D Numerie
500
No
Shared with cods
Rectingulars
Synthetie
2D Nomeric
500
No
Shared with code
CIFAR-10
Clasilication Benehmark
lunge
60.0001
32 x 8
10
No
Yes No
Overian Canoer
Histepahhloay (Pricate)
Image
20,000
128
128 x
5
Yes
Reain Tumar
MRI Benchhuokk
Image
7,023
128
128
x
Yes
Yes
In this paper, we conducted our experiments on five datasets two synthetic and three benchmark image classification datasets. As the work concerns the trustworthiness of the machine learning model in high-stakes applications, we also introduced medical diagnosis datasets to provide more insight into the potential benefit the proposed work introduced. To train the target machine learning models, we conducted data augmentations to increase the number of training data samples, including random cropping, rotation, shifting, horizontal flipping, and noise injection. Table 2 summarizes more details about the datasets.
G DATA DEBUGGING
Before describing the data debugging setting of this paper, we want recap that the data debugging functionality is a side effect/benefit of HD-Explain, which is not our main proposal. Indeed, using the prediction explanation method as a data debugging tool is still under investigation since it might be over-claimed due to the over-regularized setting in previous works (e.g. Binary classification tasks). While we relaxed some settings, we don't 't claim practical for real-world applications.
The data debugging task in this paper is a data sample retrieval task where we retrieve samples that intentionally flipped their classification labels. Higher Precision and Recall of the retrieval reflects higher performance of data debugging.
For the HD-Ex HD-Explain (and its variant HD-ExpD-aii EX-plaiii"**, the retrivial order is determined by the values of the diagonals of the KSD-defined kernel matrix, (a,a, a) for all E D. This setting is very similar to how the Influence function does the data debugging with the self-influence of a data sample. Indeed, Ke(a, a) could be treated self-influence that does not rely on model parameters.
Now, we describe our data debugging experiments to highlight the self-explanatory ability of candidate methods on the training data. In particular, we generalized previous research's binary classification- based data debugging experiment into a classficlssiiiitiioonn scenario, where we randomly flip labels of 100 training data points at each run. We adopt standard information retrieval metrics, Precision and Recall, that measure how likely the candidate methods can retrieve the mislabeled training data points. Figure shows experimental results. While HD-Explain the entire model has little
15

---

Published as a conference paper at ICLR 2025
HD-Explaie
HD-Explain"
RPS
1.0
1.0
0.8
0.8
Precision@k
RecalliK
0.4
0.4
0.2
0.2
0.0
20
40
60
80
100
20
40
60
80
100
Top-K Suspecious Mislabeling
Top-K Suspecious Mislabeling
(a) Recall@
(b) Precision @K
Data debugging comparison among candidate methods on CIFAR-10 dataset. Results collected from 30 independent runs. Error bar shows 95% confidence interval.
Figure
data debugging ability, its variant on the last layer offers outstanding performance compared the other last-layer explanation methods. Note, the data debugging functionality is a side effect/benefit of HD-Explain, which not our main proposal
H HARDWARE SETUP
We all experiments a machine equipped with GTX 1080 Ti GPU, a second-generation Ryzen 5 processor, and 32 GB of memory.
I BROADER IMPACT
The development of HD-ExD ID-xxplain, a highly precise and data-centric explanation method for neural classifiers, promises to significantly enhance the transparency and trustworthiness of machine learning models across various applications. Furthermore, HID-Explain's scalable and computationally efficient approach makes it feasible for deployment in large-scale, real-world applications. This not only promotes transparency and accountability in AI systems but also paves the way for broader acceptance and integration of AI technologies in society. By bridging the gap between complex model behavior and human understanding. HD-Explain fosters a more informed and trust-based relationship between AI systems and their users. Overall, HD-Explain's 's contributions to model interpretability and transparency have the potential to drive significant advancements in the responsible and ethical use of AI, ensuring that these technologies are developed and deployed in ways that are understandable, accountable, and aligned with societal values.
However, in terms of Negative Societal Impacts, over-reliance on explanation methods like HD- Explain might create false sense of interpretability, masking the inherent limitations and uncertainties of machine learning models. Thus, careful consideration and mitigation of these negative impacts are crucial for the responsible deployment of HD- HD-Explain and similar tools.
J LIMITATION OF PROPOSED METHOD
While HD- HD-Explain demonstrates significant promise in providing detailed explanations for neural classifier predictions, we intended investigate the limitation of our method.
For given datapoint, we sorted all data based on their Stein Kernel similarity to the target data and found that the top relevant data points selected by HD-Explain were very similar attributes such as colour palette, object position, and background colour, which are unique to the target sample (See Figure [0 Left). This observation triggered our curiosity about whether the HD-Explain sensitive to its raw input features (low-level information). To investigate this possibility, we set low threshold to capture a large portion of relevant data points. We observed that for data points with low Stein Kernel similarity (i.e. they weakly relevant), the dot product of Stein scores is low but still above this datapoint designated threshold. We noticed that the model's prediction confidence is lower for such outliers due the reduced dot product of Stein scores, indicating reliance RBF similarity.
16

---

Published as a conference paper at ICLR 2025
Original
RBF
IF
KSD
Originm niggmal
SAMPLE
Label:
Label Predi car
Labal
Label:
Label: deer
ship
Labal:
Prod: car
Pred: dear
Pred: car
Original
Label: horse
Label bird Pred: bird
Label: horse Predi horse
bell:: harse Presh: horse
Pred: horse
Pred:
Pred:
Original
Label: horse
La Label doer Pred: dear
Label horse Predi horse
La bel: horse Pred: horse
Label: Pred: horse
Label: Pred: deer
Label: deer Pred: deer
Predi horse
Pred: bird
Origiral
Label: dog Predi dog
Label: dear Predi deer
Label: dog Pred: dog
Label: dog Pred: dog
Label: Pred: dog
Label: dog Predi dog
Label: Pred: cat
Label: Predi cat
Figure 10: Demonstrative figures to show the limitations our method on CIFAR-10 dataset. Left) Given query sample, the most relevant data points selected by different methods, such RBF kernel similarity (RBF), influence-based methods (IF*), and HD-Explain. Right): By setting a low threshold the KSD, we note HD-Exppain can produce less informative explanation. HD-Explain still falsefully considered them as explanation. Our conjecture is that it relies more on the RBF similarity.
Such that, HD-Explain's performance depends on the model's prediction quality and generalization ability. As HD-Explain incorporates the complete derivative calculation, it includes more information from the model's internal weights, but generalization errors, such overlapping decision boundaries and the impact noisy data, can cause the selection data points from other classes, especially for correctly labelled but cow-confidence predictions. To balance the potential drawback of modelling the raw feature space, the user can use HDD-xxllain*.
In the main paper, evaluated HD-Explain with three well-known kernel options, including Linear, RBF, and Inverse Multi- qutiiiQuarii (IMQ). The purpose was to demonstrate the impact of kernel choices on the performance of HD-Explain. While we show that IMQ performs best in our experimental environment, we want to highlight that the selection of the kernel may influence the HD-Explain's 's performance practice. Hence, one may need conduct empirical analysis on which kernel to use before applying HD-Explain in production.
K RELATION TO DATA ATTRIBUTION ESTIMATION
Data attribution estimation is closely related to the sample-based prediction explanation but a different concept. As stated by Park et al. 2023b), the definition of data attribution is follows:
Definition K.1 (Data attribution). Consider an ordered training set of examples (= } and model output function f(z; 0). A data attribution method 7(2, S) a function Z n In that, for any example input 2, indicating importance the model output f2; 0 (S)). When the second argument S is clear from the context, we will omit the second argument and simply write T(z).
Z and training set S, assigns a (real-valued) score to each training
In particular, data attribution estimation faces a more generalized problem that does not explain the importance of data for specific pre-trained model but the importance of a family of models of the same architecture or function. The star (0*) refers to the potentially optimal model that can be trained the dataset S. Indeed, if we examine the two approaches in the data attribution estimation literature (data modelling (Ilyas et al. 2022) and TRAK (Park et al. (023bb),, we note both approaches require either training multiple models on subset of data points introducing various aggressive approximations, such as (1) linear Taylor expansion, 2) random projection, and 3) newton approximation. Both data and model manipulations will cause unfaithfulnes, to the pre-trained model.
17

---

Published as a conference paper at ICLR 2025
In fact, the data modelling algorithm (llas et al. 2022) does not involve any pre-trained model but directly optimizes the linear data modelling score as follows.
1 L8 f():0(()))) + 1118111.
TDM (2)
min BER 77
L HD-EXPLAIN: EXPLANATION PROCESS
The following algorithm shows the algorithm of HD-Explain in pseudocode.
Algorithm HDD-Explain
Input: Training set D, Test input Xiests and classifier model fo
Output:
Sample based explanations Dexplann C D
1: Step 1: Caching
Reduce redundant computation
2: initialize empty list e []
3: for (x,, 9:) D do
4: p: + [a(x;)
5: gi Vx; log Jo (x:)1
6:
c.add([x;, Pt, gil)
7:
end for
8: Step 2: Prediction Contribution of Each Training Data
9: Given test input Niest
10: Prest f- (X(es)
11: Viest f- argiax Pust
>> Best Predicted Label
12: Biest f- Vx.. log fo (Xiest) Bis
13: c.add([Xeess Yhest: Prest: gwa))
V Xuss Y log Po F.) +- Bteat Prest
14: for (xi, yi) E D do
15: V Vx,,y; log Po (xi, ys)
g: p.]
Cache-able if needed
16: Kee (x(((x; Wi), (Xrest Prest) ) t Equation
end for
18: Dexplain f argsort((re [150((x;, ((x;, (x); (Xeest: Dres)) for E (DI)
M ADDITIONAL QUALITATIVE EVALUATION EXAMPLES FOR SVHN
Diplabee
Too Traning Data
230
(a) Low-Conf Incorrect Predict
Figure 11: Qualitative evaluation of various example-based explanation methods using SVHN, We show the missing scenario in the main paper, where the target model makes low-confident prediction that does not match ground truth label (which is 7). For the sub plot, we show top-3 influential training data points picked by the explanation methods for the test example. The observation is similar to that of CIFAR-10.
18


Reviews:
[{"review_id": "IRLBaI0Jeq", "reviewer": "Reviewer_54VY", "summary": "The paper presents HD-Explain, a data-centric, example-based explanation method leveraging Kernelized Stein Discrepancy (KSD). The authors argue that existing example-based methods for explaining predictions, such as Influence Functions and Representer Point Selection, suffer from high computational costs and coarse-grained explanations. HD-Explain addresses these challenges by using KSD to establish a fine-grained, efficient link between training samples and model predictions. Through empirical evaluations on datasets like CIFAR-10 and medical imaging, the paper shows that HD-Explain outperforms traditional methods in terms of explanation precision, consistency, and computational efficiency.", "questions": "Suggestions:\n\n1. The authors might consider adding an algorithm to show how the proposed method works.\n\n2. The authors are suggested to report LDS, if possible.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Applying Kernelized Stein Discrepancy to interpretability is novel and conceptually interesting. Unlike other methods that can struggle with memory or computation limits in large models, HD-Explain maintains a low computational cost, which support its use in high-stakes applications where explanation and efficiency are essential. The focus on healthcare datasets underscores the authors’ intent to target real-world, interpretable machine learning needs.\n\n2.   The paper attempts to quantify explanation performance with metrics like Hit Rate, Coverage, and Run Time. These metrics provide a foundation for benchmarking HD-Explain against other methods in terms of granularity, diversity, and efficiency.", "weaknesses": "1. The proposed method provides a new use case of KSD, which has been used in several other cases, such as goodness-of-fit applications, which are quite related to the new use case. The paper extends KSD to explainability, but the novelty is limited to an application shift, applying an existing statistical method to a new context rather than developing a new technique for explanation in machine learning.\n\n2. The paper explains that KSD is close to zero if a model fits the data well, which is shown in previous studies. However, I feel there is a gap of why KSD is a good choice for model explanation, specifically, selecting training samples given a test sample. Also there are other discrepancy measures, such as Maximum Mean Discrepancy (MMD), it's unclear to see why KSD is a better choice for this task among other discrepancy measures. Without testing these alternatives, the choice of KSD appears arbitrary, which weakens the novelty claim.\n\n3. In addition to the newly proposed evaluation metrics, existing metrics such as Linear Datamodeling Score (LDS) seem to be widely used in training data attribution methods. LDS was not reported in this proposed paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o5HKiD6MZN", "reviewer": "Reviewer_mojk", "summary": "This paper proposed an example-based explanation method based on kernelized Stein discrepancy (KSD), which can measure the discrepancy between a predictive model and training samples.\nIn particular, using a model-dependent kernel function used for calculating KSD, the proposed method explains training samples related to the input test sample.\nIn the experiments on image classification tasks, the proposed method and its variant outperformed the comparing methods in terms of two qualitative metrics.", "questions": "- As related works and comparing methods, Datamodels [2] and its variant, TRAK [3], are not mentioned. Is there a reason why they are not addressed?\n- Compared to KCSD, what are the key points of technical novelty in the proposed method?\n- In the noise injection setting, it seems that even a simple method that returns the nearest neighbor training sample of the test sample could explain the predictions. Compared to such a baseline, why does the proposed method perform better? Alternatively, can it be demonstrated through experiments that the proposed method is superior?\n\n[2] Ilyas, Andrew, et al. \"Datamodels: Predicting Predictions from Training Data.\" Proceedings of the 39th International Conference on Machine Learning. 2022.\n\n[3] Park, Sung Min, et al. \"TRAK: Attributing Model Behavior at Scale.\" International Conference on Machine Learning. PMLR, 2023.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The proposed method is a good and natural application of KSD, and as a result, it broadens the potential of KSD.\n- The derivation of the proposed method is convincing, and thanks to the well-written explanation, it is easy to follow.", "weaknesses": "- The proposed method seems a straightforward application of kernel conditional Stein discrepancy (KCSD) [1]. Therefore, the technical novelty of the proposed method is limited.\n- Although the experiments were conducted on popular image classification datasets, the evaluation protocol is artificial, especially in evaluating the hit rate.\n\n[1] Jitkrittum, Wittawat, Heishiro Kanagawa, and Bernhard Schölkopf. \"Testing goodness of fit of conditional density models with kernels.\" Conference on Uncertainty in Artificial Intelligence. PMLR, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CIEPXIxKjo", "reviewer": "Reviewer_3D8W", "summary": "This paper proposes a new method for computing instance-level explanations, based on the Kernelized Stein Discrepancy, by poiting to examples having the biggest contribution to the classification made by the model. Indeed, given a datapoint x, the methods aims at searching the top-k training datapoints leading to the biggest value of the defined KSD kernel. The approach is tested on both classical benchmark datasets from the literature (CIFAR-10, SVHN) as well as real-life datasets needing explanations (Brain Tumor, Ovarian Cancer) and compared to a few methods from the literature.", "questions": "-Could the authors cite other works using these metrics (Hit rate, Coverage)?\n-Could the authors provide explanations regarding the observations made in 4.3?\n-Did the authors prevent undesired results for the image flip on SVHN? For example, flipping a 2 leads to a 5.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed approach is both sound and original. The method is clearly explained.", "weaknesses": "1. While HD-Explain* is simply presented as a declination of HD-Explain, details should be part of the work since it is equally important to HD-explain in the experiment section. In the same line of thoughts: it is said that « The qualitative evaluation for OCH and MRI datasets are given in the appendix due to the page limit of the main paper. », but the current submission makes 9 pages out of a maximum of 10 (https://iclr.cc/Conferences/2025/CallForPapers).\n\n2. « In addition, as Influence Function and TracIn face scalability issues, we limit the influence of parameters to the last layer of the model so that they can work with models that contain a large number of parameters. Our experiments use ResNet-18 as the backbone model architecture (with around 11 million trainable parameters) for all image datasets » Since datasets such as CIFAR-10 can tackled decently by small neural networks, it would be necessary to test whether limiting the influence of parameters to the last layer of the model for benchmarks impacts the results or not. It yields a huge effect on HD-explain, as demonstrated by HD-explain*, so it might as well for other approaches.\n\n3. I feel like there are a few conceptual incoherence/false statements in the presented approach, both theoretically and in the empirical section.\n\n3.1 Line 320: « For both correct prediction cases, we are confident that HD-Explain provides a better explanation than others in terms of visually matching test data points e.g. brown frogs in Figure 3 (a) and deer on the grass in Figure 3 (b). » It is not because the test and the explanation images look alike to us that the model did use the information contained in the explanation to classify the test. Following the line of thought presented in the article, the explanation image should be near the test image in the feature space, yet these images might not particularly look alike to us. In short: I don’t think the observations made in section 4.1 tell us anything about the quality of the explanation.\n\n3.2 I don’t feel like the considered metrics actually reflect the quality of the approaches. For instance, using the coverage metric requires making the assumption that a wide array of training samples should be used for explaining a wide array of test images. One could easily imagine a setup where an image contains the typical representation of a class, each other image being the same as that first one, but each having a unique particularity. In this setup, the most relevant explanation would be the typical representation for every image, since it is the most similar to every other image. When it comes to hit rate, once again, we can imagine that once a picture is flipped or noised, it now more resembles another picture.\n\n**Typos and such** \n\n-Figure 1, right : Error in the Y-axis name (wrong letter, d instead of D; « heta » instead of $\\theta$).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qRnAvKuXvI", "reviewer": "Reviewer_o2SA", "summary": "The paper proposes a new example-based explanation technique by using Kernelized Stein Divergence (KSD) to identify coresets of training samples that best provide support for a particular prediction on a test point by the model.", "questions": "Can the authors expand on the use of kernels. In my experience euclidean distance based kernels (like RBF kernel used in the paper) tend to fall apart when used with high dimensional data - like images, due to curse of dimensionality. Given the reliance of this method on being able to compute useful kernels how do the authors think this method can be scaled to actual images (that are significantly higher dimensional than the benchmark images used here)?\n\nIn Figure 4, the authors say the \"model predictive accuracy is near 100%, ... random search\". Why do a random search here? Shouldn't you be able to pull any mispredicted samples deterministically?\n\nIn the quantitative eval section why is only horizontal flip tested? Would other transformations of data points hold just as well? (e.g. rotation -, Figure 1 seems to suggest that if the whole dataset is rotated that could impact KSD)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The biggest strength of the paper is the simple novelty of the idea. Coresets have a established history already and using a similar approach towards finding points in the training set that explain the model prediction for a test point is a great idea that seems to have not been tried before.", "weaknesses": "The biggest weakness of the paper is poor presentation and writing. The paper is unfortunately littered with typos, grammatical mistakes, and run-on sentences which make it sometimes very difficult to understand certain passages. \n\nThe other weakness is the experiments section, especially the qualitative experiments in Figures 3 and 4 that do not seem to clearly portray the proposed method's strength when compared to the competitors and it is left to the reader to figure it out.\n\nhere's a few typos and possible presentation improvements:\n\nline 93-94: currently it reads like the research history spans from 2020 to 2020. Please add that the papers cited are survey papers.\n\nline 106-107: \"unfortunately, those approaches are often with strong assumption of good prototypes (as of training data distribution)\", this sentence is not self explanatory. Please expand on what \"good\" prototypes means here and what strong assumptions.\n\nline 138: to assess the goodness-of-fit ... of what?\n\nFigure 1: y-axis should say $P_\\theta$ in markdown\n\nFigure 2 will probably make a better Figure 1 - it shows why the proposed method could potentially work better.\n\nline 235: \"that out of\" -> \"that is out of\"\n\nline 237-240: run-on sentence and ends abruptly without a conclusion.\n\nline 319: \"high-confident\" -> \"high-confidence\"\n\nline 323: \"we note the HD-Explain produces an example that does not even...\" in figure 3(c) all examples are labelled as cat, so not sure which example this sentence is referring to, and it's visually very hard to distinguish the images.\n\nline 366: \"target model makes 1-2)\" -> remove \"-2\"\n\nline 428: refers to Figure 8 and conclusions drawn from it as \"previous observation\". There is no Figure 8 in the main manuscript. Figure 8 is in the appendix and shows up later.\n\nConclusions section is inadequate and should include discussion on limitations and future directions. ICLR allows for 10 pages of text this year.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)