kaBEDONN: posthoc eXplainable Artificial Intelligence with Data Ordered Neural Network
Authors: Erico Tjoa
Cuntai Guan
Venue: Submitted to ICLR 2023 (2023)
Keywords: Explainable Artificial Intelligence
Neural Network
instance-based learning

Abstract: Different approaches to eXplainable Artificial Intelligence (XAI) have been explored including (1) the systematic study of the effect of individual training data sample on the final model (2) posthoc attribution methods that assign importance values to the components of each data sample. Combining concepts from both approaches, we introduce kaBEDONN, a system of ordered dataset coupled with a posthoc and model-agnostic method for querying \textit{relevant} training data samples. These \textit{relevant} data are intended as the explanations for model predictions that are both user-friendly and easily adjustable by developers. Explanations can thus be finetuned and damage control can be performed with ease.

Under review as a conference paper at ICLR 2023
KABEDONN: POSTHOC EXPLAINABLE ARTIFICIAL INTELLIGENCE WITH DATA ORDERED NEURAL NET- WORK
Anonymous authors Paper under double-blind review
ABSTRACT
Different approaches to eXplainable Artificial Intelligence (XAT) have been ex- plored including (1) the systematic study of the effect of individual training data sample on the final model (2) posthoc attribution methods that assign importance values to the components of each data sample. Combining concepts from both approaches, we introduce kaBEDONN, system of ordered dataset coupled with a posthoc and model-agnostic method for querying relevant training data samples. These relevant data intended the cxplanations for model predictions that are both user-fi seer-friendly and easily adjustable by developers. Explanations can thus be finetuned and damage control can be performed with ease.
INTRODUCTION
Although machine learning (ML) algorithms ures can be detrimental e.g. the well-known incident of 'racist" algorithm [bb (2015). EXplainable Artificial Intelligence (XAI) has emerged an effort to help improve trust in the use of ML algo- rithms. It is a burgeoning field that has been recently studied from different aspects, such as (1) data influence on model training (2) post-hoc attribution methods (3) "signal methods" etc (some meth- ods fall into multiple categories as seen in surveys like Arrieta et al.) (2020); Gilpin et al. (2018); Tija & Guan (2020); Adadi & Berrada (2018)) With improved trust, powerful blackbox models like the deep neural network (DNN) can be adopted into real applications with more accountability. Combining some of these existing concepts, we introduce k-width and Bifold Embedded Data Or- dered Neural Network (kaBEDONN), which is post-hoc XAI method to query relevant data the explanation for model prediction. All python codes are available in the supp. materials.
expected to be perfect, their unexplained fail-
Here, we consider the image classification task, including experiments MNIST, CIFARIO, ImageNet. Denote a sample data as (I, y0) E D = X Y where X is the input space and y0 Y the ground-truth class label. is classified using some base model f as argmazz: where considered in this paper is the following: require explanations for the predictions. Like many XAI methods, kaBEDONN aims to provide form of explanation. We start by clarifying our three objectives.
common image datasets
f(x) E RCC and C' the number of classes/categories. The scenario
wish to know why ʃ labels the sample i.e. they
Objective 1. Relevant data as explanations. The relevance of data has been measured in differ- ent ways. In Koh & Liang (2017), training data sample is considered either helpful or harmful to the prediction made by trained model, quantified by the influence score. In Yeh et al. (2018), data samples are either excitatory or inhibitory, in Pruthi al.] (2020) proponent or opponent. For KaBEDONN, relevant data samples strongly activate main nodes or sub-nodes, hence, they are exci- tatory in a different sense than Yeh et al. (2018). Here, explanatory images are considered relevant when their features look "similar" to according to the base model f. The explanatory images then presented to users as shown in fig. [[A) and fig. [[A).
More technically, we have three different contexts of "similar". (1) A representative data is training data sample that has been used to construct a main node in kaBEDONN, kaBEDONN stores the processed signals of (also called "fingerprint" in (Tjoa & Cuntail (021)) and r's index w.r.t the ordered dataset in a main node, (2) A similar data is a training data sample that has neither been included main node nor sub-node because it is already well-represented
this case,

---

Under review as a conference paper at ICLR 2023
(A)
Sample Image
Ole
AR
Cmn1
Main explanation Moarrowwvelle aecid
Supporting explonations
('Seearr Ymaper
sub/// reso!
Sample imae
Mainnodee
A-BROOW porevtto
Relevant explanations
Knade pertover
1 1=2
Figure/ 1: Inspired by inftuential images for explainable Al Koh & Liang 2017); Yeh et al. (2018); Pruthi al. (2020), KaBEDONN provides explanations by querying relevant images from ordered data. (A) Sample images from the ImageNet dataset and the relevant data queried by kaBEDONN as explanations. (B) Data samples ordered by class y0 and position index idr. Data are queried in a deterministic order during kaBEDONN construction. (C) kaBEDONN with 3 layers and kwidth
4. Bifold embedding: the first fold consists of layers of main nodes, the second "upper" fold consists of sub-nodes. WR nodes are not shown.
by an existing main node. Only the index of (but not its fingerprint) is stored in a well-represented (WR) node (that belongs to main node). (3) A boundary data is training data sample that similar enough to a representative data r but is different (they have different class labels); this happens, for example, when similar-looking breeds of cats are labeled differently. Its fingerprint and index are stored in the sub-node of the main node constructed from
Objective 2. Adjustment of Explanation for debugging. Explanations do not necessarily convince every user to the same degree. Suppose users flag some explanations as unsuitable. kaBEDONN system that allows developers to quickly finetune explanations based on the given feedback, primarily by adding, removing or reordering the underlying data samples used for kaBEDONN construction. This is useful because problematic data samples (e.g. accidentally mislabeled data and reerettvpresenttative data*) can be identified and removed while better explanations can be incorporated to the system when available. Remark. * An example of repreetepreeenntatiie data an optical illusion; since it appears different from different point of views, it is not helpful as an explanation.
A real user feedback example from ImageNet. In fig. B panel 1, the image of interest (bullfrog image) activates main node with the wrong class label (hammerhead shark). Furthermore, we found that cartoon image of hammerhead shark in the ImageNet dataset is associated with that node. Suppose a user considers it undesirable, we need the developer to quickly readjust the expla- nation. This is done by simply reordering the cartoon image (panel 2): we push it the back of the queue by renaming the image. KaBEDONN is then reconstructed (panel 3), and a more "similar" representative presented (the node appears to respond to partially dark background). We have deliberately chosen the unclear and ambiguous bullfrog image from ImageNet to demonstrate how problematic case can be handled. The result is thus perfect; in practice, iterative user/developer feedback may be needed for better result. Also see appendix General info for more remarks.
posthoc explanatory model complementary to more complex blackbox base model f. It is constructed using the col- lection {(re, y0) xe = (x), (x. y0) E D' C D} where xe is a latent vector obtained from encoder fene. The encoder can be fene = itself or only the latent encoder part of f. In this paper, Te = CN i.c. fenc = f for the simplicity of demonstration. kaBEDONN can also act as predictive model through multi-layered processing of latent vectors, partially using the universal approximation (UA) concept in Tjoa & Cuntai (2021). While KaBEDONN no longer has the UA property from Tjoa & Cuntai (2021). the data fitting capability is still very high (see experiment and result section). The discrepancy between predictions made by kaBEDONN and the base model f (e.g ResNet) serves flag for user/developer report abnormal prediction.
Objective 3. Predictive correctness flag for debugging. kaBEDONN
2

---

Under review as a conference paper at ICLR 2023
(A)
(B)
Ponel
Image
Hommerhead Shark!
Image image
Bull
frog)
?
ResNet18
ResNet18
KaBEDONN
kaBEDONN
"Simler" Images
Sub-noue octivation, Self representative
Undesirable representattve 12652JPEG
Explanation Impresentatiie image)
n0149447:
Ponel
Ponel
Image image frag)
Freg
n01-19447: m83486675 22114664751122222PP11 1P1G2
ResNet18
KaBEDONN (new)
NaBEDONN
KOBLDOMM reconstructed
activation,
KaBEDONN
Self-reee represertttee
"Simloor image,
Develiser readlusts explonation through simple reandering.
The rodi appeas oppture dark obles parisdly dank hockgrunnd
Figure 2: KaBEDONN provides explanations for ImageNet samples. Gray arrows are what users see. (A) kaBEDONN provides
with representative image the standard explanation and "similar" images as supporting explanations. (B) kaBEDONN applied on an ambiguous image of bullfirg. User finds an explanation (through sub-node activation) with undesirable main repre- sentative: cartoon image of hammerhead shark in ImageNet dataset. Developer fixes the issue by reordering the undesirable data. New representative is queried. It is still not quite the right expla- nation although it has similar colour texture (dark object in dark background). Furthermore, with kaBEDONN's sub-node activation, self-representative sub-node activated, thus yielding the cor- rect label.
To summarize, KaBEDONN not only a posthoc XAI method that presents users with "similar" images, but is also adjustable system that is friendly model developers for two reasons. (1) Ad- justment of D'. Problematic samples can be easily readjusted, reducing the likelihood that they turn into representative kaBEDONN main nodes (i.e. less likely served explanations), thus improving the quality of main explanations. Explicit indexing (y0, idz) = (class, index) of the data makes adjustment/feeddackk process easy. (2) kaBEDONN is posthoc and model agnostic, hence it does directly interfere with the main model f. By comparison, Neural Backed Decision Tree Wan et al. (02TT) requires finetuning of the main model's weights. Regardless, kaBEDONN helps developers collect user feedback on some sample data; data flagged as problematic may later be excluded from further finetuning of f.
RELATED WORKS
Post-hoc attribution methods. XAI methods such as Local Interpretable Model Agnostic Explana-
tion (LIME) Ribeiro et al. (2016), SHapley Additive exPlanation (SHAP) Lundberg & Lee (211))
Class Activation Mappings (CAM) Zhou al. (016)), Layerwise Relevance Propagation (LRP)
Bach et al. (2055) and their variants compute the attribution values of the components of individual data. These values represent the components' "importance"
the final prediction. For image data, the components are pixels. These methods are posthoc: all computations are made after model train- ing/finetuning. thus they do not change any the model's components. Our model also posthoc, but instead of computing the importance of data components, we compute the importance of the data w.r.t the adjustable, ordered dataset underlying KaBEDONN.
Data influence. Koh & Liang (2017); Yeh al. (2018) Pruthi et al. (2020) compute the influence of training data on the final model. The most influential training sample then shown as the explana- tion the prediction of sample. kaBEDONN similarly presents relevant images explanations; our corresponding influence score being the strength of neuron activation. The influence score in Yeh al. (2018) is called the representer value, obtained through representer theorem optimization
3

---

Under review as a conference paper at ICLR 2023
Schölkopf et al. (2001). A data point is then considered inhibitory/excitatoo if it suppresses/aug- ments the activation value. kaBEDONN is also in a sense excitatory: if x is represented well by r, then will strongly activate a node that has been constructed using More precisely, our influence score is measured as the strength of node activation measured in [0, 1]: strong activation is near 1, and weak activation is near 0 i.e. it has an excitatory aspect; KaBEDONN does not explicitly define inhibitory response. Furthermore, training sample r that has been converted into kaBEDONN node influential in the sense that it is able to represent a number of other samples W R((r) (defined more precisely later).
In extreme cases, the influence of a training data is computed by comparing the model that has been trained with given data and another model that has been trained without the same given data Koh & Liang (0017). This is a prohibitively expensive process, thus Koh & Liang (2017) modified the influence functions (a method from statistics) for efficiency. The effect of upweighting sample on the resulting model parameter computed, from which its influence obtained. Pruthi et al. (2020) introduces TracIn, inspired by Hara et al. (2019), which also measures the "influence" of training samples on a particular test sample by computing the difference in loss between predicted and actual test sample before and after training. TracIn is thus strietly posthoc, in the sense that the model f is altered after TracIn training. By contrast, kaBEDONN is posthoc since f is left intact, and can be finetuned independently based on feedback gathered from KaBEDONN.
SQANN. Our model KaBEDONN is a direct modification of Semi-Quantien Activation Neural Network (SQANN) TJoa & Cuntai (202T). SQANN has been developed universal approxima- tion (UA) of functions in which each node represents the k-th training data (xx(8), (()) in an ordered dataset. Activating the node associated to sample &o strongly (i.e. [uj (e - 1) causes SQANN to
fetch the a value that corresponds to ( yielding the correct output. SQANN is constructed by testing one sample after another from an ordered dataset. KaBEDONN similarly draws samples in sequential manner, although our system specifically organizes dataset by classes; see fig. [B)
Like SQANN, neuron activation in kaBEDONN is also computed with double selective activation daa- Given selective activation π (I, a) = atz2 and Super Gaussian sz(2, a) exp(-(z/a)")
4, 0.5, then desa (I,, 1,, (22, r) (1 x T (xx an) r x (g(2, 02). This is used by Node object (see appendix, pseudo-code 2
Nodes and a values. Similar to SQANN, KaBEDONN stores fingerprints of samples in nodes and stores their corresponding labels as a values. When strong activation is detected (i.e. act > Tod), the value of the strongly activated node is fetched for the computation of output. Interpolations are also performed for weak activation, and the interpolation method is customizable. In this paper, the interpolation choice is simple: a value of the node with the strongest activation will be used (even when its activation does not exceed Tad).
Differences from SQANN. Although SQANN is constructed to remember training data output per- fectly, it comes with some caveats. For example, layer size is not controlled. A layer can grow arbitrarily large and imbalanced. KaBEDONN, each layer has k nodes (except possibly the last layer) as the name suggests. Furthermore, it has a "bifold embedding" structure: kaBEDONN has layers of main nodes/neurons like DNN, but each node may contain one or more sub-nodes. Collectively, the sub-nodes form the second fold of lay- ers as shown in fig. [ The UA property is no longer available in our model, replaced with the capability to identify explanatory training data samples in the three contexts previously mentioned.
the other hand, is designed with k-width, i.e.
3 KABEDONN
In this section, we introduce our main model and illustrate it with the donut toy data, kaBEDONN's parameters are adjustable but, here, we focus on presenting the concept and only test few particular choices like 41, 42, 7 = 0.1, 0.5, 0.2, Tad = 0.5, Tact = 0.95 (admission and activation thresholds respectively). Optimization of parameter choices left for future studies. This section is descrip- tive; more details are available in pseudo-codes and full codes in the main code repo
At the core of KaBEDONN, some technical specifications are briefly described as the following (see the appendix for more details). A node object consists of (1) the main node and its key (main key): they are the fingerprint and index (yOnode, idd of a data sample respectively (2) the well-
4

---

Under review as a conference paper at ICLR 2023
represented nodes W Ru and (3) the sub-nodes Sn where 1 denotes layer and i the i-th node in that layer. Other concepts include the normalized node-relative vector that store information as a relative vector rather than full vector, elastic set of size kwidth (to limit layer sizes) and finetuning. In essence, they serve to distinguish nodes that are representative of certain classes from sub-nodes that correspond more ambiguous data. kaBEDONN helps to identify such data for the users.
Ordered dataset for classifications. Dataset is ordered in folders, where each folder corresponds to a class label (interchangeably y0). This is compatible with ImageFFoldrr arrangement from pytorch. For a class c, the perceived order of data samples depends on python sorted function. For a given set of parameters and data ordering. the construction will be deterministic. We denote data sample with its index (y0. idx) (c. idz) with its input x: (dd0 whichever convenient in the context. As illustrated in fig. BB), data samples are put into a queue in a "transverse" manner: fetch the first samples of all classes, then the second samples of all classes and so on (unless the class has no more samples). This ordering helps kaBEDONN to query explanatory data precisely and quickly, making the explanation adjustment process easier.
A LayerNodes object is an ordered collection of nodes in layer of kaBEDONN that has an as- semble receptors method. A receptor R₃ is a matrix assembled from the fingerprints of all main nodes in the layer. It is used in the intermediate computation of the activation Uy of some layer L;; see appendix pseudo-code 3 More precisely, each R₂ is used in the 1-th iteration of kaBEDONN forward propagation, shown below in its recursive form:
(I, == 2x(x)= dee (1((==1(z) - Ro)/N-
(1)
where x is the input, N₁ = |oa| + c and c a small number to prevent division by zero (i.e. for smoothness). The term [1(0;-1 - Ra)/N. ||| is called normalized stimulation.
KaBEDONN Training. The training is essentially a series of LayerNodes constructions. In each layer, the goal is build an elastic set with size kwidth i.e. to choose main nodes, as the following. As seen from filter_ by- layer_ activation() function in pseudo-code[] training sample (y0, idz) is admitted into the clastic set if it does not activate any previous nodes that have been chosen as the candidate main nodes i.e. all i components of activation [ui]s < admission filtering, illustrated in appendix fig. A(..
Tad- Let us call this process the
KaBEDONN's ss bifold embedding. Recall that SQANN Tjoa & Cuntai 2021) achieves UA by an extreme form of restructuring: all layers beyond the collision layer Le are destroyed, and the sample T. that causes collision is then added to le so that it contains fingerprint that will overwhelm the activation that results in collision and inaccurate prediction. KaBEDONN is not as extreme. A sample not used as main node is tested for bifold activation. This mechanism either (case 1) sets the sample Ze aside as a well represented node if its label matches the main nodc's. ss see appendix fig. B), dotted-arrow numbered 6 (case 2) embeds the nrvec of I. sub node (appendix fig. GB), dotted-arrow numbered 5.9); this is similar to collision in Tjoa & Cuntai (202T), and usually occurs to aforementioned boundary data. Future stimulation by xe will strongly activates this sub-node, causing it to return the y0 (corresponding to xx)) that has been stored as a, hence yielding the correct prediction.
Obtaining representative data samples as explanations. Finally, how do we obtain explanations like fig. [[])? In essence, forward propagation (see pseudo-code 1 is used to retrieve the explana- tions. When strong activation in the main node or sub-node are detected, the fingerprints stored in respective nodes are used to fetch the relevant data. Otherwise, interpolation is performed (in this paper, the most strongly activated node is used to fetch the main and supporting explanations).
Donut Toy Data for Illustration. We apply kaBEDONN with kwidth 8 on 120 data points. Each data point (at, y0) E R², x {0, 1,2) resides within a dataset with donut-shaped distribution as shown in fig. B); see full version in appendix fig. R[ A). Each point also belongs to one of the three classes 1,1, coloured black, dark grey and light- light-grey respectively. Used as predictive model, the accuracy performance of this particular kaBEDONN instance is 1.0/0.925 on training/test datasets respectively, although larger KaBEDONN easily fits the data perfectly. The final model has 3 layers (appendix table 2 Out of 120 data points, 20 are automatically filtered and embedded into the model. Also see appendix subsection on Toy Donut Data for more details
How exactly are main nodes, vell-represented nodes and sub-nodes distributed? See fig, B(A.)) A main node is labeled magenta triangle and it represents several other data samples in its prox-
5

---

Under review as a conference paper at ICLR 2023
(A)
(B)
X
O
Man
%
X
X
X
rode
Figure 3: (A) Illustrative mini toy donut. (B) Donut toy dataset with three labels (0 black, 1, grey, light grey). Best viewed electronically (zoom in; also, full version in appendix).
Table I:
Accuracies comparison when KaBEDONN is applied as a predictive model with various k widths. CNN denotes the original performance (no kaBEDONN).
kwidth
16
32
48
64
CNN
cifar mnist
train acc 0.987
0,996
0.999
0.999
0.948
test acc
0.888
0.913
0.925
0.926
0,951
train ace
0.965
0.993
0.998
0.999
0.861
test acc
0.681
0.722
0.721
0.722
0.776
imity. All data samples linked to this particular main node by magenta lines are well-represented, hence they are called well-represented (WR) nodes. As the name suggests, a training sample used by kaBEDONN as a main node is expected to represent its locality well i.e. it has the same label as its WR nodes.
However, "locality" is not always well delineated. See for example fig. [(A) "sub node". The sample belongs to the circle class but stays near an i.e. it is the so-called boundary data, close enough to activate the x's main node. kaBEDONN bifold embedding is exactly used resolve such situations: the circle at the boundary might be embedded into sub-node in the "upper fold". See appendix for more explanations the full donut dataset.
Sub-nodes.
Explanations for XAI. KaBEDONN quickly tells users which training data are "similar" to the data point being tested. These representative samples act the explanations. Although there is no individual image like fig. A[A) to be shown for toy donut data, KaBEDONN points to the directory of the explanatory data (text version of explanations printed by kaBEDONN like in appendix fig. 10). See appendix for more details on (1) fig. 3 B) where we explain more kaBEDONN nodes w.r.t the entire dataset with visual references (2) main and supporting explanations on donut data similar to fig. A).
EXPERIMENTS AND RESULTS
We apply kaBEDONN on MNIST, CIFAR and ImageNet datasets. In fig. 4 of Koh & Liang (2017), 900 images from two ImageNet classes (dog and fish) are used to compute their influences on test samples. We refer to this process as the class-selective explanation. We perform similar but modi- fied experiments.
Convolutional Neural Network (CNN) for MNIST. MNIST is relatively simple dataset thus we use a simple CNN with successive conv+Rel.U modules followed by conv (fully connected layer) with 4 epochs of training, batch size 16 (details in the available code). The training/test accuracies attained are 0.948/0.951 respectively. kaBEDONN is constructed us- ing ((re, 50) 2. CNN(s): (x. y0) E D).
MNIST with class-selective explanations. For MNIST, we demonstrate kaBEDONN with class- selective explanation
class 4 and here, especially because handwritten 4 and 9 might look similar. We can thus see the effect of node and sub-nodes activations clearly. Fig. shows samples with different types of KaBEDONN neuron activations. First MNIST sample (first row) shows

---

Under review as a conference paper at ICLR 2023
(A)
MNVIS
cample
Man nattl
Wi/ makks
Act 15 Bucidear dit
CEAa sampl:
Witbbb nodes
4
4
4
9
4
4
4
Fmpery Wille noder
4
4
4 4
9
1
9 4
9
4
9
9
9
9
9 9
9
Figure 4: Display of image samples associated to main/WR/sub nodes for XAI. Best viewed elec- tronically (zoom in) (A) MNIST (B) CIFAR, sub: (0) indicates that subnode is (not) activated. sub:0(1), then WR(sub)-node associated to the main node are displayed. Act Euclidean dist is similar to fig. Koh & Liang| (2017), except each plot for a single sample and activations of neurons, act, are our "influence" values. Orange dots are for the activations of sub-nodes. It shows only a loose trend: shorter Euclidean dist. appears to be weakly correlated to higher activation.
standard activation: the very 1st training sample activates the main node that corresponds exactly to itself. The 2nd sample is a digit that activates the main node of a digit 9. However, thanks to bifold activation mechanism, the sample correctly activates the sub-node that refers itself (hence marked sub:1). yielding a correct prediction. The 4-th sample activates a main node corresponding to another digit 4. While they do not necessarily look the same visually, they are similar enough according to the CNN. The 6-th sample is similar to 2nd sample, and 7-th to 4-th. Act vs Euclidean distance plots like Koh Liang (2017) are also shown, although there is no conclusive trend to be found.
Relevant data samples for XAI. As previously mentioned, we present relevant data as the explana- tions for model's 's prediction. In fig. 4 whenever no sub-node activation occurs (sub:0), the image associated with the main node presented as the explanation, and the associated images in its WR nodes are the "supporting" explanations. Otherwise (sub:1). the main node is only "similar but not quite the same", and the sub-node is presented as the actual representative explanation.
KaBEDONN with the whole MNIST dataset. Used as predictive model, the accuracies of KaBEDONN models with different kwdith is shown in table Compared to its base model CNN, KaBEDONN overfits the training dataset. Fortunately, they able maintain reasonably high test accuracies given certain hyperparameters. i.e. kaBEDONN has generalization power. This is a very promising prospect it opens up new research direction: is there latent encoder different from CNN that can be used with KaBEDONN to improve both training and test accuracies? Although UA property in Tjoa & Cuntai 1021 is not achieved, generally, kaBEDONN's accuracy on the training dataset is high and improves with kwidth.
More details on similar experiments on CIFAR can be found in the appendix A.4 Some XAI results are also shown in fig. [ As predictive model, the accuracy results are also equally promising (table Furthermore, a brief look at fig. [CIFAR shows a promising trend for further research: for example, row 1 and 2 both classify images of a deer, but the first main node clearly reflects deer in a green background with plants/grass. With good latent encoder, KaBEDONN demonstrates a good potential for fetching mcaningfully similar images explanations.
CIFAR.
Data ordering as hyperparameter. KaBEDONN can be used as a complementary predictive model. For each kwidth, different accuracy results (on test data) are obtained when we shuffle the data ordering randomly. The results are shown in fig. 5 This reflects the idea that some data samples suitable main nodes, while some others more suitable as WR- subenodes. The intuition clearly illustrated by a boundary data. Suppose sample data 29 resides near the boundary between different classes. If accidentally place 26 early in the ordering, it might acci-

---

Under review as a conference paper at ICLR 2023
dentally become main node. Many samples around the boundary will activate za and this might not be desirable because 25 over-represents different classes. Many redundant sub-nodes may be attached to its main node, although the sub-nodes could have been well-represented by only a few other nodes that provide better representation. However, the suggestion above is only intuitive; we believe there may exist boundary with extreme variations that may need such wver-representation. In any case, fig. Simplies that it is possible to create a better representation for kaBEDONN through appropriate ordering, and this is a potential direction for further studies.
ImageNet. Experiments similar to class-selective expla- nations on ImageNet have been performed. Generally, the results are similar to MNIST and CIFAR (fig. 4. thus we will briefly discuss them here. Fig- AT) shows partial results for ImageNet; the full results can be found appendix fig. IT in which two classes of images are ar- bitrarily chosen, and 900 images are used each, just like previous similar works in this topic. KaBEDONN works expected: when there is sub-node activation (sub:0 scenario), main nodes are queried as explanations along with "similar" images (or the images in the WR nodes). Otherwise (sub:1 1 scenario), the image used to construct the sub-node is returned. For training data, sub-node ac- tivation typically returns the correct class by querying the node associated with the sample itself.
0.033
0000
0.04
0:64
0.03
0.00
0:08
0:58
6:54
Figure 5: Test Acc. vs kwdith. For the same kwidth, different accuracies are obtained due to different data ordering.
KaBEDONN model on the entire million images from ImageNet dataset is trickier. Ideally. kaBEDONN is de-
signed to extract subset of representative images and use them as its representative nodes. How- ever, the large dimensions of features in ImageNet coupled with the large size of dataset have caused some inefficiency. While kaBEDONN has helped resolve the imbalance layer size (with kwdith lim- itation), we believe further improvement in the model can be aimed at improving its filter efficiency. One way is to modify the preprocessing pipeline e.g. by reducing the dimension and subcatego- rizing classes, which is especially relevant because many classes in ImageNet has overlaps (e.g. many breeds of the same species of animal, dog). A controlled selection of representative data samples could then be performed, e.g. by repeated application of KaBEDONN on ImageNet data subsets and collecting only main node samples. Our temporary results (not shown) indicate that even with smaller dataset, the large number of labels being trained at once into kaBEDONN harms the construction process, further supporting the idea that subcategorizing labels is the better potential solution. Future studies on efficiency will be helpful.
CONCLUSION
We have introduced KaBEDONN, posthoc model-agnostic method that queries relevant data as explanations for model prediction. Systematic ordering of training dataset is required in our sys- tem: we aim encourage more transparent and organized way of building a model that puts more emphasis on individual training data. Advantages: (1) explanations are user-friendly since they are queried from familiar dataset (2) explanations are adjustable and easy to finetune e.g. by rearrang- ing data order removing problematic explanations. Hence, kaBEDONN system improves the query/feedback process in XAI.
We have tested kaBEDONN on raw donut data and images that have been encoded into vectors by CNN. From here, several directions for further studies are available as the following. (1) Systemat- ically study the precise effect of dataset ordering. (2) Modify kaBEDONN such that UA and CF in Tjoa Cuntai (02I) are recovered. (3) Improve its generalizability so that kaBEDONN not only works XAI method, but also a predictive model. This includes finding the optimal latent encoder (e.g. transformer rather than CNN) that works well with KaBEDONN.
Listing 1: Pseudo-codes for kaBEDONN. Some components are in the appendix. def fit_date (X, Y)
2 # inin KaBRDONN 1oop
3 # eset: elastics iastic_set
8

---

Under review as a conference paper at ICLR 2023
4 q create ()euee)
I_now=1 cuzrent layer caet, altset=[),
7
while is not empty or len (eset)> >0: eset, htt..ggghhn-mmlllllastt.ccc eset, setaside, nodes layes_
8
(q)
9
filter_Bh baaaerr_b__laye_aattiiii exteed
(eset, 1_now)
11
12
13
if 1en (enset)==*_wiidt
14
info Alit:AAAil_SSD___P11 (1__oo
16
nodes_layer)
17
(1 now, altset)
18
eset, [ltset= reset Inowt=1
20
21
def bifold_activation (I, a
22
x, darrch_daaaaallae
for (0, in enum (X, Y):
beode=-ge__cttivated_eee
(x,
25
y0_node, ftax-aanddemaii__ee...
26
ANSERT_SUENORE if node if INSERT SUBNCDEE
nrvec=compute nrvec
nrree (x)
anode.new_subne (nrvec, y0, idx)
30 else:
31
anode www.sewwwwoddweeeeeeeee appedd (y0, idx)
32
def (x)
34 act,, act_pre None, x
35 for 1 in range (1, 1--_aaay):::
36
receptoss-ascchlleeecpposs ()
attrnomaalieeeeeesmmmtttttoorr.e http:aaaaatattttttteeeoeeeeeeeee receptoos)
if any ann(acttthreahold(1))
39
act_id - argmax # now get act detivated_node
40
41
prred,, NODE INFO anode. a fart pre)
NODE_INFO
43
OUTPUT INFO = (act_ NODE INFO etc)
44
return yred,, OUTPUT_INEF
45
hodate_interpolattooo_nttt
customizable
act_pre=act
y_pred, CUTPUT_INFO use_interpa usc_interpolation/
48
return y_pred, OUTPUT_INFO
49
50 def filter_by filteer_b www.ayeeaattvaaiionoooooo (eset, 1):
X, fetch_data (eset)
52
& nodes_layez LayerNodes ()
53 new new_eset, set.setssttssdd:_...
54 for (x, in (x,
55
act=commuteeactivvttii INSERT_NOU if len (nodes_ or if all aat if INSERT NODE: INSERT_NODE:
(x)
57
58
nodes laades_layer.o new_eset (cset else:
62
sctaside. (cset (i])
63
return new_eset, setaside, Lodes_layee
ACKNOWLEDGMENTS
Anonymous for

---

Under review as a conference paper at ICLR 2023
REFERENCES
Google apologises for photos app's racist blunder, Jul 2015. URL https:/ www.ww.b.. .bc. com/ news/technology 33347866
Amina Adadi and Mohammed Berrada. Peeking inside the black-box: A survey on explainable artificial intelligence (xai). IEEE Access, 8:52138-52106 2018. doi: 10.1109/ACCESSS20.. 2870052.
Alejandro Barredo Arrieta, Natalia Diaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez. Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (xai): Concepts, tax-
onomies, opportunities and challenges toward responsible ai. Information Fusion, 58:82
115, 2020. ISSN 1566-2535. doi: https:/doiiorg//00000.....ninin 7/www. http::/wwcccccaccccccooooooooaaa
URL http:
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer- wise relevance propagation. PLOS ONE, 10(7):1 07 2015. doi: 10.137 /joumaallllll.oe...... 000014.....110......
Utt.ttts::/ //ottts:://o... http://///oooo....1131131 fouronnnnan pone. 01 0100140
L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 2018.
Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Data cleansing for models trained with sgd. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL pttps: http://ww...aaaas.oo..oo///////a pttppp//pppp///1////eeeee 55161156666011102220000088888888
pdf
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
doi: 10. N0.1109/CVPR.2016.990
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885-1894.
PMLR, 06-11 Aug 2017. URL sttps:://roceecings www. ww...ppress/////kohh11
html
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: htttps:://proceeddnng neurips........................s wttpppp..c/ppppe/21//1/.. 8a20a8 621 17221786667 978632d www36fd2266676611116 w PPpp:228666777-Papp..ppo pdd
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Re- search, 01:2825-28800 2011. URL nei sheors. html
http://iiii..aa.....oo/oaaa/llaa stable/modules) www.--iteeern.... org/
Es- data influence by tracing gradient descent. In NeurIPS. URL httpps://proceedngg.s neurips. wwwp:nnwwrrr.ppppeeeeeeeeeeeeeee cc//paper/222//aash e (c33433c333E2333522eee...... Abstraaatttccctttttttttaaaaaaac. html
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. timating training
2020.
Marco Tulio Ribeiro. Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '16. pp. 1135-1144, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322. doi: 10.1145/2939672.
2939778. URL httpt:tt//c/c.. .rroooooorrgg//// 0.5.///9900..114/99339777222...0
10

---

Under review as a conference paper at ICLR 2023
Bernhard Schölkopf, Ralf Herbrich, and Alex J. Smola. A generalized representer theorem. In David Helmbold and Bob Williamson (eds.), Computational Learning Theory, pp. 416-426, Berlin, Heidelberg, 2001. Springer Berlin Heidelberg. ISBN 978-3-540-44581-4
Yi Tay, Vinh Quang Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, The Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. Transformer memory as differentiable search index. ArXiv, abs/2202.06991, 2022.
Erico Tjoa and Guan Cuntai. Two instances of interpretable neural network for universal approxi- mations. ArXiv, 12.20662,, 2021.
Erico Tjoa and Cuntai Guan. survey on explainable artificial intelligence (xai): Toward medical xai. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-21, 2020. doi: 10.1109/ 109/
2020.3..22030223114
Alvin Wan, Lisa Dunlap, Daniel Ho, Jihan Yin, Scott Lee, Suzanne Petryk, Sarah Adel Bargal, and Joseph E. Gonzalez. (NBDT}: Neural-backed decision tree. In International Confer- Learning Representations, 2021. URL https: INCLVeEppINE,
net. net/forunffr
Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: httttppss//prcceedingg.. neurips............. cc/ pdf
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CV (CVPR), June 2016.
A APPENDIX
A.1 GENERAL INFO
Adjustment of Explanation. Recall that of kaBEDONN's objective is to allow the Adjustment of Explanation. Some relevant points:
1. Through kaBEDONN, users' feedback can now be used to modify the dataset; dataset can be used to retrain f and the resulting model can be compared for further analysis. In this paper, present only the core system and algorithm design. We leave contextual user-dependent data analysis for future studies.
2. In fig. B)), the bullfrog image appears activate node that responds to dark background texture
3. Unlike the process shown in fig. BB).. we might want to consider deleting problematic data sample altogether. Furthermore, many more samples can be adjusted before the model f is reconstructed.
4. Notice that a complimentary pipeline to collect users' feedback will be needed; the user interface for displaying explanation is not polished in our code, since we focus mainly on the kaBEDONN model.
More related work: differentiable Search Index (DSI) Nearest Neighbour. Data samples in the proximity of each node's 's locality will activate the node strongly, i.e. KaBEDONN uses the concept of distance. This is reminiscent of the concept of distance in nearest neighbours methods and, like SQANN, it has the properties of instance-based learning approaches. However, nearest neighbours methods have been said to be non-generalizing! Pedrreooa al. (2011). Like SQANN, KaBEDONN also has a non-linear multi-layer structure, thus it does not suffer from non-generalizing property.
Tay et al. (022) has been used to fetch document id (docid) j given a text query q. In our system, data index (30, idx)) is our "docid" while kaBEDONN's vector input is our "text query". In its standard mode, kaBEDONN returns the index of relevant data sample (y0, idx) along with the predicted class label.
11

---

Under review as a conference paper at ICLR 2023
(A]
Cew
xCa BH
Coss1
fhh www.eeeeee.....o.. www.eetacttttatttoooeeeeooo activation INSERT
Clnx2
2m2 2w3
CaspC
Clos1
Figure 6: (A) Step I to 4. Data samples are admitted into the model through admission filtering. The samples distinct, they don't t activate each other's nodes. (B) Sample 6, 9 are admitted through the mechanisms in bifold activation. Note that sample 5 and 9 are similar to sample and respectively (illustrated with slightly different colours), but they don't share the same classes, hence they are embedded as sub-nodes.
An elastic set (denoted as eset in pseudo-code 1 is flexible set in which data indices can be added and removed. The main purpose of elastic set is to provide a running list of data indices that will be used for constructing the main nodes of kaBEDONN layer. Repeated application of filter by layer activation (see pseudo-code is performed until an elastic set of size kwidth is obtained; this helps control the layer size.
A node object consists of (1) the main node and its key (main key): they are the fingerprint and index (yOnode, idL of data sample respectively (2) the well-represented nodes W Ru and (3) the sub-nodes Sy. where I denotes layer and the i-th node in that layer. See appendix pseudo-code
As seen from pseudo-code I W Ris populated by training samples that trigger bifold activation but require no INSERT NODEE. In other words, each such sample (y0, idz) is well-represented: it activates a main node and labeling the sample with yOnode is correct (y0 = yOnode). The fingerprint memory. As
of well-represented sample is not stored in the model, helping the model in the donut example later, through KaBEDONN's filtering approach, only a fraction of the whole training dataset is eventually embedded into the model.
St is similar, except y0 * yOnode in which quired for more accurate representation of the sample. Intuitively, this occurs at the boundary between classes: the sample activates a main node with a different class yOnode (i.e. to activate the particular node) yet they not the same closer inspection. The sample's fingerprint is stored in the model. Further- more, tunable Tact will be associated each sub-node. High acti- vation threshold is used to reduce the likelihood that samples well- represented by the node (labelled y(node accidentally activate this newly admitted sub-node (labelled y0) i.e. reduce the likelihood of making the wrong predictions.
sub-node is re-
f((, Iv:l
similar enough
nrvec A/|A|
Main node [vd]
f((x .
Figure 7: Normalized node- relative vector.
A normalized node-relative vector (nrvec in pseudo-code 1b is
shown in fig. 7 During sub-node embedding, nrvec is stored as
the fingerprint (unlike main node which stores activation Vz). Why?
Intuitively, two samples that activate the same main node are similar iee their main vectors are close each other. Nrvec helps segregate them away to better distinguish their relative positions in the higher dimensional space.
The intuition is simple: we want a layer with main nodes that are representing distinct samples. They might even be from the same class but possess features that are distinct enough from each
12

---

Under review as a conference paper at ICLR 2023
other e.g. different breeds of cats. This mechanism thus avoids redundancy in the sense that no two main nodes in the same layers are so similar that they activate each other's ' nodes.
Finetuning. After training, we run through the training data once for finetuning. For each sample, check for the correctness of classification against the ground-truth class, primarily to double-check samples in W nodes. If it wrong, finetuning is performed recursively until the prediction correct. During this process, (1) if sub-node is activated, then Tact is increased. The intuition is also simple: we reduces the sensitivity of the sub-node so that wrong samples do not easily activate it (2) if no sub-node is activated, then a sub-node is added into the node (similar to bifold activation with INSERT- SUBNODE). See remark in appendix; also see the main codes.
Fine-tuning in this paper is performed by going through the entire training dataset once, as men- tioned in the main text. Remarks. (1) the process will terminate because either Tact will eventually be high enough or a sub-node is added. We exclude unresolvable ill-defined dataset that contains two samples with exactly the same input values but different labels. (2) that UA property in (Tjoa & Cuntai (202)) is no longer applicable here i.c. training dataset is no longer perfectly modeled (we empirically show later that the accuracy is still very high). This is because after one fine-tuning run, (A) the increased threshold might cause the node to be too insensitive for samples that previ- ously depend its activation (B) additional sub-nodes might be too sensitive they are stimulated by previous samples that used to only activate the main node for correct prediction.
(A)
(B)
30.0
0.0
0.-
-0.5
-055
-1.0
1.0 -10
-0.5 0.0
-3.0
0.0 0.5
Figure 8: Best viewed electronically. (A) Donut toy dataset with three labels (0 black, 1, grey, 2 light grey). Magenta triangle: a data sample embedded as a main kaBEDONN node. Node linked to magenta triangle by magenta line is data sample well-represented by the main node. Node linked to magenta triangle by a dotted red line is sub-activation node that activated the main node but has different value. It usually occurs at the class boundary. (B) Test data samples (coloured marks, magenta (y0, idz)) and the main nodes (small magenta triangle) they activated (foreground), superimposed on the transparent version of fig. (A) (background). Red pentagon mark denotes wrongly predicted data.
A.2 TOY DONUT DATA
Donut data is generated with the following procedure. Draw an array T of size N from uni- form random distribution with U(0 2n N_ --p ((nos()).,sin(TT))
N l). Then X is the ordered set containing X i =, 1....,N N) and is its corresponding label Y = {argmaz, (V)) [Vi, V2. V3l [cos(T,), cos(2T,), cos(3T,)). The corresponding test
i = 1...., N) where data samples are generated similarly by perturbing X with U[[[tads ted)). In our experiments, N 120, Lad = 0.1.
Main nodes. As before, main nodes are magenta triangles in fig. nodes nearby.
8
They represent several other
Sub-nodes. As mentioned in the main text, "locality" is not always well delineated. For example, fig. g Node (0, 51) (pointed by a red arrow). The sample has ground-truth label 0, but it is stranded somewhere in the boundary between class label 0 and 2. With existing training samples alone,
13

---

Under review as a conference paper at ICLR 2023
=== Index:free input! ===
Index:free input!
x: [-1. 002]
X: [ 0.007 -0.83
y0:-1
y0:-1
activation:standard
activationssuaactivttin
y_pred:1
Y_pred:0
layer:1
layer:1
node_idk:
node_idx:7
dala_ (ola_idx((1.2) 2)
data dota_idx:(0,.55 51)
data sra_locc('claas/] '17.npy')
data iaaallocc''class0 70.npy']
parent_nooee moodeloccssmee as data_loc
crvent_node_loc((bbar '110.npy']
Figure 10: Sample explanations for donut toy data. Yellow highlight shows filenames of relevant explanatory data samples. This is equivalent fig. [[), but "relevant data", except, rather than images we show the positions of the data in the folder).
there is not enough information to label any samples around the boundary. Furthermore, (0, 51) close enough to activate main node (2, 4) but is not well represented, since (2, 4) belongs to class 2. Hence, KaBEDONN's mechanism will embed (0, 51) into the "upper fold": it becomes the sub-node.
Three types of explanations for XAI mentioned in the main text. (1) Overarching visual explanation showing KaBEDONN nodes w.r.t the entire dataset can be seen in fig. 8 KaBEDONN quickly tells users which data samples are the representative samples relevant to the data sample being tested. Fig. B) shows test sam- ples (coloured x marks) predicted using KaBEDONN.
Small magenta triangles are kaBEDONN main nodes ac- tivated by the test samples and red pentagons indicate wrong predictions. Each small magenta triangle in fig. B(B) exactly corresponds to big magenta triangle in fig. A). For example two yellow marks i.e. test ples (2, 0). (2, 32) activate a main node the corresponds to (y0, idx)) (2, 0) in the training dataset.
(2) Apart from visual reference, report is printed to provide users with more explanations, as the following. For example, let inspect yellow x marks test sam- ple 32) in fig. B). The explanation for its prediction reported in the following format [T/ Fasplee ==> mainnode and list of "like" nodes e.g. (2, ==> (2,0) and [(2, 1), (2, 2), (2, 7), ... (like main and sup- porting explanations similar to fig. [()).. Like nodes are simply well-represented nodes corresponding to the main node. For users, they are data points that are "similar" to the user's sample of interest.
Figure Similar fig. donut.
except for big
(3) In our codes, submode find and setting find, what to activate_ nodes allow user to further find the exact fold- er/name of the training data file that corresponds activated node, as shown in fig. C). Yellow highlights show the exact data folder name and file name. During sub-node activation, parent. node_ loc points to the the exact data folder name and file name for data sample of the related main node.
the
Big donut data is similar, except y; = cos cos(jT) (T)) for = 1,2, 3, 4,5 i.c. has five classes of data samples. Furthermore, N 1000. See fig. 9
14

---

Under review as a conference paper at ICLR 2023
Table Layer information of KaBEDONN constructed on Donut data. y0, ida denotes the indices of data sample that has become kaBEDONN main node. n(wr), n(sub) denotes the number of well-represented data sample and sub-nodes respectively.
LAYER
LAYER 2
LAYER 3
y0
idx
n(wr)
n(sub)
y0
idx
n(wr)
n(sub)
yO
idx
n(wr)
n(sub)
0
0
16
0
0
5
4
0
28
1
0
6
1
2
6
1
0
2
0
14
0
0
7
1
0
0
1
15
1
1
10
0
0
0
2
10
0
2
10
2
0
1
2
11
0
0
12
1
0
2
3
4
0
0
13
3
0
2
4
9
1
0
19
3
0
Table 3: Layer information of KaBEDONN constructed on the whole MNIST data. We show only
layer 1, and the last layer, layer 8. Notation similar to table
LAYER
LAYER 2
LAYER 8
y0
idx
n(w))
nsuub)
y0
idx
n(wr)
n(sub)
y0
idx
n(wr)
n(sub)
0
0
989
7
8
1
2283
429
3
2102
0
4107
4
9
1
2796
762
2
2523
2
0
106
0
2
2
710
338
4
2917
0
0
844
0
3
2
2257
8
0
2949
4
0
1229
0
7
2
529
37
3
3013
3
8
2
3281
6
0
2953
14
O
3
656
1026
8
3617
7
0
2140
I
3
1311
11
9
5582
8
0
338
483
2
3
2367
152
9
0
80
3
3
1365
5
O
298
1
4
3
56
496
2
13
0
5
3
925
131
4
512
7
3
1094
10
5
44
1
8
3
95
199
6
757
19
3
1175
7
652
5
0
4
1280
14
A.3 KABEDONN ON MNIST
Most results already shown in the main text. Using kwidth 16, kaBEDONN with 8 layers have been constructed. 15147/60000 samples are embedded into the system. Table shows more details the nodes after KaBEDONN construction.
A.4 KABEDONN ON CIFAR
For CIFAR, the CNN used is ResNet1 8 et al. (0066) with its FC layer replaced 512 x 10 since there are 10 classes in CIFAR. Similar to MNIST, CNN is fine-tuned for 4 epochs on batch size 16 with further hyperparameter search. Training/test acc. attained are 0.861 and 0.776 respectively.
CIFAR with class-selective explanations, CIFAR explanations reasonably similar to the expla- nations we see for MNIST, as shown in fig. Sample I (row 1) shows a standard activation of main node by deer image itself; here, the main node image is served the explanation. As pervi- ously mentioned, sample 2 similar, except it shows deer not on a green/plant background. Sample 3 and 6 show sub-activated nodes. The deer/truck activate main nodes of the other class. With
15

---

Under review as a conference paper at ICLR 2023
Main node
Wi/SSbb nodes
Act Eucl dean dist
sample
Emjty WR/Sub nodes
Emply WiR/Sub nodes
Figure 11: ImageNet display of image samples and kaBEDONN exlpanations, similar to fig.
sub-nodes activation, KaBEDONN recognizes the samples as they are. The other samples provide relevant images explanations previously explained. Also, Act Euclidean dist plot shows better pattern; nearer to the sample (i.e. nearer to distance), the activation values tend to be higher.
KaBEDONN with the whole CIFAR dataset. The results have also been tabulated in table [
A.S KABEDONN ON IMAGENET
We perform class selective explanations on two classes ImageNet, and then show the some results after explanation adjustment.
A.6 MORE PSEUDO-CODE
Listing 2: Pseudo-codes for kaBEDONN's individual nodes.
class Node:
2
def init():
3 main_node, main key None, None
4 sub nodenode____
is dict
wr nodno II is 1ist
7 def setprblle__u_reepttrrrtttss (self):
8
# Node's method non-memoized version of this function
10
receptors = [] $ shape: (n, act_pre dim)
11
temp_ide []
12
for (y0, idx), nrvec_ in self. sub surf.suu_oddee_ jub_nooee__.iemm () aappotorr.pppeddnnnnnnn [nrvec_["nrreec]] ['nrvec']] temp_idx. (y0, idx)
13
15
receptors = ap.array (receptors)
16
return receptors, temp_idx
1
def Forward act_pre): & Node's method
2
act_pre is the activation values
16

---

Under review as a conference paper at ICLR 2023
MN.ST
Man
Wh/Sub Euclicean
Enecty MRySabasee
Impty MTcSubnncee
Figure 12: Similar to fig. but after adjustment of explanation is done in fig. BB. The results appear to be stable the sense that the explanations for other images are not changed by a local adjustment.
3 # propagated from previous Layer
4 y_node, idx- nrvec act_pre self.main_nod. nrvec nrvec/norm (nrvec)
5
# actually, we alsohandle 0 norm
8
9 receptors, temp idx idx assembee Bectunnbl_____eeeeeeee_ee sub_receptoora)
10 side act np. aap.matmul (receptors, nrvec. activation_thresho...
[subnode [accivattooo_ttrrcooool for subnode in sub_.subboddda
12
13
$ we do convert it to numpy array
14
15
if np. (side__aaatt> iattivvttttooooteeeeeee d) receptor - activation ttressoodd
np. ar.argmax (s acc
17
18
pred temp idx receptor idx [0]
19
NODE INFO update
node node info
else:
y_pred y_node
22
return y_pred, NODE_INFO
Listing 3: Pseudo-codes for kaBEDONN's layer.
1 class LayerNodes:
2 def init():
3 node_list - [] list of Node objects
def desemble__eceptteee (self,)
2 # LayerNodes' method
3 non-memiized version of this function nodes =
5 for n in self. node_
6 if len (nodes)==0:
7 nodes [n.main_node]
8 else:
17

---

Under review as a conference paper at ICLR 2023
9 nodes np. concatenate(
10
(nodes, (n.main_nodel) axis=0) return nodes convert to numpy array
18


Reviews:
[{"review_id": "eFw-8LnX_X", "reviewer": "Reviewer_ZXST", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a method called kaBEDONN that produces \"relevant\" samples as a local explanation. ", "summary_of_the_review": "The paper has an interesting idea of generating more relevant samples as local explanations. However, the empirical evaluation felt incomplete (see above), and a user study to link relevance to interpretability is missing.", "strength_and_weaknesses": "Strengths:\n- The idea of generating more relevant samples as local explanations is interesting.\n\nWeaknesses:\n- No comparison to other post-hoc methods that generate samples as explanations, despite referencing them.\n- Method performs worse than a CNN (table 1). This could be acceptable, since this is a post-hoc method. Standard deviations would help in judging the significance of these results.\n- Interpretability evaluation a bit lacking -- are the sample explanations claimed to be more relevant generated by the method actually perceived as more relevant / helpful / interpretable by humans?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ABbs_zIMDe", "reviewer": "Reviewer_gXuS", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents kaBEDONN, a method for post hoc explaining neural network classifiers by identifying relevant data. It is inspired by existing sample importance-based explanations, such as the influence function and representer point. ", "summary_of_the_review": "Due to the numerous issues with the current draft, I vote for rejection. ", "strength_and_weaknesses": "**Strengths**\n\nThe paper tackles an important problem: the interpretability of neural networks. \n\nThe proposed architecture seems novel. \n\n**Weaknesses**\n\nThe writing quality is poor. Generally, this does not feel like a well-polished academic paper. I would recommend the authors to ask others not involved in this project to read the paper and identify points of confusions. \n\nIt is not clear how this model supports the three objectives described in Sec. 1, especially the latter two, which seem to be not demonstrated. \n\nThere is no comparison with existing sample-importance methods in the experiment. \n\nIt is not clear what conclusions can be drawn from Fig. 4 results. It looks like main node often is the same as data sample. Why should this happen? If the data sample is from the test set, there shouldn't be a duplicate in the training set? It's not clear what additional insights we could get from the WR/Sub nodes either. \n\nWhy are the results in ImageNet not centrally highlighted? I would expect those to be much more interesting than that on MNIST and CIFAR, but most of the results are on the latter. \n\nMinor: \n\nI recommend the authors to familiarize themselves with latex commands `\\citep` and `\\citet`. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject