Prototypical Self-Explainable Models Without Re-training
Authors: Srishti Gautam
Ahcene Boubekki
Marina MC Höhne
Michael Kampffmeyer
Venue: Accepted by TMLR ()

Abstract: Explainable AI (XAI) has unfolded in two distinct research directions with, on the one hand, post-hoc methods that explain the predictions of a pre-trained black-box model and, on the other hand, self-explainable models (SEMs) which are trained directly to provide explanations alongside their predictions. While the latter is preferred in safety-critical scenarios, post-hoc approaches have received the majority of attention until now, owing to their simplicity and ability to explain base models without retraining. Current SEMs, instead, require complex architectures and heavily regularized loss functions, thus necessitating specific and costly training. To address this shortcoming and facilitate wider use of SEMs, we propose a simple yet efficient universal method called KMEx (K-Means Explainer), which can convert any existing pre-trained model into a prototypical SEM. The motivation behind KMEx is to enhance transparency in deep learning-based decision-making via class-prototype-based explanations that are diverse and trustworthy without retraining the base model. We compare models obtained from KMEx to state-of-the-art SEMs using an extensive qualitative evaluation to highlight the strengths and weaknesses of each model, further paving the way toward a more reliable and objective evaluation of SEMs\footnote{The code is available at https://github.com/SrishtiGautam/KMEx}.

Published in Transactions on Machine Learning Research (05/2024)
Prototypical Self-Explainable Models Without Re-training
Srishti Gautam"
srishti grishl..goutam@@si...o
Deportment of Physics and Technology
U.T The Arctic University of Norway, Norway
Ahcene Boubekki
alcene..oubekkii@yyl
Machine Learning and Uncertainty Physikalisch- Techmische Bundesanstall, Germany
Marina M. C. Höhne
Data Science in Bioeconomy, Leibniz Institute for Agriculture and Bioeconomy Institute for Computer Science, University of Polsdam, Germany
Michael C. Kampffmeyer
michael. haataaal....mmmp/eeeeeruu.....k
Deportment of Physics und Technology UTT The Arctic University of Norwoy, Norway
Reviewed on OpenReview: https: // openreview. net/
bet/forunt forum? id= HU5DDU6666
Abstract
Explainable AI (XAI) has unfolded in two distinct rescarch directions with, on the one hand, post-hoc methods that explain the predictions of pre-trained black-box model and, on the other hand, self- explainable models (SEMs) which are trained directly to provide explanations alongside their predictions. While the latter is preferred in safety-critical scenarios, post-hoc approaches have received the majority of attention until now, owing to their simplicity and ability to explain base models without retraining. Current SEMs, instead, require complex architectures and heavily regularized loss functions, thus necessitating specific and costly training. To address this shortcoming and facilitate wider use of SEMs, we propose simple yet efficient universal method called KMEx (K-Means Explainer), which can convert any existing pre-trained model into prototypical SEM. The motivation behind KMEx is to enhance transparency in deep learning-based decision-makinn via class-prototype-baseee explanations that are diverse and trustworthy without retraining the base model. We compare models obtained from KMEx to state-of-the-art SEMs using an extensive qualitative evaluation to highlight the strengths and weaknesses each model, further paving the way loward a more reliable and objective evaluation of SEMst.
1 Introduction
XAI has become a key research area with the primary objective of enhancing the reliability of deep learning models (Yosinski et al., 2015; Tjoa & Guan, 2021). This domain has notably evolved along two parallel trajectories in recent years. One focuses on post-hoc methods (Ribeiro et al., 2016; Selvaraju et al., 2017), where the algorithms aim to explain the behavior of the black-box models after they have been trained. The other promising branch focuses on SEMs (Rudin, 2019), where the models are strategically designed and trained to generate explanations along with their predictions.
The easily employable post-hoc techniques have become widely adopted in recent works due to their ability to offer insights into any black-box models without retraining (Bodria et al., 2023). Nevertheless, the need
"qqual contribution
The code available httpe///tthab.com///ttillaaaaaaa

---

Published in Transactions on Machine Learning Research (05/2024)
for inherently interpretable models has taken some momentum fucled by the unreliability and high variability of these post-hoc methods, which inhibits their usability for safety-critical applications (Rudin, 2019). SEMs offer explanations that align with the actual computations of the model, thus proving to be more dependable, which is crucial in domains such as criminal justice, healthcare, and finance (Rudin, 2019). However, existing SEMs rely on complex designs based on large deep-learning backbones and require intricate training strategies. The associated computational and time costs limit their accessibility and sustainability.
We tackle this limitation by introducing simple but efficient method called KMEx (K-Means Explainer), which is the first approach that aims to convert trained black-box model into prototypical self-explainable model (PSEM). PSEMs provide inherent explanations in the form of reprssetprinntttiviv concepts, also called prototypes, in the latent space that can be visualized in the human--understandable inpul-space (Kim et al., 2021). These prototypes serve as global explanations of the model (this looks like that (Chen et al., 2019b]), and their visualization provides knowledge about their neighborhood in the learned embedding. KMEx keeps the trained encoder intact, learns prototypes via clustering in the embedding space, and replaces the classifier with a transparent one. This results in an SEM with similar local explanations and performance to the original black box model, and such enables the reuse of existing trained models.
Comparing models obtained using KMEX to existing PSEMs requires comprehensive evaluation strategy which, for this fairly new field, is still lacking. Differing from conventional black-box classifiers, PSEMs yield global (prototypes' visualization) and/or local (activation of individual prototypes by input images) explanation maps alongside the predicted class probabilities. Yet, the assessment of SEMs until now has been limited to comparing the predictive performance to the black-box counterpart with the same backbone architecture as the SEM, followed by quantifying the robustness of local explanation maps and qualitative evaluation of global explanations Wang & Wang (202)); Parekh et al. (2021). We argue that this approach overlooks crucial facets of SEM explainability, failing to establish standardized framework for thorough analysis and comparison of existing models. For example, we observe that most of the prototypes learned by recent SEMs might never be used by their classifier, which challenges the rationalc of a transparent model. Further, the diversity captured by different prototypes in the embedding space, while being a driving force behind the development of several SEMs (Wang et al., 2021), has traditionally only been validated by highly subjective visual inspection of the prototypes.
We, therefore, presenl a novel quantilative and objective evaluation framework based on the three properties ies that arose as predicates for SEMs (Gautam et al., 2022): transparency, diversity, and trustworthiness, The rationale is not to rank models but to highlight the consequences of modeling choices. Indeed, in some applications, having robust local explanations might be more valuable than diverse prototypes. Yet, this behavior needs to be quantified in order to support practitioners in choosing the best model for their use case.
Our main contributions are thus as follows:
We propose simple yet efficient method, KMFx, which converts any existing black-box model into a PSEM, thus enabling wider applicability of SEMs.
We propose novel quantitative evaluation framework for PSEMs, grounded in the validation of SEM's predicates (Gautam et al., 2022), which allows for an objective and comprehensive comparison. Nonetheless, it is important to note that the interpretability of the model still relies on the visual analysis of the prototypes as well as their similarity with test data.
Our key findings are as follows:
Experiments on various datasets confirm that KMFx matches the performance of the black-box model while offering inherent interpretability without altering the embedding, making it an efficient benchmark for SEMs.
Most existing PSEMs tend to ghost the prototypes, i.e., never utilize them for prediction, which gives a false sense of needed concepts but also undermines the rationale formalized by the predicates, especially transparency.
Unlike KMEX, the large variations in the design and regularizations of other SEMs lead to drastically different learned representation spaces and local explanations.
While many SEMs incorporate mcasures to obtain diverse prototypes, these efforts are not necessarily reflected in terms of captured input data attributes. We illustrate how KMEx can be leveraged, without
2

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: Design strategies used by state-of-the-art SEMs.
Similarity Measure Classifier
Protolypes
Diversity Loss
ProtoPNet Distance based
Linear Layer
Projected from training data Min/max htraa/inter-class distance
FLINT
Linear Layer
Linear Layer
Weight of the network
Min/max similarity entropy
Pro Distance based
Linear Layer
Learned ad-hoc parameters
Orthonormality + KL Divergence
KMEs
Distance based
Nearest Neighbor k-means
Clustering
the need for retraining, to improve the prototype positioning on the SEM's embeddings and to better cover the attributes and their correlations.
2 Prototypical self-explainable models
In this section, we review the recent literature on PSEMs for the task of image classification, which is the focus of this work, emphasizing their design considerations as well evaluation approaches.
PSEMs for image classification typically consist of four common components: encoder, set of prototypes, a similarity function, and a transparent classifier. The encoder is typically sourced from a black-box model, thereby making the latter the closest (to the SEM architecture) natural baseline for comparison until now. Prototypes are class-concepts that live in the embedding space and serve as global class explanations, i.e., representative vectors, that climinate the necessity to examine the entire dataset for explaining the learning of the model. The similarity function compares features extracted from the input to those embodied in the prototypes. Ultimately, a transparent classifier transforms the similarity scores into class predictions. The fact that the final classification revolves around the protolypes makes them critical component of SEMs. In addition to these, other modules have also been utilized in the literature to facilitate the learning of prototypes, such as a decoder to align the embedding space to the input space (Parekh et al., 2021; Gautam et al., 2022), or a companion encoder to learn the prototypical space (Parekh et al., 2021).
2.1 Predicates for SEMs
PSEMs are designed to learn inherently interpretable global class concepts. Three principles arise from the literature to form a framework for their construction: transparency, diversity, and trustworthiness (Gautam et al., 2022).
A model is said to be transparent if the downstream task involves solely human-interpretablle concepts and operations.
The learned concepts are diverse if they capture non-overlapping information in the embodding space and, therefore, in the input space.
Trustworthiness comes in several dimensions. An SEM is deemed faithful if its classification accuracy and explanations match its black-box counterpart. In addition, local and global explanations should be robust (similar inputs yield similar explanations) and truly reffect the important features of the input with respect to the downstream task.
2.2 Related work
Prototypical networks, first introduced by Snell et al. (2017), aim to learn an embedding space structured such that points representing data from the same class are closer together, while points from different classes are further apart. In these networks, 'prototypes' acts representative centroids for each class within the embedding space. This approach has proven to be effective in scenarios where classifiers need to generalize to new classes that were not encountered during the training phasc, a challenge typical of few-shot learning (Snell el al., 2017; Trosten et al., 2023). While the structure learned in PSEMs resembles the class-wise structuring of the embedding space used for few-shot learning, semantic meaning from the input-space attributed to these prototypes, enhancing their inherent interpretability (Alvarez Melis Jaakkola, 2018; Chen al., 2019b). KMEx builds upon this approach and transforms any black-box model into self-explainable one without the need for restructuring the already learned embedding space.
3

---

Published in Transactions on Machine Learning Research (05/2024)
Protocype
Input
Protetype
KMEx
like
like
Black box encoder
Prococype
Input
Prototype
Looks
Looks
Black-box linear classifier
5 random test images activating prototypes of class 'dog'. Since we use I-ncarest neighbor classifier, classiBication conveniently depends on the closest prototype.
Isput
Prototype
Looks
like
deer
dog
horse
Figure 1: Schematic representation of KMEx. Left: The black-boc classifier is removed and replaced by & nearest neighbor classifier based on prototypes learned using k-means in the embedding space. The UMAP (MeInnes et al., 2018) representation is the projection the learned embedding space for STL-10, along with prototypes, depicted as squares. Right: The prototypes are visualized in the input space using the closest training images.
The first general framework to compute interpretable concepts was SENN (Alvarez Melis Jaakkola, 2018), which relies a complex architecture and loss function to ensure interpretability. Following this, several SEMs have emerged, one of the most popular being ProtoPNet (Chen et al., 2019b). The latter introduces learnable prototype similarity layer with fixed number of prototypes per class. Several methods have followed to address the limitations of Pro. For example, ProtoPShare (Rymarczyk et. al., 2021), Proto Tree (Nauta et al., 2021) and ProtoPool (Rymarczyk et al., 2022) proposed learning of shareable prototypes across classes, (Donnelly et al., 2021) proposed adaptive prototypes which change their spatial location based on the input image and TesNet (Wang al., 2021) introduced a plug-in embedding space spanned by basis concepts constructed the Grassman manifold, thereby inducing diversity among prototypes.
In parallel to ProtoPNet and its extensions, several other SEMs have been proposed. FLINT (Parekh et al., 2021) introduces an interpreter network with a learnable attribute dictionary in addition to the predictor. SITE (Wang & Wang, 2021) introduces regularizers for obtaining a transfornationn-quivaaaa SEM. ProtoVAE (Gautam et al., 2022) learns a transparent prototypical space thanks to a backbone basod on a variational auloencoder, thereby having the capability to reconstruct. prototypical explanations using the decoder.
While all the existing SEMs have demonstrated effective generation of explanations alongside comparable accuracies, they invariably demand significant architectural modifications and integration of multiple loss functions. This often introduces several additional hyperparameters to achieve satisfactory performance. For instance, in the case of ProtoPNet, a three-step training process involves encoder training, prototype projection for explainability, and last-layer training. Furthermore, as highlighted in FLINT (Parekh et al., 2021), a simultancous introduction of all losses can lead to suboptimal optimization. Their workaround strategy involves distinct loss combinations for fixed epochs. These intricate training strategies, combined with the challenge of training large deep learning architectures, complicate the accessibility of SEMs, thereby emphasizing the demand for more resource-efficient alternatives. KMEX, universally applicable method that necossitates no re-training, no additional loss terms for training the backbonc, and minimal architoctural adjustments for learning the protolypes, presents an elficient solution to this challenge. Considering our general contributions to SEMs, we use ProtoPNet, a representative approach encompassing all its extensions, as bascline this work. Additionally, we also consider FLINT and Pro,oVAE, which cover the diversity of the SEM's literature in terms of backbones, similarity, and loss functions. A summary of these baselines is given in Table 1, along with KMEx, which is presented in the following section.
4

---

Published in Transactions on Machine Learning Research (05/2024)
3 KMEx: universal explainer
In this section, we introduce KMEx which transforms black-box model into an SEM, fulfilling all the aforementioned predicates. Note that to enhance legibility, KMEx may refer to both the method and the transformed model in the following.
3.1 Notations
We will now present some notations that will be utilized in the subsequent sections. We consider an image dataset X'= H= made of N > 0 images split into K > 0 classes, where 2; E IR W xxxC is an image of width W> 0, height H > O, and with C'0 channels, and yı E [1... K] encodes its label. Any model in the following contains an encoder f which projects an input a, into point f(x,,) = 2; E RR of a D > prototypes per
dimension vector space, also referred as the embedding space. We consider sct of class {p (PI1...PII. ... P.I. p.t....KKI.. J C IR that are vectors of the embedding space. Finally, we denole as a generic similarity measure between vectors of the embedding space that returns larger values to pairs of vectors deemed similar. This could be the cosine function the negative of the l2 distance.
3.2 KMEx
Let us consider trained model made of encoder and classifier. It can be converted into self-cxplainable model using the following procedure:
1. Learn L prototypes for each of the K classes using k-means on the embedding of the training data of the class.
2. Replace the classifier with a 1-nearest neighbor classifier comparing the embedding of an imput with the prototypes using the negative of the & distance as the similarity measure.
The resulting model is referred to as the K-Means Explainer (KMEx) of the original model. A schematic representation of the operations is depicted in Figure 1. We chose constant number of prototypes per class in order to case the notations. However, in practice, more complex classes may require more prototypes than others, which is straightforward to adapt in KMEx. Note that the KMEx conversion is not a post-hoc explainability method per se as what it produces is another model that inherently interpretable owing to the central role of the prototypes and the transparent classifier. Although the trained encoder re-used, the KMEx's predictions are computed differently using prototypes that were not part of the initial model.
We further highlight. that k-means is computed per class and the embedding space, which usually has reasonable number of dimensions (512 for ResNet34). Hence, the computational cost is limited and manageable by classic implementations, irrespective of the complexity of the data. For very large datascts, it can be approximated by computing k-means on a subset of the training set or by using other efficient. implementations (Johnson et al., 2019).
3.3 KMEx is an SEM
Visualisation of explanations The explanations for a PSEM are two-fold. Global explanations involve visualizations of prototypes in the input space, providing insights into the model's acquired knowledge. Local explanations, on the other hand, entail pixel-level explanations for input images, revealing which portions of an image are activated by each prototype. For KMEx, we provide global explanations by visualizing the training images that are closest the corresponding prototypes the embedding space. This approximation is justified by the problem solved by k-means, which makes it unlikely for prototype to be out of distribution. adhere to previous works and employ Prototypical Relevance Propagation (PRP),
For local explanations, technique demonstrated to be efficient and accurate for ProtoPNet (Gautam et al., 2023).
Transparency The nearest protolype classifier of KMEx allows backtracking of the influence of a prototype on the predictions, which relates to a distance in the embedding space, thus embodying transparency. Faithfulness of the predictions If the original trained model learned separate well the classes in the embedding, there should be enough inter-class distance for the linear partition of k-means to yield KMEx prototypes that also correctly separate the classes and thus achieve classification performance akin to that of
5

---

Published in Transactions on Machine Learning Research (05/2024)
Table 2: Evaluation strategies for the predicates used by state-of-the-art SEMs. The proposed evaluation
framework is italicised.
Trustworthiness
Transparency
Diversity
Baseline
Faithfulness
Robustness
ProtoPNet FLINT ProtoVAE
Visualization
Black-box
Accuracy
Visualization
Black-box
Accuracy
Visualization
Flack-box/SEM
Accuracy
AT/AD/RO
Reconstruction visualization
Proposed
Ghosting
Black-box/SEM/
Accuracy/
Area under RO curve
Inter-prototype
Evaluation
KMEs
KL Divergence
similarity
the trained model.
Faithfulness and robustness of the explanations The only difference between black-box and its KMEx is how the predictions are derived from the embedding. Therefore, considering identical weights in both models' encoders, most of the operations involved in the generation of local explanation maps are common to both, thus similar explanations with similar level of robustness are expected, regardless of the technique chosen to generate local explanations.
Diversity The purpose of the prototypes is to serve as representatives of their neighborhood the embedding space. The diversity predicate implicitly requires that they also spread over the embedding. To satisfy this predicate without compromising their function, we aim to position the protolypes the accumulation points of the embedding. These are captured as the modes of Gaussian density estimate. Computing such model for high dimensional and sparse dataset is costly, hence we approximate it using k-means. Finally, given that k-means employs a uniform prior on the cluster probabilities, this method has the advantage of covering as much of the data in the embedding space as possible, thus fostering diversity.
4 Evaluations
As stated earlier, existing SEMs build upon three shared predicates but adopt varied strategies ensure their fulfillment. Thansparency is assumed based on architectural choices and, at best, confirmed through visualization of prototypes using different strategies, such upsampling (Chen et al., 2019b), activation maximization (Parekh et al., 2021; Mahendran & Vedaldi, 2016) and PRP (Gautam et al., 2023), accompanied with similarity scores. The trustworthiness predicate is the most quantifiable one. The faithfulness of the performance with respect to the "closest" black-box often reduced to a comparison of accuracies, and the robustness of the explanations evaluated via recent measures such as Average Increase (AT), Average Drop (AD), and the area under the Relevance Ordering curve (AUROC) test (Lee al., 2021; MacDonald et al., 2019; Hedström et al., 2022). Nonetheless, the quantification of disparities between local explanations generated by SEM and its nearest black-box model has been largely disregarded. We emphasize that this aspect grows in significance, particularly as we transition to techniques that transform existing black-box models into interpretable ones without re-training, a domaín where KMEx stands as the first approach. Finally, prototypical diversity has been largely overlooked in prior research, with evaluations, if conducted, being primarily qualitative in nature (Gautam al., 2022).
In this section, we first evaluate KMFx following the evaluation protocols used in the original papers of the selected basclines, which are summarized in Table 2. Following this, we propose our full quantitative evaluation framework based on the predicates for SEMs, highlighting the gaps in the evaluation of SEMs existing until now, also summarized in Table 2. Additionally, we present a quantitative study of the diversity and subclass representation captured by the prototypes learned by existing SEMs and their KMEx counterparts.
4.1 Datasets, implementation and baselines
We evaluate all methods on datasets, MNIST (Lecun et al., 1998), FashionMNIST (Xiao et al., 2017) (IMNIST), SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky, 2009), STL-10 (Coates et al., 2011), ), subset.

---

Published in Transactions on Machine Learning Research (05/2024)
Table 3: Prediction accuracy for SEMs demonstrating the effectiveness of KMEx as an SEM baseline. Reported numbers averages over runs along with standard deviations. Statistically larger values are highlighted in bold.
MNIST
IMNIST
SVHN
CIFAR-10
STL-10
QuickDraw
CelebA
ResNet34
99.4
92.4+0.1
92.6#0.2
85.6
91.8
86.5=01
98.5
FLINT 99.210.1
91.8
91.110.7
82.2
87.5
87.3-1.2
97.2 97.2-10.3
ProtoPNet
99.4
92.4
94.4=0.1
84.9
88.1
87.8
18.0
ProtoVAE
99.4
92.7
93.8
83.0
85.6+1.
85.1 10.8
98.6
R94+KMEr
99.4
92.3
92.440.
85.3
91.9100.
86.6
98.3
of QuickDraw (Parekh et al., 2021) and binary classification for male and female for the CelebA datasot: (Liu et al., 2015). We use a vanilla ResNet34 (He et al., 2016) the encoder for all the models and fix the number of prototypes per class as 20 for CelebA and 5 for all other datasets. Further implementation details are provided in Appendix A.2. For baselines, we train ProtoPNet (Chen et al., 2019b), FLINT (Parekh et al., 2021), and ProtoVAE (Gautam et al., 2022) for learning image-level prototypes. For ProtoPNet, we use average pooling to generate image-level prololypes. For FLINT, we use the interpreter network FLINT-g.
4.2 Traditional evaluation of KMEx
In this section, we evaluate KMEx following previous lines of works (Wang & Wang, 2021; Gautam al., 2022). We start with comparing the prodictive performance of KMEx, which is then followed by an evaluation of explanations consisting of visualization of prototypes and evaluating the robustness of explanations. In Appendix A.7, we present preliminary results for patch-based KMEx.
4.2.1 Predictive performance
We report the accuracy achieved by KMEx, well selected basclines in Table As can be observed, with maximal discrepancy of 0.3 points, KMEx performs par with ils corresponding ResNet34 black-box base model, thereby validating the change of classifier On the other hand, FLINT performs consistently and significantly worse than the black-box on all the datascts. ProtoPNet and ProtoVAE outperform ResNet34 and R33+ KMEx on two the datasets.
4.2.2 Evaluation of explanations
In previous works, the evaluation of explanations was twofold: 1) qualitative evaluation of prototypes and 2) evaluation of robustness of prototypical explanations.
Qualitative evaluation We visualize prototypes learned by KMEx for MNIST and STL-10 datascts in Figure 2 (lop row). Additional visualizations for other datasets are given in Appendix A.8.1 and A.8.2. We demonstrate the "this looks like that" behavior exhibited by KMEx for test images in the bottom row of Figure 2, along with their corresponding PRP maps, demonstrating the regions activated in the test images by their closest prototypes. As observed, for the MNIST dataset, the activations are in response the shape of the digit in the prototype. Similarly, for STL-10, the closest prototype has emphasized key features of bird, such as the head, beak, and eyes, as well portion of the sky in the background.
Robustness evaluation We evaluate the robustness of the local explanations using the area under the relevance order curve (AUROC) Schulz et al. (2020); MacDonald et al. (2019). The relevance order (RO) curve is the evolution of the probability of the predicted class as the least. relevant pixels are progressively replaced by white noise. The relevance derives from PRP local explanations computed with respect to the most similar prototypes. For the black-box model, we rely instead on the Layer-wise Relevance Propagation (Bach al., 2015) (LRP) explanations with respect to the class probability. The local explanations are computed using the Zennit package Anders et al. (2021) with EpsilonPlusFlat composer.
7

---

Published in Transactions on Machine Learning Research (05/2024)
7
7
7
7
7
MNIST
STLIO
Input
Prototype
PRP
Input
Prototype
PRP
7
Looks
7
7
Looks
like
like
Qualitative evaluation of KMEx: Prototypes learned by KMEx for MNIST for class '7' (left) and STL-110 for class 'bird' (right) are shown at the top, demonstrating global explainability. This looks like that behavior for test images are shown at the bottom, along with PRP maps demonstrating the regions activated by closest prototypes (in red) for the test images, exhibiting local explainability.
Figure 2:
Table 4: Area under the Relevance Order curves evaluate the robustness of the PRP local explanations each model produces. Reported numbers are averages over 100 test images along with standard deviations. Statistically larger values are highlighted in bold.
ResNet34
0.397
0..00
0.308400000
0.35440.000
0.466 10.0000000
0..088100000
0.671 10.00
FLINT
0.116#0.061
0.142=0.082
0.009
0.133+00005
0.090=0.055
0.108
0.484+0.073
ProtoPNet
0.57220.0000
0.339
0.388=+0.032
0.313=0.013
0.445=00004
0.667=0.005
0. 1.6998+0.007
PAO 0.200
0.291 0.29110.040 10.040
0.
0.680
0.332
0.113
0.129
0.832-0.006
R34 KKMEr
1.401=0.000
0.379=0.000
0.317#00000
0.347=00000
0.45440.000
0.686=00000
0.616=0.000
Curves averaged over 100 test images and all prototypes for each model on MNIST, CIFAR-10 and CelebA are depicted in Figure 3. For reference, the plots also show the respective average RO curves when the pixels are masked at random. The datasets were chosen for the different behaviors they induce in the lested models. In the three cases, FLINT's (blue) curves are relatively flat. On MNIST, the three other models show the three characteristic possible behaviors. ProtoVAE (green) shows poor robustness as the curve decreases sharply early on and remains below its random curve (dashed green). ResNet34 (gray) and its R34+KMEx (red) decrease somehow linearly before flattening the proportion of masked pixels reaches 0.7. On the other hand, the explanations of ProtoPNet (yellow) demonstrate some robustness its RO curve remains high for longer before starting to decrease. On CIFAR-10 all the models but FLINT present a similar, not-so-robust behavior. Finally, ProtoVAE provides the most robust explanations as its curve remains above all the other curves.
MNIST
These observations are reflected Table 4, where we report averages AUROC with and standard deviations for 100 test images on each dataset. On MNIST, ProtoPNet returns the largest value. On CIFAR-10, all but FLINT report similar values despite being statistically different. On CclebA, ProtoVAE reports the largest AUROC, followed by ProtoPNet. Overall, R34+KMEx returns AUROC close to ResNet34, its black-box base model. Yet the differences are always significant given the very low standard deviations. These results suggest that KMEx does not produce more robust explanations on its own. This is anticipated as KMEx aims to facilitate the interpretation a learned latent representation but does not enforce robustness or stability of the prototypes during the training process, unlike other SEMs.
4.3 Quantitative evaluation of SEMs
Having demonstrated the traditional evaluation of the proposed KMEX, we now address the lack of a comprehensive evaluation framework for SEMs that quantitatively evaluates the predicates. First, we expose for the first time how transparency is often undermined by unused prototypes (ghosting) and measure the phenomenon. Next, we objectively quantify the faithfulness of local explanations and the diversity of the prototypes without resorting to visual inspection. Again, the rationale is to propose a framework assess objectively each model's strengs and weaknesses.
8

---

Published in Transactions on Machine Learning Research (05/2024)
MN
CIFAR10
CELEBA
Prediced class probabiltty
0.9
0.8
0.6
0.4
0.4
0.6
0.
0.7
0.0
0. Proportion of least
00
0.2
3.0
0
0.8 0.8
10
Proportion of the least important pixels masked
Propertion the least importent pixels masked
important pixels masked
ResNet34 RRnndm
FUNT /Random
ProtoPNet /Random
ProtovAE /Random
R34+KMEX /Random
Figure 3: Relevance Ordering curves computed on different datasets and with different architectures, along with the respective random basclines (dashed).
4.3.1 Transparency and concept ghosting
The transparency predicate allows the user to backtrack the influence of the learned concepts on the predictions and is usually enforced through architecture design. However, we observe for state-of- sfate-of-the-art SEMs that, in practice, some learned prototypes are not reachable from the predictions. More specifically, they never activated by any training data point of their class, i.c., they are never the most similar prototype of any training data. This so-called ghosling of the protolypes not only gives a false sense of needed concepts but also undermines the notion of transparency itself, as the link between prototypes and prediction can not be fully trusted.
In the case of a distance-basel similarity measure (Gautam et al., 2022; Chen et al., 2019b), if k argmax, 5(2,,t)) then prototype p& is the closest 24. If several points activate pk, this indicates that the data embedding aggregates around concentration point close to Pk- In contrast, if pa never maximizes any s(2,,),, ), then there might be data in its neighborhood. The prototype is either out-of-distribution area of low density. In the case of a dot-product-basel similarity measure (Parekh et al., 2021), if pie is activated by 2;, then and are aligned. Assuming that is not the only maximizer of s(s,k)), then ps carries a direction along which the data accumulates. In contrast, if pk never maximizes any s(2,, :), then the data does spread along direction and may represent either a variation within a class or, the worst case, noisc.
lies in
We propose quantify this ghosting phenomenon based the average activation frequencies over the prototypes the training set:
seep = KL C>E [i E [1...N] argmax(5(22)) (s(2).P.))) (ks, ISuSK
(1)
15e5L
where [[-]] is the Iverson bracket which returns 1 if the contained statement is true and 0 otherwise. The values of Desp range between 0 and 1, with lower values indicating loss ghosting.
In Table 5, we report average Dtsp scores (Fquation 1) with standard deviation over five runs. We observe that ghosting affects all models but, unsurprisingly, never KMEx. Indeed, for such a low number of prototypes, relative the size of the data, k-means unlikely create an empty cluster. Interestingly, ProtoVAE also does not ghost any protolype on three out of the seven datasets, suggesting that SEMs with geometrical constraints more robust. to ghosting.
4.3.2 Trustworthiness and faithfulness
According to its definition, the trustworthiness predicate encompasses two major axes. The first is the faith- fulness of the predictions, which we have quantified in terms of accuracies in Table 3. The second aspect
9

---

Published in Transactions on Machine Learning Research (05/2024)
Table 5: Tronsparency: frequency of ghosted prototypes by SEMs.Statistically smaller values are highlighted
in bold.
MNIST
FMNIST
SVHN
CIFAR-10
STL-10
QuickDraw
CclebA
FLINT
0.246-0.148
0.251 258
0.060-4..25
0.17442.066
0.156:10.070
0.230 10.360
0. 0.220010.365
ProtoPNet
0.610-0.022
0,580-0.061
0,490-40.055
0.165-10.051
0.221 0020110.081
0.156 6..06
0.695
ProtoVAE
0.000
0.012=0.016
0.012#0.010
0.000#0.000
0.642#20043
0.561=0.07
0.335
KKM+
0.000
0.000=00000
0.000=0.000
0.000#0.000
0.000#00000 000
0.000 000
0.000#00.000
Table 6:
Foithfulness of explanations: divergence of LRP explanation maps from the black-box. Reported
numbers are average over 100 test images. Statistically smaller values are highlighted in bold.
MNIST
IMNIST
SVHN
CIFAR-10
STL-10
QuickDraw
CelebA
FLINT
0.696=0.011
0.883+00.088
0.790+00.03
0.739#0.054
2.468#00.11
0.783=0.050
4.411=0.199
ProtoPNet
0.749 0.749900.045
1.071 1.071 +0.084
0.906-4.0034
0.896-10.046
2.59g400.109
0.898
0.899-00004
4.785-40.106
ProtoVAE
0.656=0.005
0.667#0.019
00.01
0.605#0.019
2.441#0.11
0.716
4.268++0.30
KMMF
0.130+0.000 000
0.105=0.000
00000
0.117#0.000
0.461 +0.000 0.0
0.088 000
0.71 1711700.000
concerns the robustness of the ezplanations, which we measured using RO curves (MacDonald et al., 2019; Schulz et al., 2020) in Table 4. An often overlooked aspect of the trustworthiness predicate is the faithfulness of the erplanations. Indeed, SEMs differ from black-box models in their architecture and training and, therefore, also in their local explanations. However, as move towards met that convert black-box it becomes crucial to quantitatively evaluate this discrepancy. We propose to use the Kullback-Leible (KL) divergence (DRI.) between the Layer-wise Relevance Propagation (LRP) maps (Bach et al., 2015) for the prediction probabilities produced by the SEM and the black-box baseline. Since the divergence acts on distributions, the relevance maps need to be normalized. The use of LRP aligns with the previous utilization of PRP. Other methods could be used, yet our intention here is not to evaluate the SEMs with respect to these methods but rather with respect to the predicates.
models into self-explainatie
Let us denote the output of the local explanation method for an input as e(x)) WWHHC The corresponding normalized relevance e, (x)) IR WXH is defined as:
max..1..c (e(z)](,,,c)
en (x) Eu=I.. En=I... maxc=1...c [e(x)](w,h,d h)
The divergence of en (x) with respect to the normalized local explanation maps produced by the black-box backbone (x)) is measured by Diai defined as follows:
H
(n,)) C (x)(x,)
Dia Σ
en (w; ho log
(3)
The KL divergence is zero if, and only if, the distributions are equal. Consequently, Dian can be null if, and only if, the SEM and the black-bos models always produce the same explanations.
In Table 6, we report the average Dia based on LRP and standard deviation for cach SEM on 100 images of each dataset. Normalized LRP maps for each model and on different datasets are represented Appendix A.4. Again, KMEx produces the most faithful feature importance maps, which is expected since most of the operations happen in the encoder, which originates from the black-box model.
4.3.3 Interpreted diversity
The abundance of existing strategies to guarantee the diversity of an SEM reflects the subjectivity of the notion. Thus, it not obvious how to evaluate this predicate in the input space, espocially given that very few public image datasets provide attributes describing the image. Thus, the evaluation has to be done in the embedding space. However, using metric based on distances in the embedding space would disad
10

---

Published in Transactions on Machine Learning Research (05/2024)
Table 7: Interpreted diversity; quantitative evaluation of diversity for different SEMs. Statistically larger values highlighted in bold.
MNIST
IMNIST
SVHN
CIFAR-10
STT-10
QuickDraw
CelebA
FLINT
0.991=0.000
0.991
0.967#0.017
0.963#0.018
0.999
0.968+0.010
0.948+0.013
ProtoPNet
0.172=0.017
0.160
0.173=t0.018
0.233#0.014
0.120##0.05
0.241=0.022
0.715#0.021
Pro
0.014=0.000
0.014#0.000
0.014+0.000
0.014#0.000
0.013##0000
0.014#0.000
0.014+0.000
R94+KMEr
0.000
0.000=0.0000
0.000#0.000
0.000
0.000
0.000 10.000
0.026
methods relying on a dot-produut-baseed similarity measure and the other way around. We therefore propose to evaluate SEMs the basis of their own interpretation of diversity and to base our diversity metric on the models' own similarity function. In other words, the idea is to assess the extent to which models achieve diversity on the basis of their own model choices.
The overarching objective of cxisting approaches for diversity is to prevent prototype collapse. In such a case, the information captured by the prototypes highly overlaps, yielding inter-prototyp similarities (s(pkt Puv) with (k: 1) # u)) as high as prototypical self-similarities (a(pu, On the other hand, if prototype collapse well alleviated, the inter-prototype similarities are low, while the self-similarities remain high. This observation motivates the use of the entropy function.
Accordingly, we quantify the diversity of a set of concepts using Davs defined as the average of the normalized entropy of the similarities between each prototype. The computation compares each prototype to all the others without discarding the ghosted prototypes, as they may indicate a collapse.
K
L
Davs KTlog( K L)
1
ΣΣ
H (Softmmxx (slbms (wlp) I
(4)
where H is the entropy function. The log(K normaaiiiiiiinnn restricts the measure to [0, 1] and allows comparisons between different runs, number of prototypes, as well as models. Large values indicate more similarity between the clusters and, thereby, less diversity.
In Table 7, we report the average Davs (Equation 4) score with standard deviations over five runs for each SEM on each dataset. Recall that Dave estimates how well a model satisfies its own interpretation of diversity. Following this, KMEX and FLINT are, respectively, the most and the least satisfying models. We emphasize here that a low diversity docsn't reflect the caliber of the learned embedding space and only suggests an important overlap of information between the representative prototypes learned for the embeddings.
4.3.4 Summary
We summarize the quantitative assessment presented a radar plot in Figure 4. The axes correspond, respectively, to the average results over the datasets presented in Tables 3, 4, 5, 6 and 7. The axes are set such that a larger polygonal area is better. The transparency (Transp.) axis ranges from in the center to 0 on the outer circle. The faithfulness of accuracy (Faith.Acc.) corresponds to the difference from that of the black-box base model. Along that axis, points on the outer circle mean equal accuracy with the black-box base model. models below perform worse, with the center indicating difference of 5 points. Models performing better have their points beyond the outer circle. The faithfulness of the explanation is restricted to Día ranging from 0 on the outer circle to 2 on the center. A better robustness of explanation ERob.Expl.) means a value closer to 1. The worst possible value is 0. Finally, the diversity score (Div.) ranges from on the outer circle to 0 on the center. Plots for individual datasets are presented in Figure 7 in the appendix.
Transp.
Better
Div
Falth.
Acc.
Rob Expl
Farth. Expl.
(aith.Expl.) axis
FUNT
ProtoVAE
ProtoPNet
K34-
Figure 4: Summary of each model's strengths and weaknesses.
11

---

Published in Transactions on Machine Learning Research (05/2024)
Table 8: Quantitative evaluation of diversity by applying KMEx learned SEM embeddings. Statistically
significance difference with respect to Table 7 are marked in bold
MNIST
IMNIST
SVHN
CIFAR-10
STT.-10
QuickDraw
CelebA
MUIN+KKME,
0.455
0.456=0.232
0.513+0.135
0.358+0.007
0.530
0.406#0.226
0.520+0.223
Difference
-0.536
-0.535
00.44
-0.605
-0.469
-0.662
-0.427
KMEX
10.008
0.237-0.015
0.276-10.009
0.257
0.028 10.003
0.293-10.000
0.717-10.019
Difference
0.163
0.077
0.104
0.024
00.92
0.052
0.003
ProloVEE+KMI
0.014#0.000
0.014=0.000
0.014
0.01440.000
0.019#00001
0.016=00000
0.015+#0.000
Difference
0.000
0.000
0.000
0.000
0.005
0.002
0.001
This visualization makes it easy to identify the strengths and weaknesses of each SEM and thus determine the most suitable model according to the problem statement at hand. KMEx suffers the least of ghosting (good transparency) and is the most faithful model with respect to the original black-box both in terms of accuracy and explanation. ProtoPNet performs well in terms of predictions and robustness of the local explanations, but it underperforms in terms of diversity and transparency. This is due to the lack of an inter-class diversity constraint in the ProtoPNet's 's design. On the other hand, ProtoVAE leads in terms of diversity, but its explanations often differ from the black-box base model's. This likely due to the utilization of a VAE backbone, which deviates a lot from the architecture of the black-box baseline. Finally, FLINT produces the smallest area. If, on average, it performs on par with ProtoVAE in terms of transparency and faithfulness of the accuracies, it lags behind in all the other measures, especially in terms of diversity.
4.4 Diversity and embedding
In this section, we show that for the same embedding learned by an SEM, the KMEx paradigm for prototypes may also be used to improve both the measured and qualitative diversity without retraining the embedding.
KMEx improves measured diversity We evaluate first how changing the paradigm of an SEM to KMEx may improve the quantified diversity (Davs). We report in Table 8 average scores and standard deviations over five runs for the KMEx of each SEM baseline and the average difference with Table 7. The change of paradigm equips the embedding of FLINT with more quantitatively diverse prototypes. For ProtoPNet, the change is worst on all but one dataset. As for ProtoVAE, which learned prototypes with a very low Dava. the changes are very small. Recall that Dave can only serve as an internal evaluation. Therefore, any further analysis of the prototypes requires an external criterion. A visual comparison of the prototypes presented in Appendix A.8.3.
KMEx improves minority subclass representation We further study here the diversity of the prototypes in light of the reprosentation of the attributes they capture. We interpret the notion of fair subclass representation for SEMs as whether prototypes are able to capture the information about the underrepresented subclasses. For this experiment, we trained ResNNet34KKMFx, ProtoPNet, PAO, FLINT, and their KMEx on the CelebA dataset for male and female classification with varying numbers of prototypes. Prototypes are represented in the input space by their nearest training images, which come with 40 binary attributes as annotations.
To put the observations in the other figures into perspective., we plot first in Figure 5.a the number of prototypes ghosted against the number of prototypes trained. We see here clearly the depth of the issue for FLINT and ProtoPNet. The two following plots (Figure 5.b) depict the number of captured attributes given number of trained prototypes, including ghosted ones. The left plot shows the results for the basclines and the right one for their KMEx. FLINT starts and ends with fewer captured attributes, and il seems unstable with a large number of prototypes. As for ProtoPNet, it caps at 32 attributes when trained with 12 or more prototypes. On the other hand, the KMEx of any method (right plot), including RosNet34+KMEX (red), always captures more attributes as the number of prototypes increases. The last experiment aims to evaluate how many combinations of attributes are captured using the mean absolute error (MAR) between the attribute
12

---

Published in Transactions on Machine Learning Research (05/2024)
a. Evolution of the number of prototypes ghosted.
b. Evolution of the number of attributes captured the prototypes.
MAE between the attribute correlation matrices c.
of the dataset and of the prototypes.
Prototypes ghosted
46
Altributes captured
0.3
-6
30
0.2
20
20
MAE
20-
10
0.1
02
12
20
40
12
20
40
6
12
20
40
0.0
12
20
40 2
12
20
40
Prototypes trained
Prototypes trained
Prototypes trained
Prototypes trained
Prototypes trained
FLINT
ProtoPNet
ProtoVAE
ResNet34
KMEK
+KMEX
Figure Analysis of the attributes captured by SEMs for different numbers of prototypes for CelebA
correlation matrices computed from the training set and the prototypes. (Figure 5.c). The correlations based on ProtoPNet's prototypes are the most divergent, whereas Proto VAE and ResNet34+KMBEx consistently come closer to the ground truth as the mumber of prototypes increases. Again, the attribute correlation computed for any KMEx consistently improves as more protolypes are available.
Overall, KMEx of FLINT improves the most its original model in both criteria: the number of captured attributes and the faithfulness of the attributes correlations. This observation reinforces the intuition that FLINT learns embedding with much more potential in terms of global explanations than it is able to leverage through the prototypes it learns and its similarity mcasure.
5 Conclusion
In this paper, we introduce KMEx, the first approach for making any black-box model self-explainable. KMEx is a universally applicable, simple, and resource-efficient method that, unlike existing methodologies, does not require re-training of the black-box model. Furthermore, we reconsider the subjective evaluation practices for SEMs by introducing a quantitative evaluation framework that facilitates objective comparisons among SEM approaches. The proposed framework adopts set of novel metrics to quantify how well SEMs adhere to the established predicates. An extensive evaluation with the help of the proposed framework highlights the strengths and weaknesses of existing SEM approaches when compared to the models obtained from KMEx. This work, therefore, additionally serves as foundational step towards an objective, comprehensive, and resource-efficiet advancement of the SEM field.
One notable limitation of the proposed KMEx is its reliance selecting a priori the number of prototypes, characteristic it shares with current state-of-the-art SEMs (Parekh et al., 2021; Gautam et al., 2022; Chen et al., 2019b). Additionally, note that the proposed detailed quantitative evaluation framework is meant to provide an additional perspective and not replace qualitative evaluations of SEMs, which are still required due to the subjective nature of explanations.
6 Acknowledgments
This work was financially supported by the Research Council of Norway (RCN), through its Centre for Rescarch-based Innovation funding scheme (Visual Intelligence, grant no. 309439), and Consortium Partners. The work was further partially funded by RCN FRIPRO grant no. 315029 and by the German Ministry for Education and Research (BMBF) through the project Explaining 4.0 (ref. 011$200551).
We would like to thank Ulf Brefeld for granting us access to the Leuphana University cluster. Further, we would also like formally acknowledge the reviewers for their invaluable input. Their advice, time, and efforts have significantly aided in enhancing the overall quality of this work.
13

---

Published in Transactions on Machine Learning Research (05/2024)
References
David Alvarez Mclis and Tommi Jaakkola. Towards robust interpretability with self-cxplanning neural networks. Advances in neural information processing systems, 31, 2018.
SeungHwan An, Hosik Choi, and Jong-June Jeon. Exon: Explainable encoder network. arXiv, 2021.
Christopher J. Anders, Talmaj Marine, David Neumann, Wojciech Samek, Klaus-Robes Müller, and Sebastian Lapuschkin. Analyzing imagenct with spectral relevance analysis: Towards imagenet un-hans' ed. ArXiv, abs/1912.11425, 2019.
Christopher J. Anders, David Neumann, Wojeiech Samek, Klaus-Robert Müller, and Sebastian Lapuschkin. Software for dataset-wide xai: From local explanations global insights with Zennit, CoRelAy, and ViRclAy. CoRR, abs/2106.1Bs
2021.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1-40 46, 07 2015.
Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and Salvatore Rinzivillo. Benchmarking and Survey of Explanation Methods for Black Box Models. Data Min Knowl Disc, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Alexander Camuto, Matthew Willetts, Stephen Roberts, Chris Holmes, and Tom Rainforth. Towards a theoretical understanding of the robustness of variational autoencoders. In International Conference on Artificial Intelligence and Stalislics, pp. 3565 3573. PMLR, 2021.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In SafeAI@ AAAAA, 2019a.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in newral information processing systems, 32, 2019b.
Rewon Child. Very deep vacs generalize autoregressive models and can outporform them on images. arXiv preprint Xiv:2200 10650, 2020.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 215 223, Fort Lauderdale, FL, USA, 11-13 13 Apr 2011. PMLR.
Alex J DeGrave, Joseph D Janizek, and Su-In Lee. AT for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610-6i) 619, 2021. ISSN 2522-5839.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 248-255, 2009.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Jonathan Donnelly, Alina Jade Barnett, and Chaofan Chen. Deformable protopnet: An interpretable image classifier using deformable prototypes. 2022 IEEE/CVE Conference on Computer Vision and Paltern Recognition (CVPR), pp. 10255 10265, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale. ICLR, 2021.
Xavier Ferrer, Tom Nuenen, Jose M. Such, Mark Coté, and Natalia Criado. Bias and discrimination in ai: A cross-disciplinary perspective. IEEE Technology and Society Magazine, 40(2):72-80, 2021.
Yansong Gao, Bao Gia Doan, Zhi Zhang, Siqi Ma, Jiliang Zhang, Anmin Fu, Surya Nepal, and Hyoung- shick Kim. Backdoor attacks and countermeasures on deep learning: comprehensive review. ArXiv, abs/2007.10760, 2020.
Srishti Gautam, Ahcene Boubekki, Stine Hansen, Suaiba Salahuddin, Robert Jenssen, Marina Höhne, and Michael Kampffmoyer. Protovac: A trustworthy self-cxplainable prototypical variational model. Advances in Newral Information Processing Systems, 55:17940- 17952, 2022a.
Srishti Gautam, Marina M.-C. Höhne, Stine Hansen, Robert Jenssen, and Michael Kampfmmeyr. Demon- strating the risk of imbalanced datasets in chest x-ray image-based diagnostics by prototypical relevance propagation. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 5, 2022b.
Srishti Gautam, Marina M-C Höhne, Stine Hansen, Robert Jenssen, and Michael Kampfineyer. This looks more like that: Enhancing self-explaining models by prototypical relevance propagation. Pattern Recognition, 13912, 2023.
David Ha and Douglas Eck. A neural representation of sketch drawings. In International Conference on Learning Representations, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 630, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46493-0.
Anna Hedström, Leander Weber, Dilyara Bareeva, Franz Motzkus, Wojeiech Samek, Sebastian Lapuschkin, and Marina M.-C. Höhnc. Quantus: An explainable ai toolkit for responsible evaluation of neural network explanations. 2022.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive supervuper learning. arXiv preprint arXiv:2110.0990 09348, 2021.
Younghyun Jo, Sejong Yang, and Scon Joo Kim. Srflow-da: Super-resolution using normalizing flow with deep convolutional block. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 364-372, 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégon. Billion-scale similarity search with GPUs. IEEF Transactions on Big Data 7(3):535 547, 2019.
Eunji Kim, Siwon Kim, Minji Seo, and Sungroh Yoon. Xprotonet: Diagnosis in chest radiography with global and local explanations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp.
June 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LoCun (ods.), 3rd International Conference on Learning Representations, ICLR 2015, Son Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
M. Kohlbrenner, A. Bauer, Nakajima, A. Binder, W. Samek, and Lapuschkin. Towards best practice in explaining neural network decisions with Irp. In IJNN, pp- 1-7, 2020.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Zhifeng Kong and Kamalika Chaudhuri. Understanding cestanco-based interpretability of variational auto- encoders. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.); Advances in Neural Information Processing Systems, volume 34, pp. 2400 2412. Curran Associates, Inc., 2021.
Alex Krizhevsky, Learning multiple layers of features from tiny images. 2009.
Y. Lecun, T.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998a.
Y. Lecun, L. Bottou, Y. Bengio, and P. Hafiner Gradient-basse learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998b.
Jeong Ryong Lee, Sewon Kim, Inyong Park, Taejoon Eo, and Dosik Hwang. Relevance-cam: Your model already knows where to look. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14944-14953, June 2021.
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through proto- types: A ncural network that explains its predictions. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirlieth Innovalive Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI 18/AA1// 1 AAAI Press, 2018. ISBN 978-1-57735-800-8.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Scott M. Lundberg and Su-In Lee. A unified approach interpreting model predictions. In Proceedings of the 31st International Conference on Newral Information Processing Systems, NIPS'17, pp. 4768-4777, Rod Hook, NY, USA, 2017. Curran Associales Inc. ISBN 9781510860964.
Jan MacDonald, Stephan Wäldchen, Sascha Hauch, and Gitta Kutyniok. A rato-distortion framework for explaining neural network decisions. ArXiv, 0bss/1955.1109 2019.
Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120:233 255, 2016.
Leland Mclnnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018.
Gaurav Menghani. Efficient deep learning: A survey on making deep learning models smaller, faster, and better. ACM Compul. Surv., 55(12), mar 2023. ISSN 0360-0300.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. Advances in neural information processing systems, 27, 2014.
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klbes- Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65: 211 11-2222,22 2017.
Meike Nauta, Ron van Brec, and Christin Seifert. Neural prototype trees for interpretable fine-grained image recognition. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pollern Recognition (CVPR), pp. 14933-14944 June 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupercised Feature Learning 2011, 2011.
Sajad Norouzi, David J Fleet, and Mohammad Norouzi. Exemplar vae: Linking generative models, nearest neighbor retrieval, and data augmentation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 8753-8764 Curran Associates, Inc., 2020.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Dario Augusto Borges Oliveira and Laura Elena Cue La Rosa. Prototypical variational autoencoders, 2022.
Daniel Omeiza, Helena Webb, Marina Jirotka, and Lars Kunze. Explanations in autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, pp. 1-21, 2021.
Jayneel Parekh, Pavlo Mozharovskyi, and Florence d' d'Alché Bue. A framework to learn with interpretation. Advances in Newral Information Processing Systems, 11-22233-22285 2021.
Adam Pasuke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Toror::: An Imperalive Style, High-Perforraane Deep Learning Library. Curran Associates Inc., Red Hook, NY, USA, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530 1538. PMLR, 2015.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. KDD '16, pp. 1135-1144, 2016.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215 215, 2019.
Dawid Rymarezyk, Fukasæ Struski, Jacek Tabor, and Bartosz Zielinski. Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '21, pp. 1420-1430, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383325.
Dawid Rymarczyk, Lukasz Struski, Michat Górszezak, Koryna Lewandowska, Jacek Tabor, and Bartosz Zieliáski. Interprotable image classification with differentiable prototypes assignment. In European Conference on Compuler Vision, pp. 351 368. Springer, 2022.
Sascha Saralajew, Lars Hokdijk, Maike Rees, Ebubekir Asan, and Thomas Villmann. Classification-byy components: Probabilistic modeling of reasoning over a set of components. In NeurIPS, 2019.
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. Xixi preprint orXiv:2011.00396 2020.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 518-626, 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://pet //prc/eeding.... wwtrriis.cc/pppape files/ halles/paper/2017// /ataa///c6/aaa777711111100000000 Paper .pff.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'117 pp. 3319 3328. JML..org, 2017.
Erico Tjoa and Cuntai Guan. A survey on explainable artificial intelligence (xai): Toward medical xai. IEEE Transactions on Neural Networks and Learning Systems, (111):4733-881) 2021.
Daniel J Trosten, Rwiddhi Chakraborty, Sigurd Lokse, Kristoffer Knutsen Wickstrom, Robert Jenssen, and Michael C Kampffineyer Hubs and hyperspheres: Reducing hubness and improving transductive few-shot. 2023.
learning with hyperspherical embeddings. pp.
17

---

Published in Transactions on Machine Learning Research (05/2024)
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing. Interpretable image recognition by constructing transparent embedding space. In 2021 IEEE/CVF International Conference on Compuler Vision (ICCV), pp. 875-884, 884, 2021.
Yipei Wang and Xiaoqian Wang. Self-interpretable model with transformation equivariant interpretation. Advances in Newral Information Processing Systems, 14:2359-2372 2021.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. Technical Report CNS-TR-201, Caltech, 2010.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.007477, 2017.
Jason Yosinski, Jeff Clune, Anh M Nguyen, Thomas J. Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. 1CML Deep Learning Workshop, 2015.
A Appendix
In this section, we provide additional details for the proposed evaluation framework. First, we provide additional datasets and implementation details, followed by reporting runtimes for different SEMs used as baselines in this work. We then provide qualitative results for the foithfulness of explanations for completeness. This is followed by detailed summary plot for quantitative evaluation for each dataset. We then provide ablation study with respect to the similarity measure, number of prototypes per class, and clustering algorithms. Finally, provide preliminary results on a more complex dataset CUB200 (Welinder et al., 2010), followed by additional qualitative results for KMEx.
A.1 Dataset details
All datasets used in this work are open-source. For all datasets, we use the official training and testing splits, except for QuickDraw (Ha & Eck, 2018) for which we use subset of 10 classes that was created by (Parckh et al., 2021). This subset consists of the following 10 classes: Ant, Apple, Banana, Carrot, Cat, Cow, Dog, Frog, Grapes, Lion. Each of the classes contains 1000 images of size 28x 28 out of which 80% are used for training and the remaining 20% for testing. The MNIST (Lecun et al., 1998), IMNIST (Xiao et al., 2017), CIFAR-10 (Krizhevsky, 2009) datasets consist of 60,000 training images and 10,000 test images of size 28 28x28, 28x28 and 32x32, respectively. The MNIST, IMNIST and QuickDraw images are resized to 32x32 to obtain a consistent latent feature size. SVHN (Netzer et al., 2011) consists of 73,257 training images and 26,032 images for testing of size 32x STL-10 (Coates al., 2011) consists of 5000 images for training and 8000 for testing of size 96x96. All datasets have 10 classes, except for CelebA (Liu et al., 2015) for which we perform binary classification of male vs female. The number of training and testing images for ColebA are 162,770 and 19,962, respectively, of size 224x224. The licenses for the datasets are provided in Table 9.
For preprocessing, every dataset's respective mean and standard deviation for training data is used for normalization. For MNIST, IMNIST, SVHN and QuickDraw, no augmentations were performed. For STL-10, CIFAR-10, and CelebA, we applied a horizontal flip with probability of 0.5 followed by random cropping after zero-padding with size 2 was applied for augmentation.
18

---

Published in Transactions on Machine Learning Research (05/2024)
Table Licenses for datasets used in this work. N-C is used denote that the data is free for non-commercial
MNIST
FMNIST SVHN CIFAR-10
License CC BY-SA 3.0
MIT CCO 1.0
MIT
STL-10
QuickDraw
CelebA
License
N-C
CC BY 4.0
N-C
A.2 Implementation details
The experiments in this work were conducted an NVIDIA A100 GPU. The backbone network used for all models as well as all datasets consists of an ImageNet (Deng et al., 2009) pretrained ResNet34 (He et al., 2016). Proto VAE relies on encoder models from the original work (Gaulam el al., 2022) on all datasets but STT-10, QuickDraw, and CelebA, where it is a ResNet34. This choice was made due to the inability of ProtoVAE to converge on certain datasets with larger encoder like ResNet-34 and a CNN decoder. The size of the latent vector is 512 and the batch size 128 for all datasets as well as models. Stochastic gradient descent (SGD) is used as the optimizer for training ResNet34 with momentum of 0.9 for CelebA and 0.5 for all other datascts. For ProtoVAE and FLINT, an Adam (Kingma & Ba, 2015) optimizer is used. Other hyperparameters including learning rate, nunber of epochs, and number of prototypes are mentioned in Table 10. Note that, unlike other SEMs, KMEx requires tuning of only one additional hyperparameter i.e., the number of prototypes per class, compared to the closest black-box model.
A.3 Runtimes
Comparing the runtimes for different SEMs is not straightforward owing to the various very specialized and multi-step training schemes. For example, ProtoPNet requires training the whole architecture, followed by the projection of prototypes to the imput data, and finally training the last layer for few more epochs. Therefore, for an exhaustive and complete comparison, we report the full training time for all models in Table 11, where each model is trained to achieve the accuracies reported in Table 3 in the main text, using different hyperparameters, as reported in Table 10. Additionally, we report the test times for a single image for all models in Table 12. These experiments are conducted using PyTorch with a more accessible NVIDIA GeForce GTX 1080 Ti GPU. As can be seen, RMEx requires very low training time as compared to all other SEMs. Further, KMEx is also faster for the prediction of single test image as compared to other models. These experiments further highlight the effectiveness of the proposed method.
A.4 Faithfulness of explanations
In this section, we qualitatively evaluate the faithfulness of explanations generated by an SEM to the closest. black-box. For this, mentioned in the main text, we compute Layer-wise Relevance Propagation (I.R)) maps (Bach et al., 2015) for prediction probabilities using Zennit (Anders et al., 2021). In Figure 6, we visualize the LRP maps for random images from the CIFAR-10, CelebA, and MNIST datasets. The LRP maps for the black-box ResNet34 and SEMs ProtoPNet, PAF, and KMEx are shown. For MNIST, we also show LRP maps for a CNN backbone. The CNN architecture used is from (Gautam et al., 2022). As observed, instead of producing non-robust explanations, KMEx remains most faithful to the black-box. This makes KMEx the SEM closest to the corresponding black-box. box, thereby proving to be an efficient baseline.
A.5 Detailed Summary plot
We provide the results for the proposed quantitative evaluation framework for each datasel in Figure 7. As observed, KMEx performs the best for transparency, diversity and faithfulness of explanations for all datasets. ProtoVAE performs on par in terms of diversity for all datasots, however it suffers all other dimensions. ProtoPNet capable of generating highly robust explanations for nearly all datasets, but it falls short in other areas. FLINT, on the other hand, performs the worst for all evaluations. This detailed summary plot.
19

---

Published in Transactions on Machine Learning Research (05/2024)
Table 10: Hyperparameter values for KMEx, ProtoPNet, FLINT and ProtoVAE for all the datasets.
MNIST IMNIST SVHN CIFAR-10 STL-10 QuickDraw CelebA
No. of prototypes
5
5
5
5
5
5
20
per class
KMEz
No. of epochs
10
10
10
30
30
30
10
Learning rate
0.001
0.001
0.001
0.001
0.001
0.001
0.001
No. of prototypes
5
5
20
per class
No. of epochs
waarm
5
5
5
5
etrain
15
15
15
35
35
35
15
push interval
10
10
10
10
10
10
10
ProtoPNef
last layer
5
5
5
5
5
5
5
Learning rates
joint, warm,
last layer
0.001
0.001
0.001
0.001
0.001
0.001
0.001
prototypes
Loss weights
Cross entropy
1
1
Clustering
0.8
0.8
0.8
0.8
0.8
0.8
0.8
Separation
-0.08
-0.08
-0.08
-0.08
-0.08
-0.08
-0.08
0.004
0.004
0.004
0.004
0.004
0.004
0.004
No. of prototypes
5
5
5
5
5
20
per class
No. of epochs
10
10
30
30
30
10
Loss weights
Cross entropy
0.8
0.8
0.8
0.8
0.8
0.8
0.8
FLINT
Input fidelity
0.8
0.8
0.8
0.8
0.8
0.8
0.8
Output fidelity
1.0
1.0
1.0
1.0
1.0
1.0
1.0
Conciseness
0.1
0.1
0.1
0.1
0.1
0.1
0.1
Entropy
0.2
0.2
0.2
0.2
0.2
0.2
0.2
Diversity
0.2
0.2
0.2
0.2
0.2
0.2
0.2
No. of prototypes
5
5
5
5
5
20
per class
No. of epochs
20
60
60
60
20
Loss weights
Proto VA
e Cross Entropy
1
1
Reconstruction
0.1
0.1
0.1
0.1
0.1
0.1
0.1
KL Divergence
1
100
100
100
100
100
100
Orthogonality
0.1
0.1
0.1
0.1
0.1
0.1
0.1
highlights the effectiveness of the proposed qualitative evaluation framework, which can be employed to weigh the advantages and disadvantages of various prototypical SEMs for different datasets.
A.6 Ablation Study
In this section, we perform ablation studies on the similarity measure used in this work, the number of prototypes for each class, and finally the clustering algorithms.
20

---

Published in Transactions on Machine Learning Research (05/2024)
Table 11:
Time (s) for full training
MNIST
INNIST
SVHN
CIFAR-10
STT-10
QuickDraw
CelebA
R34+KMEX
346
330
348
626
182
662
3975
ProtoPNet
3597
3492
3006
6441
733
10806
20885
FLINT
429
437
569
1089
237
1123
4489
ProtoVAB
5030
2725
6055
6313
804
5694
44857
Table 12: Single sample test time (s)
MNIST
EMNIST
SVHN
CIFAR-10
STL-10
QuickDraw
CelebA
R34+KMEX
0.007
0.006
0.007
0.007
0.007
0.007
0.190
ProtoPNet
0.011
0.204
0.018
0.072
0.011
0.011
0.210
FLINT
0.037
0.029
0.037
0.040
0.025
0.026
0.224
ProtoVAE
0.008
0.012
0.008
0.011
0.011
0.009
0.200
CIFAR-10
Input image
ResNec34
ProtoPNet
ProtoVNE
R
CelebA
image
ProtoVAE
ResNet34
MNIST with ResNet34 backbone
Input Image
ResNet34
ProtoPRett
ProtoVAE
ResNet34 KNEX
MNIST with a CNN backbone
Input Image
CNN
ProtoPNeet
ProtoVAE
CNN-KMEX
Figure 6: Normalized LRP maps computed on different. datasets and with different architectures.
21

---

Published in Transactions on Machine Learning Research (05/2024)
MNAS Tarsp:
PUNIST Terap
SJIN
CHAR1D Terrp:
Tersp:
ACC.
ACC.
ACC.
Rcb
Fott.
Rob
Falth
Sob
Farth
Reb
Falth
CELEBA
Tarsp:
Tanrp
Farsp:
Tarsp
Dry
ACC.
ACC.
neb-
Fehtb
Sull
Taith
Toith.
Taith
TUBT
ProtoPMet
PronsWM
R24+1
Figure 7: Radar plot summarizing the proposed quantitative evaluation framework for each dataset.
A.6.1 Similarity measures
In Figure 8, show the radar plot for the proposed evaluation framework for ablation study on the similarity measure (described in Sec 3.2 of the main text). The similarity measures used for this experiment include & f2, log(f2 + 1) - log(A2 f + <), ProtoPNEt's similarity measure log((E} + 1)/( c)), dot product similarity (dotprod), cosine similarity (cosine) and Normalized Euclidean Distance based similarity (NED). Since downstream operations expect that the more similar, the larger the value, l, l2, dotprod, and NED distances are multiplied by -1 before further processing.
The first observation is that distance-based measures (except NED) perform all very similarly in terms of transparency, accuracy, and quantitative diversity. This is not surprising since they differ mostly in terms of their tail for larger distances. This seems to be the key to the robustness as both é,, l2, and dotprod report the best behaviors: these have infinite asymptotes while the others converge to Overall, the based similarity performs the best and clearly outperforms all other measures.
A.6.2 Number of prototypes class
In the radar plot in Figure 9, we provide full qualitative comparison of KMEX with different numbers of prototypes per class I for all datasets. Note that for I. diversity undefined. In the case of STL110, an increase in the number of prototypes results in a decrease in transparency. This is anticipated as an excessive number of prototypes may result in the capture of redundant information. Apart from this, everything else remains invariant to the number of prototypes. However, from qualitative perspective, utilizing too few or too many prototypes per class could potentially impact explainability. As depicted in Figure 5, the information captured by the protolypes increases with the increase in protolypes but reaches a saturation point eventually. This experiment further demonstrates the effectiveness of the proposed evaluation framework selecting hyperparameters, such as the number of prototypes per class, for different models well datasets.
A.6.3 Clustering algorithms
It indeed possible utilize other kinds of clustering algorithms instead of K-Means. Yet, this algorithm must be based centroids in order to have prototypes the end. Among the centroid-based options, the
22

---

Published in Transactions on Machine Learning Research (05/2024)
HNGS anss.
PNNET Tarup.
SVIN Tanup
CIARRD Tarup:
ACC.
ACC.
ACC
Acc
fooe
50b
Fotth
Rob
fath
Rob
Falth
Rob
Falth
STL10 Tarsp
QUCKOAA Tanrp
CS1R8A Tansp
AVERACE Tanpp
ACC.
Acc.
Fenth.
Rutt
Taith
Salto
Tenth
Ruth
Taith
-5
log/: 3) logl:++.
log 1op-looo0--- +1
-drtpred.
codlre
-NID
Figure 8: Radar plot for ablation study on the similarity measure.
Teppp
Taopp
sopp
Tansp.
DIy
AzE.
ALE.
Taith.
Bett
Taith
Taith.
Butr
Taith
3 Taspp
Fanap:
3 Tanp.
Fansp.
Di
Fath.
Falts.
Fath.
Acc.
Rob Expt.
Falth Expl.
Feb
Tath Expl.
Rob
falth Expl.
Feb
Falth Expl.
Expl.
L-30
L=20
L-100
Figure 9: Radar plot for ablation study on the number of prototypes per class.
algorithms mostly derived from k-means like k-medoids (Park & Jun, 2009) GMM (gmm, 2019). The drawback of the latter is their high complexity and slow computation time in high dimensions. Bisccting K-Means (Steinbach et al., 2000) is an alternative algorithm that achieves K-Means-like clustering: splits the data in two using k-means until a convergence criterion is satisfied. However, it still follows a hierarchical structure. Additionally, for low number of prototypes per class, the centroids of Bisecting K-Means are unlikely be positioned on accumulation centers of the data as this might unsettle the trade-off between bisecting the clustering and the uniform cluster distribution prior of k-means.
23

---

Published in Transactions on Machine Learning Research (05/2024)
Table 13:
Prediction accuracy comparison of R34+KMEx with Bisecting K-Means (R34+ (is4+ttBiooct))KMMF.) Reported numbers are averages over runs along with standard deviations.
MNIST
IMNIST
SVHN
CIFAR-10
STL-10
QuickDraw
KMEX
99.4
92.3-0011
92.4
85.3
91.9
86.6#0.2
R34+ BisectingMMEx
75.1
62.0=6.7
57.143.
51.1=6.0
58.9+10
26.4=0.8
Table 14: Classification performance CUB200 dataset.
ResNet34+KMEx
ResNet34
Full images
Patches
Accuracy
78.6
78.4
70.0
Since the design of KMEx assumes non-hierarchical relationship among prototypes, which is in line with all the basclines used in this work, other non-centroid- based alternatives would be inferior to K-Means. To demonstrate this, in table 13, we report the accuracy achieved by BisectiinnigK (where we replace k-means with Bisecting K-Means) and compare with R34+KMEX. As observed, Bis+ttiseeinngMMMM consistently performs worse, with huge loss in accuracy for all datascts.
A.7 Preliminary results for a patch-based KMEx on CUB200
We report here preliminary results for the CUB200 (Welinder et al., 2010) dataset. The data consists 6000 images of 200 classes of birds. We also present a naive extension of KMEx at the patch level. The idea is to compute the patch prototypes right before the final average pool (7 x 49 patches per image). The class predictions for an image are then derived the majority vote of the KMEx predictions for each patch.
We report accuracy in percentage in Table 14 for ResNet34 and its KMEx based on the full images and patches both with 10 prototypes per class. Similarly to (Chen et al., 2019b), we show in Figure 10 the patch prototypes for 10 classes as red rectangles in the closest training image. Nole, that some protolypes capture background regions, indicating that the model has learned to exploit background cues.
The drop in accuracy when using patches not surprising, since the task more complex. Yet, the results are encouraging and highlight the versatility of KMEx.
A.8 Additional qualitative results
As mentioned in the main text, quantitative evaluation is not meant to replace the qualitative evaluation of SEMs. Therefore, in this section, we provide qualitative results including prototype visualizations for KMEx. We also show the visual classification strategy used by KMEx using the prototypes for different test examples, thereby exhibiting this (image) looks like that (prototype). We show this for both correctly and incorrectly test examples to further understand the decision-makinn process of our SEM. Additionally, we also qualitatively compare the diversity of prototypes for different SEMs.
A.8.1 Prototypes learned by KMEx: Figure 11
We visualize the prototypes of KMEx as the images in the training set that have the closest embedding in the latent space to the prototypes. The prototypes shown for MNIST, fMNIST, SVHN and STL-10 in Figure 11. It can be observed that the protolypes are very diverse and therefore efficiently represent different subgroups of classes.
A.8.2 KMEx: This looks like that: Figure 12
We now visually demonstrate the decision-makinn process of the proposed SEM. In Figure 12, for random test examples, show the closest prototype for the MNIST, IMNIST, CelebA, SVHN, STI.-10, and CIFAR-10
24

---

Published in Transactions on Machine Learning Research (05/2024)
pa1
#223
p221
7.33.52
p4R!
2.6.55
62)
812.71 p001
220) p221
620)
p42)
2400
Figure 10: Patch prototypes labeled with class id, importance and patch id.
datasots. It can be observed that the images look very similar to the closest prototypes, which illustrates that representative prototypes are learned. Additionally, we also demonstrate this behavior for misclassified examples (marked by a red rectangle) in Figure As can seen, the misclassified test. images look very similar to prototypes from different classes. Therefore, the simple this looks like that behavior exhibited by KMEx is able to provide meaningful and transparent decisions.
A.8.3 KMEx improves the diversity prototypes: Figures 13-15
We now compare the prototypes of different SEMs, thereby qualitatively comparing the diversity of the prototypes. For consistency and fair comparison, we visualize the closest training images for all the models. In Figure 14, visualize the prototypes learned by KMEx and ProtoPNet for the MNIST dataset. As
25

---

Published in Transactions on Machine Learning Research (05/2024)
observed, ProtoPNet's prototypes lack diversity, which is especially visible for classes 1, 4, 5, and 7. Applying KMFx ProtoPNet's s embeddings drastically improves the diversity, as shown in Figure 14 (right). Similarly, we compare the prototypes for KMEx and ProtoVAE in Figure 15 for the STL-10 dataset and for KMEx and FLINT in Figure 13. In all the cases, KMEx efficiently improves the diversity of the prototypes.
References
k-means as variational em approximation of gaussian mixture models. Pattern Recognition Letters, 125: 349-356, 2019.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1-46, 07 2015.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 215-223, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2009.
Srishti Gautam, Ahcene Boubekki, Stine Hansen, Suaiba Salahuddin, Robert Jenssen, Marina Höhne, and Michael Kampffmeyer. Protovae: A trustworthy self-explainable prototypical variational model. Advances in Neural Information Processing Systems, 35:17940-17955 2022.
Srishti Gautam, Marina M-C Höhne, Stine Hansen, Robert Jenssen, and Michael Kampfineyer. This looks more like that: Enhancing self-explaining models by prototypical relevance propagation. Pattern Recognition, 136:109172, 2023.
David Ha and Douglas Eck. A neural representation of sketch drawings. In International Conference Learning Representations, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - ECCV 2016, pp. 630 645, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46493-0.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representalions, ICLR 2015, Son Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basee learning applicd to document recognition. Proceedings of the IEEE, 86(11):2278 2224,, 1998.
Jeong Ryong Lee, Sewon Kim, Inyong Park, Taejoon Eo, and Dosik Hwang. Relevance-cam: Your model already knows where to look. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 14944 19944,14953, June 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Jan MacDonald, Stephan Wäldchen, Sascha Hauch, and Gitta Kutyniok. A rate-distortion framework for explaining neural network decisions. ArXiv, abs/19 hbs/1905.11 11092,
11092, 2019.
26

---

Published in Transactions on Machine Learning Research (05/2024)
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NTPS Workshop on Deep Learning and Unsupercised Feature Learning 2011, 2011.
Jayneel Parekh, Pavlo Mozharovskyi, and Florence d' A'Aché Buc. A framework to learn with interpretation. Advances in Neural Information Processing Systems, 01:22233-22285 2021.
Hae-Sang Park and Chi-Hyuck Jun. simple and fast algorithm for k-medoids clustering, Expert Systems with Applications, 36(2, Part 2):3336-3344 3341, 2009.
Michael S. Steinbach, George Karypis, and Vipin Kumar. A comparison of document clustering techniques. 2000.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pictro Perona. Caltech-uesd birds 200. Technical Report CNS-TR-201, Caltech, 2010.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/ bb//708.007777, 2017.
27

---

Published in Transactions on Machine Learning Research (05/2024)
MNIST
FMNIST
0
/
1
TESTAE
TIKAL
TENAN
2
2
2
2
2
3.001
IOUN
3
3
3
3
3
4
4
4
4
4
5
5
5
5
5
6
6
6
6
6
1000
7
7-
7
7
8
8
8
8
8
9
9
9
9
9
SVHN
STL10
O
SIND
JILK
JEAM
3
33
I5
61
6
6
T014
1:/11
TOLM
8
9
Figure 11: Prototypes learned by KMEx for several datasets. The class label written on the top of each prototype image along with its importance in the brackets.
28

---

Published in Transactions on Machine Learning Research (05/2024)
MNIST
JMNIST
CelebA
Input
Prototype
Input
Prototype
Input
Prototype
7
Looks
7
Looks
Looks
like
like
like
7
Looks
7
Looks
Looks
like
like
7
Looks
7
Looks
Looks
like
like
like
Looks
7
Looks like
Looks
like
like
SVHN
STL-10
CIFAR-10
Input
Prototype
Input
Prototype
Input
Prototype
Looks
Looks
Looks
Looks like
Looks like
Looks
55
Looks
Looks,
Looks
like
like
like
Looks,
Looks
Looks
like
Figure 12: This looks like that behavior exhibited by KMEx for MNIST, fMNIST, QuickDraw, SVHN STL-10 and CIFAR-10 datascts. The classification is based l-ncarost-ncighbor, therefore only the closest prototype for each input image is required as the explanation. Misclassified examples are marked in red.
KMEx
FLINT
FLINT KMEx
Figure 13: Prototypes learned by KMEx (top) and FLINT (middle) and FLINT-KMEx (bottom) for the CelebA dataset. KMFx generates diverse prototypes and is again additionally able to improve the prototypes learned over FLINT's embeddings.
29

---

Published in Transactions on Machine Learning Research (05/2024)
KMEx
ProtoPNet
ProtoPNet + KMEx
O
O
0
0
O
/
I
1
/
/
/
/
6
1
/
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
9
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
5
S
5
S
6
6
6
6
6
6
6
6
6
6
6
6
6
6
4
7
7
7
7
7
7
7
7
7
7
7
7
7
7
8
8
8
8
8
8
8
8
8
8
4
8
8
8
9
9
9
9
9
4
9
9
9
9
9
7
9
9
9
Figure 14: Prototypes learned by KMEx (left) and ProtoPNet (middle) and ProtoPNet-KMEs (right) for the MNIST dataset. KMEx generates more diverse prototypes and is additionally able improve the prototypes learned over ProtoPNet's embeddings.
KMEx
Proto VAE
Proto VAE + KMEx
Figure 15: Prototypes Iearned by KMEx (left) and ProtoVAE (middle) and ProtoVAE-KMEs (right) for the STL-10 dataset. KMEx generates more diverse protolypes and again additionally able to improve the prototypes learned over Proto VAE's embeddings.
30


Reviews:
[{"review_id": "FDcv2miTHi", "reviewer": "Reviewer_grUW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- Simple, easy to use method\n- Good results on standard datasets, particularly nice accuracy results for keeping performance similar to non-interpretable method\n\nCons:\n- Presentation is pretty hard to follow in current form. Please move notation definitions out of evaluation section and before you introduce the method. Readers have to hunt around the paper right now to understand it.\n- Results mostly on easy datasets, unclear how well the method generalizes to practical settings, though the initial investigation presented here is satisfactory overall", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces KMEx (K-Means Explainer). The general idea of this method is to take a base model (e.g., image encoder), learn a set of class prototypes and then predict based on choosing closest prototype for the image. They authors evaluate on several image datasets including MNIST, fMNIST, CIFAR-10 and others. They find roughly find equal performance for the base model (there's a bit delta on a few datasets, but not much). They additionally find the explanations are diverse and high quality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While the paper is interesting, the presentation of this paper in its current form needs a bit of work, as the method is hard to follow. For example:\n\n1. In section 3.1, the method is introduced. Yet, this notation is never defined for readers. What is s, z, and p_k?\n2. Only scrolling down to 4.2.1 do we see the definitions of notation. Why is the notation introduced in the second subsection of evaluation? This makes it extremely hard to follow.\n3. For the similarity measure, why use this measure? Why introduce \\epsilon? This needs to be made clearer when introducing the method.\n\nThough the evaluation is comprehensive overall, it would be interesting to see evaluation on more challenging or less data rich situations, because most of the evaluation is currently on datasets that are fairly easy. Are there scenarios where this method breaks down and the prototypes explanations fail to yield useful results? I could imagine this happening in situations with less finetuning data, where the model fails to generalize well to certain test instance.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KY683EtxhX", "reviewer": "Reviewer_J8fa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n- The proposed method is simple but efficient to provide explanation on top of most black-box models.\n- Some of the proposed evaluation metrics make sense and promising.\n\n**Weakness:**\n- Methodology section needs much more details. Currently, most details are missing or wrongly located.\n- It is unclear how the proposed SEMs can be used in \"safety-critical\" scenarios given that it is implemented on top of black-box models.\n- It would be good to clarify more that how the proposed method is different from post-hoc explanation methods with prototypes.\n- In multiple metrics, the proposed method shows worse performance. It is unclear why we should use the proposed method even though we see these worse performance.\n- More can be found in the requested changes section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors proposed a novel self-explainable model called KMEx.\n- The advantages of KMEx is that we can apply on top of most black-box models only with limited amount of computations (k-mean computations).\n- The authors also proposed some interesting metrics for SEM. Note that evaluation of interpretability is challenging.\n- The authors provided various experiments to show the pros and cons of the proposed method in comparison to the baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**1. SEMs on safety-critical scenarios**\n- Could we really utilize the proposed SEMs for safety-critical scenarios? \n- As explained, SEMs are preferred for safety-critical scenarios. However, it is on the representations of black-box models and those black-box models are totally un-interpretable. In that case, only with the interpretable components after representation extraction, could we use this method for \"safety-critical\" scenarios?\n- If the representation is not perfect, all the following explanations inherit the issues of the representations and we do not know the issues of the representation interpretably. So, I think it is very hard to utilize the proposed SEMs on \"safety-critical\" scenarios\n- Also, if this approach can be used for these scenarios, I think all the other post-hoc XAI approaches can be also utilized for the same scenarios.\n- The statement that KMEx is not a post-hoc method (in Section 3) is weak. Please explain more based on my above comments. \n\n**2. Difference from other prototype based XAI** \n- I think there are many prototype based XAI including this paper (https://papers.nips.cc/paper_files/paper/2016/file/5680522b8e2bb01943234bce7bf84534-Paper.pdf) and the papers in the related works of this paper.\n- It is highly unclear how this proposed method is different from these works.\n- It would be good to add more thorough related works about comparing the proposed method with other prototype based XAI.\n\n**3. Method section needs more details**\n- In Fig 1, what is D_tsp? It seems like the explanation comes in Section 4 which is way far from Fig. 1.\n- Also, based on Section 4, with KMEx, D_tsp = 0.0. In that case, 1-D_tsp = 1 for all? But why are the sizes of squares different? \n- Do we use the same \"k\" values per class?\n- What is z? What is p_k? What is epsilon?\n- It would be always good to define the notation first before using them in the equation.\n- It seems like the authors claim that trustworthiness of the explanation comes from the black-box model (this part: ...to both, thus similar explanations...). But we cannot say that the black-box model's embedding space is trustworthy.\n\n**4 Experiments (until Section 4.1)**\n- Table 3. Could you explain why some baselines have better results than the proposed method and even baseline method? Can this be the disadvantage of the proposed method?\n- Figure 2. Could we show the qualitative results of other methods that we can compare with? Also good to add how we can interpret PRP (like what is the meaning of the red part)\n- Table 4. It seems that the performance of KMEx is significantly and consistently worse than ProtoPNet. So, in that case, could we understand that KMEx is less robust? In that case, could we use this method for \"safety-critical\" scenarios where robustness is key?\n\n**5. Experiments (Section 4.2)**\n- Notation part should come much earlier (like Section 3)- Good to define D as well.\n- Good to provide the example of \"s\"\n- Equation (1). It seems like Equation (1) is wrong. See the inside of sigma. We can take out 1/KN part and now we can focus on the nominator with the sigma. As we know, each sample has one nearest prototype. Thus, the sum of the nominator should be N. In that case, D_tsp = 1/K regardless of the method. I do not think this is the definition that the authors want to define.\n- Also, I think using \"testing set\" instead of \"training set\" to evaluate D_tsp would be more meaningful because we would like to use this XAI in a test scenario. Working well with training data may not impact a lot.\n- What is p. in Equation (2)?\n- In Table 7, ProtoVAE is much better than KMEx. Is this the right understanding?", "broader_impact_concerns": "In general, this is XAI paper. Thus, it would be important to clarify that explanations that the proposed method is provided are not perfect and should not fully believe the explanation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9pg6DWohv3", "reviewer": "Reviewer_GmHw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Simple idea\n+ Variegated experiments and evaluation measures\n\n- Lack of runtime\n- Lack of deeper experiments on clustering\n- Lack of statistical significance", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents KMEx (K-Means Explainer) a post-hoc XAI technique to convert pre-trained models into prototypical interpetable-by-desing distance-based models. KMEx is compared against state-of-the-art prototypical-based approaches through an objective evaluation to assess predictive performance, the robustness of explanations, transparency, concept ghosting, faithfulness, and diversity of the explanations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The idea behind KME is simple and intuitive. This makes the approach widely usable and effective. We recommend the authors fix the following issues/missing aspects in order to improve the soundness of the paper.\n\nFirst, there is no theoretical nor empirical justification of why the similarity measure on page 4 is selected/designed. Both aspects should be curated, analyzing the sensitivity of  KME to alternative similarity measures such as  1-NormalizedEuclidean or other baselines based on existing metrics.\n\nSecond, a crucial aspect of the proposal is its usability. However, runtimes are not reported while I believe that these would further highlight the effectiveness of the proposal. \n\nThird, being based on k-Means I expected a study on the sensitivity of the evaluation metrics reported w.r.t. the number of clusters k. Besides, theoretically, any clustering algorithm can be used, but it is better if a centroid-based one is selected in order to rely on prototypes by construction. However, I would love to see that, indeed, k-Means is the best choice and not some other alternatives such as bisective k-Means that would also provide a hierarchy or even directly use a Hierarchical Agglomerative clustering with Ward criterion.\n\nFinally, the statistical significance of the experiments should be reported to show if the various alternatives are statistically exchangeable or f KME is a method that is significantly better than the others.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes to use a prototype classifier over a learned embedding model to visually attribute the classification decisions of a machine learning system to prototypes. The proposed method is simpler than prior approaches to tie visualizations to decisions (self-explainable models; SEM) and is evaluated similarly to prior work in the subfield.

The concerns of one reviewer, who noted a lack of Evidence for Claims, were on impact ("do not think this can be used in safety-critical scenarios") and on subjective weighing of trade-offs ("[simplicity] does not compensate for ... performance loss") which are not within the scope of the [evaluation criteria of TMLR](https://jmlr.org/tmlr/acceptance-criteria.html). Another reviewer requested more systematic investigations of the clustering step and evaluations on larger datasets; these extensions would be interesting but are not necessary for the work to have an audience with the present claims.

The AE proposes the following revisions:
- to clarify the relationship between the proposed method and existing work on prototype classifiers, especially those used in few-shot learning: [Snell et al. (2017)](https://arxiv.org/abs/1703.05175); [Pahde et al., 2021](https://arxiv.org/abs/2011.08899);
- to avoid aspirational and broad language (e.g., "we introduce our resource-efficient and universal method"; "... push towards more transparent deep learning-based decision-making via class-prototype-based explanations that are *guaranteed* to be diverse and trustworthy...") when describing the contributions of the work and instead describe the proposed method objectively;
- to add nuance to the "main contribution" claim that the evaluation framework is entirely quantitative, as the interpretability component still relies on the visual interpretation of the prototypes and the interpretation of the similarity of a test point to these prototypes;
- avoid any direct claims about the applicability of the approach to safety-critical scenarios, as the evaluation does not support such claims, and these claims are not central to the work;
- to make further improvements in clarity and readability, as noted, could be improved by several reviewers.
Decision: Accept (poster)