This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations
Authors: Chiyu Ma
Brandon Zhao
Chaofan Chen
Cynthia Rudin
Venue: NeurIPS 2023 poster (2023)
Keywords: deep learning
interpretability
prototype-based neural network
case-based reasoning

Abstract: We present ProtoConcepts, a method for interpretable image classification combining deep learning and case-based reasoning using prototypical parts. Existing work in prototype-based image classification uses a "this looks like that'' reasoning process, which dissects a test image by finding prototypical parts and combining evidence from these prototypes to make a final classification. However, all of the existing prototypical part-based image classifiers provide only one-to-one comparisons, where a single training image patch serves as a prototype to compare with a part of our test image. With these single-image comparisons, it can often be difficult to identify the underlying concept being compared (e.g., "is it comparing the color or the shape?''). Our proposed method modifies the architecture of prototype-based networks to instead learn prototypical concepts which are visualized using multiple image patches. Having multiple visualizations of the same prototype allows us to more easily identify the concept captured by that prototype (e.g., "the test image and the related training patches are all the same shade of blue''), and allows our model to create richer, more interpretable visual explanations. Our experiments show that our ``this looks like those'' reasoning process can be applied as a modification to a wide range of existing prototypical image classification networks while achieving comparable accuracy on benchmark datasets.

This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations
Chiyu Ma* Dartmouth College
Brandon Zhao"
Chaofan Chen University of Maine chaofan.chent maine.edu
Cynthia Rudin Duke University
Caltech myyzhoo@calteche
chiyu.ma.gred dartmouth.edu
cuutia@cs.duke.edl
Abstract
We present ProtoConcepis, method for interpretable image classification com- bining deep learning and case-based reasoning using prototypical parts. Existing work in prototype-basee image classification uses "this looks like that" reasoning process, which dissects a test image by finding prototypical parts and combining evidence from these prototypes make final classification. However, all of the existing prototypical part-based image classifiers provide only one-to-one compar- isons, where single training image patch serves prototype to compare with a part of our test image. With these single-image comparisons, can often be difficult identify the underlying concept being compared (e.g.. "is comparing the color or the shape?").. Our proposed method modifies the architecture of prototype-basee based networks instead learn prototypical concepts which are visualized using multiple image patches. Having multiple visualizations of the same prototype allows us to more easily identify the concept captured by that prototype (e.g., "the test image and the related training patches are all the same shade of blue"), and allows our model create richer, more interpretable visual explanations. Our experiments show that our "this looks like those" reasoning process can be applied modifi- cation to wide range of existing prototypical image classification networks while achieving comparable accuracy on benchmark datasets.
Introduction
As machine learning models are increasingly adopted in high-impact, high-stakes domains such as healthcare 2 self-driving cars (24] facial recognition [23] etc., the design of human- interpretable models has become essential for ensuring accountability and transparency of their decisions. In particular, class models called prototype networks have combined the power of deep learning with the explainability of case-based reasoning to make accurate, bumman-nteerpretbbee
26 36 These models make decisions by dissecting an image into informative feature patches, then comparing them to prototypical parts learned during training. Evidence of similarity to each prototy pical part then combined to make a final classification.
decisions on fine-grained image classification tasks 3
6
19
25
When evaluating the interpretability of image classification network, it is important to consider only the model's ' capability to explain its reasoning process, but also the quality of its explanations. The image features analyzed by prototype networks can encode variety of visual properties, such as color, shape, texture, contrast, or saturation. Because each prototype in the existing prototype network corresponds exactly to single patch of single training image, it can be difficult for human users to understand the exact semantic content underlying a prototype's visualization. Although there has been some work in quantifying the importance of each of these feature categories prototypes 20 the exact semantic content in an image patch can still remain ambiguous (e.g. "Texture is
Denotes equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

ProtoPool:
ProtoPool-Concepts:
Why is this classified as Brown Thrasher?
Why this bird classified Brown
Thrasher?
Looks
Dor Coraetp
Shay Drepcden
Ueeak
Coneet
Theater
Figure 1: Image of Brown Thrasher and how the ProtoPool (left) and ProtoPool-Conceee (ours, right) explain their decisions. Prototype classifications are made by finding patches in the image similar to learned prototypical parts. mingle-issualzzttonn methods such as ProtoPool can make visually ambiguous decisions when the semantic features underlying a prototype are unclear. Our method provides clearer explanations by providing multiple visualizations of prototypical concepts found in the test image.
important in this prototype, but what kind of texture in particular?"). The ambiguity underlying prototype visualizations has been shown cause a gap between visual explanations from prototype networks and human understanding of visual similarity [5]. Additionally, variety of prototype networks have been proposed which prototypical parts from a single image can serve as evidence for many different classes, thus reducing the total number of prototypes needed for the network to make classifications 199 26 25). The rationale behind this prototype sharing is that certain distinctive features (e.g. "long beak, solid black head, spotted belly") can be found in multiple classes, and thus do not need be represented as distinct class-specific copies. However, while reducing the number of prototypes simplify visual explanations by reducing the amount of considered evidence, these methods can often produce seemingly nonsensical explanations when a training patch from an image used spuriously evidence for different class.
Inspired by the difficulty identifying the meaning behind single-image prototypes, we propose ProtoConcepts, novel modification to prototype geometry enabling visualizations of prototyp- ical concepts from multiple training images. This allows human users to better understand the conceptual content of prototype by observing the features shared among its visualizations. An example of visual explanation produced by our ProtoConcepts is shown Figure (Tightt)rrigh)).. (right). Un- like previous prototype networks where prototype is visualized using single training image patch (Figure []] left), our ProtoConcepts visualizes each prototype with multiple visualizations. Because our model offers many visual examples for each prototype, the semantic meaning of each prototype can be determined with less ambiguity we can look at the commonality be- tween the visualized training image patches to infer the semantic meaning of each prototype. Our method can be applied to a wide range of prototype-basee networks, resulting better hu- man interpretability while achieving comparable classification accuracy. Our code is available at like-those ProteceeooooCCooottttaa
https://ggt wtt......o/eeeerryccccyaaaattsss
2 Related Work
Attempting to understand deep neural networks, posthoe explanation techniques such activation maximization 7
21
37 38
image perturbation 8 12] and saliency visualizations 32 27
fail to explain network's ' reasoning process, and their results can be risky and unreliable 18 1
On the other hand, prototype-based networks compare learned latent feature representations, called

---

prototypes, to the latent representations of an image to perform classification. These models are constrained to be intrinsically interpretable. The Prototypical Part Network (ProtoPNet) [ uses class-specific prototypes, where pre-determined number of prototypes are assigned to each class. Each prototype trained to be similar to feature patches from images of its own class and dissimilar to patches from images of other classes. This gives such class-specific models high discriminative power suitable for fine-grained image classification. In the end, prototype similarity scores are aggregated as positive evidence for each class in "scoresheet." The Transparent Embedding Space Network (TesNet) (36] improved on the class-specific ProtoPNet architecture by transforming the latent space geometry into hypersphere, and representing prototypes near-orthogonal basis vectors specific to a class subspace in the Grassmannian manifold.
Other prototype-basee based networks found that abandoning class-specifi prototype constraints allowed for a significant reduction in the number of prototypes, making explanations simpler and allowing prototypes to represent similar visual concepts shared in images of different classes. In [25] ProtoPNet first trained with enforced class specificity, and then a novel data-based similarity metric is used to "merge" highly similar prototypes across classes into a single prototype representing multiple classes, Neural Prototype Trees (ProtoTree) [19), the "scoresheet" of the ProtoPNet is replaced by binary decision tree, in which the scored presence absence of a prototype in each node determines the traversal path of the tree. Thus, each prototype in the tree represents either positive negative evidence for the classes the leaf nodes of the subtree for which the prototype is in the root node. Finally, in the ProtoPool network [26]. the soft assignment of prototypes to classes is learned throughout training, and is eventually transformed into hard assignment at test time.
Our work also relates to works that try to understand the semantic meaning behind a learned prototype of a prototype-based network. For example, in 220, Nauta et al. developed method to quantify the importance of various visual characteristics relevant to each prototype. Our approach is complementary to, yet differs from theirs in that we illuminate the concept behind each prototype by presenting multiple visualizations of the same prototype, thereby allowing one to better infer the semantic meaning behind each prototype.
Finally, our work related to posthoe methods in concept visualization such as TCAV and ACE 19]. Posthoc methods do not show any part of the model's actual reasoning process. Worse, their visual explanations may not be faithful to the model's reasoning process. Sometimes, multiple concepts have the same concept vector [as discussed in 5. and concepts may not be clustered in the latent space (since they are not trained to be that way), leading to meaningless concept vectors. Instead, our method is inherently interpretable, and derives its visual concept-based explanations directly from the decision-making process.
3 Methods
We begin with a brief general review of traditional prototype-bast baste models, then propose method, which represents prototypes as interpretable concepts with multiple visual explanations. We show how our method can be applied as modification to existing prototype-based networks, and introduce a novel training algorithm to learn meaningful prototypical concepts. Details for the implementation of specific prototype-baa based classification models are in the Experiments section.
3.1 Prototype Architectures
Existing prototype-based networks 3 combination of three components. First is convolutional neural network (CNN) f parameterized by weights Weony, which acts as a latent feature extractor on input images x. Next the prototype layer p considers the convolutional output fxx) as H x W "patch" vectors E IR each of which corresponds features from patch of the original image space. Each patch is compared to m learned prototype vectors P {p,) } where each prototype Pj E RD lies in the same space as the image's 's latent feature patches. The prototype layer p calculates a similarity score Ip, o fx) for each prototype py which monotonically decreasing with respect to the distance between PA and the closest image patch 7 E fxx) in the model's feature space. Finally, the prototype layer Yp is followed by a prototype class assignment mechanism h which assigns evidence logits to each class based on the calculated prototype similarity values 9p, fx) and class assignment weights wn. The evidence
19
25
26
36
can be understood architecturally as the
3

---

1 Logit value:37.9
Chestnut sided Warbler
3.67
CNN
=3.36
Logit value:27.0 Golden winged Warbler
=3.52
Ranked Predictions By logits
Convolution Layer f
ProtoConcepts Layer
Evidence Layer h
Figure 2: Architecture of ProtoConcepts. The convolutional layer includes all types of architectures, including ResNet, VGG and DenseNet. Evidence Layer h consists of a prototype class assignment mechanism and a fully connected layer. Depending on the specific prototype-based architecture, TesNet and ProtoPNet would have a one-to-one prototype class assignment mechanism, while ProtoPool and ProtoPShare would have shareable prototype mechanism among the classes.
logits are then normalized by softmax to yield predicted probabilities for given image belonging to each class.
In the final model, each prototype vector Pj is restricted so that is equal to latent patch fx;;) from some training image x,. Thus, the model's reasoning process can be interpreted visually as an evidence scoresheet comparisons between test image patches and salient features underlying previously-seen training image patches. We propose to modify the geometry of prototypes p, by instead representing them as ball in the latent space encompassing multiple training image patches, Mathematically. each prototype is ball P; = B(cj, (Ty) around central vector c, E RD with radius r, E R+ This ball representation has the property that the similarity scores of any latent patch vector Z, RD and any vector d Pj in the prototypical ball are constrained be the same. Hence, with this design, as long as prototypical ball Рj contains a number of latent training patches, we can visualize the prototype Pj by visualizing any of those latent training patches it contains. This set representation allows our prototypes represent concepts found in all latent training patches contained within prototypical ball, thereby providing richer semantic explanations than that of visualizing, just a single latent training patch. Implementing our concept geometry prototype- based model entails minor changes to the architecture and loss; model-specific implementation details are described in Sec. 4.1.1
3.2 Prototypical Concept Representations
The prototype layer Sp, of previous prototype-hased classification models uses one of two activation functions for calculating similarity scores between prototypes and image patches: log-based activation on inverse f2 distance, and a cosine similarity score. In this section, we formalize our ball representation of prototypical concepts with respect to the two corresponding latent spaces.
Log-based Activation: Models such ProtoPNet 3 and ProtoPool 26 use baseb func- tion g08 to compute a similarity score, which is monotonically decreasing with respect to the & distance. In particular, for the latent representation z f(x) of input image x, this function computes 9p, (z) = max log(([}² py 12 + 1) 1)/(l pyll + +)) with small constant to avoid numerical overflow. For prototypical concepts, propose analogous similarity function

---

corresponding to the distance to center c, of prototypical ball P, = B(c,, rÿy) bounded below by ry:
c; 2 2 c
2
log G + r3 12+6
log Pp
max [Epatcbos(2)
min
log
(I)
+
Cosine Similarity: Models such as TesNet [36 and Deformable ProtoPNet [6] have found that representing prototypes as unit vectors on a hypersphere can improve the accuracy of prototype-basd baaeed models. In these models, the prototype layer compares prototypes to latent image patches via the cosine similarity function Sp, (z) maxzEpatches(e) x² / [P [2)). This cosine similarity can be interpreted as a cosine activation function on the angular distance between normalized vectors on the hypersphere. For models with hypersphere-base based latent space, replace the cosine similarity function dos by calculating the cosine similarity to the ball ((cj,r5) on the hypersphere:
zTc, Cj
2005 Pp: (z)
max pEpachhes(s)
min
c cos(T,))
(2)
Both Eqs. and (2 ensure that for any patch vector E RD, if z; lies outside the ball Pý- its distance to Pj is the distance to its center Cjs and if z, lies inside Pj. its distance to Py is kept the
same as the radius of the ball. During training, pass-through estimator of each activation function is used to avoid a zero gradient when training patches lie within the prototypical ball.
3.3 Training Algorithm
The training of prototype-based networks is typically divided into three stages: (1) stochastic gradient descent (SGD) of layers before the last layer; (2) projection of prototypes; (3) optimization of the last layer h. Because of the ball representation of prototypical concepts, the second stage can be skipped as prototypical balls are visualized by the training patches they contain at the end of the first step.
SGD of layers before last layer: The first training stage prototype-hasee based networks aims to learn meaningful latent space that clusters important training patches near semantically similar prototypes. To do this, a joint optimization problem is solved optimize the network parameters using SGI) while keeping the last layer weight matrix fixed. To learn semantically meaningful prototypical concepts, we introduce two novel loss terms, Loisk and LRadi
k
CcIskk
min PjEP,. Pj-
min 2Eparches( (x.)) d(ž,p;), p.), CRad =
Σ PJCP p)=B(e,.r,)
2
(3)
Here is used as the distance function corresponding to the respective latent space geometry of the ProtoConcepts module. The minimization of Leisk encourages k prototypical concepts of the correct class to be close to at least one latent patch of each training image. The specific value of k is treated as hyperparameter and is chosen by cross-validation. As opposed to the traditional prototype cluster loss (3). which only penalizes the distance of the closest prototype latent training patch, that our top-k formulation is essential for ensuring that prototypical concept balls contain multiple training patches. The minimization of CRad encourages prototypical concept balls to be compact. This allows learned prototypical concepts to encapsulate more specific semantic features important for classification.
find
Prototypical Concept Visualisation: In previous prototype-based networks, the trained prototype vector py is projected (pushed) the nearest training patch produce single patch visualization. However, since we use a set representation for prototypical concepts, it is no longer reasonable to project the prototypical ball or central vector to the training patches. Instead, we visualize each prototypical concept by the training patches that are contained within each ball at the end of training. Thus, we can produce multiple visualizations from each prototypical concept and skip the projection step altogether.
Pruning and Fine-tuning: Because the number of prototypical concepts is fixed at the beginning of training, it is possible for a trained ProtoConcepts model to have small number of prototypical
5

---

concepts that do not contain any latent training image patches. In other words, these prototypical concepts do capture any visualizable latent features, and should not be considered when making predictions. To prune the non-visualized prototypical concepts, we design 0/1 mask on the prototype class assignment mechanism so that the given non-visualized prototypical concepts do not provide evidence for any image classes. After pruning these prototypical concepts, we finally perform sparse convex optimization on the last layer weight matrix Wp as in previous prototype-based methods [3]
Experiments
4.1 Case Study 1: Bird Species Identification
To demonstrate and examine the effectiveness of our method, we implement the ProtoConcepts module on the ProtoPNet. ProtoPool, and TesNet networks for the Caltech-UCSD Birds-200-2011 (CUB 200-2011) dataset [35]. The dataset contains 5,994/5,794 images for training and testing across 200 different bird species. We perform offline data augmentation, and crop training and testing images using bounding boxes provided with the dataset.
4.1.1 Implementation
In this section, we describe the implementation changes made to each prototype- based model for our experiments to incorporate prototypical concepts. A full list of exact parameter settings and training schedule for each model can be found in the Appendix.
The ProtoPNet model 3 consists of a CNN backbone, log-based prototype layer ااo and a fully connected layer [3. Each 200 classes assigned 10 class-specific prototypes, which are constrained to provide evidence only for their corresponding class through the last layer weighting. We train a Proteptet-Concepts model by substituting the prototype layer as Eq. (T) and then training with additional losses Lead and Leisk with A 10, radius initialization 7.5, and loss weights 0.01 and 0.8, respectively. The training schedule and other hyperparameters are unchanged from the original paper.
ProtoPNet:
The TesNet model [36] uses a cosine similarity function in its prototype layer to represent the latent space hypersphere and incorporates several hypersphere based loss functions during training. After modifying the activation function as in Eq. (2). we train TesNet-Concepts with a radius initialization of 8.05, and top-k cluster loss Lessk with k 3. We set the weight for the radius loss as 3 x 10-5 and the weight for top-k cluster loss as 0.8 with a learning rate of 1 x 10-1 during the warmup stage, and reweight the loss functions described its original paper with new training schedule.
TesNet:
The ProtoPool architecture 26 employs a log- based activation on the prototype layer, and adds prototype class assignment mechanism that allows sharing prototypes among multiple classes, thereby reducing the number prototypes. After modifying the prototype layer as in Eq. using radius initialization of 4.5 and top-k cluster loss LeKa with 10. We set a radius loss weight 10-3 3, top-k cluster weight 0.8, and learning rate for the radius as 0.5 x 10-3, The training schedule and other hyperparameters are left unchanged from the original paper.
ProtoPool:
(I) we train ProtoPool-Coneee
For baseline comparisons, we retrain CNN backbones and previous prototype-based models using publicly available code. As shown in Table [ for the CNN architecture, our ProtoConcepts network achieves similar test accuracy to corresponding baseline models.
Radius Study: We performed an ablation on ProtoPool-Conceptt with different set of radius initialization and top-k cluster loss with & = 10 under the same training settings described above. Intuitively, larger radius would encourage more prototypical concepts to capture information and an infinitesimal radius would be equivalent to single visualization. As shown in Table the number of visualizable prototypical concepts increases with the radius. In addition, radius also influences the accuracy of our ProtoConcepts model after pruning and fine-tuning. If the radius is too small, the number of visualizable prototypical concepts also tends be small this means that pruning will remove too many prototypes, and consequently adversely impact performance. On the other hand, if

---

Table 1: Comparison of ProtoConcepts implementation on ProtoPNet, ProtoPool, TesNet with the corresponding baselines and various backbone architectures. All algorithms on the same architecture perform similarly; the advantage is instead in interpretability.
Arch.
Model
Prototype #
Acc.[%]]
ProtoPNet-Concepte (ours)
1993 3
77.9 + 0.2
ProtoPNet 3
2000
78.0
VGG19
TesNet-Concepts (ours)
1862 20
78.2
TesNet [36
2000
0.1
Baseline (given in
N/A
75.1 0.4
ProtoPNet-Concepls (ours)
1980 = 2
80.7 0.4
ProtoPNet 3
2000
80.1 0.3
TesNet-Concepts (ours)
1994 + 2
81.4
Densenet- 161 11
TesNet [36
2000
81.5 + 0.3
ProtoPool-Concepts (ours)
199
81.5
Protopool [26]
202
80.3
Baseline (given in
N/A
82.2
TesNet-Concepts (ours)
1957 + 10
79.8 1.4
Resnet-34 10
TesNet [36
2000
80.7 00.7:033
ProtoPool-Concepds (ours)
185 1:4
80.4
ProtoPool [26
202
ProtoPool-Conceepts (ours)
188 2
85.2
Resnet-50 (iNat) 10
ProtoPool 26
202
85.5 +5.5+0111
ProtoTree 19
202
82.2
ProtoPNet-Concepts (ours)
1972 + 10
78.0 + 0.1
Resnet- 152 10
ProtoPNet
2000
78.0 0.3
Baseline (given in 3
N/A
81.5 + 0.4
radius is too large, each prototypical ball would capture too many latent representations whose semantic meanings may be too diverse, thereby making the prototypes less meaningful. Hence, we need to tune the radius to be within a reasonable range.
4.1.2 Interpretability
The reasoning process of ProtoConcepts network mirrors that of a traditional prototype network, but provides multiple visualizations of the prototypes used in its explanations. model classifies a testing image of Baltimore Oriole into the correct class. It worth noting that because of the shareable prototypes mechanism inherited from ProtoPool, the visualizations of learned prototypical concept now come from different classes. The most representative features that an ornithologist would use to classify an adult male Baltimore Oriolc are the bright orange and black plumage and the slender beak (28). As shown in Figure our model able to compare the test image patch containing orange and black plumage with the learned prototypical concept that captures similar features from its own class. Additionally, our model compares the patch containing the wings to the prototypical concept shared among Baltimore Orioles and other similar birds such as Orchard Orioles. Lastly, our model compares the part of image that contains slender beak along with its black head to the prototypical concept that captures the slender beak and black head of birds such as Orchard Orioles and Eastern Towhees.
shows how
Figure I shows an example of how a test image of a Brown Thrasher is correctly classified by ProtoPool and ProtoPool-Concepi Because of the prototype sharing mechanism, it is frequently the case that training image patches from birds of one class are used as evidence of a test image belonging another class. As result, the similarities presented by ProtoPool's visual explanations can be unintuitive. As shown in the plot, it is unclear how ProtoPool found the head of a Brown Thrasher to be similar to the blue head of Barn Swallow. Moreover, is hard determine whether the model is comparing the background or the belly of the Brown Thrasher, as shown at the bottom. However, comparisons made by our method are much clearer thanks to multi-vissulizaaions. Although both models attempt to compare the lower part of the bird to the learned prototypes, it is clear that our PotoPPol--Coneppt looking at the pattern on the belly, which an identifying feature of the Brown Thrasher (28).
7

---

Evidence this bird Baltimore Oriole:
Original Image (box showing part that looks like prototype)
Activation Map
Prototypical Concept visualizations
Training images where visualizations come from
Similarity Score
connection
contributed
3.72
x1.
x1.052
=3.91
3.59
x1.051
3.77
3.51
x1.040
=3.65
Baltimore Oriole:
33.24
Figure 3: The reasoning process of our network in classifying a test image as Baltimore Oriole. Test image patches are compared to prototypical concepts in latent space, then visual similarity scores are compiled as evidence for each class.
ProtoPNet:
ProtoPNet-Concepts:
Why is bird classified Female Ruby Throated Hummingbird?
Why this bird classifled Female Ruby Throated Hummingbird?
Looks like
Comes
Looks like
ProtoPNet:
ProtoPNet-Concepts:
Why this bird classified Male Ruby Throated Hummingbird?
Why this bird classified Male Ruby Throated Hummingbird?
Looks like
Looks like
Comes
Figure 4: Reasoning of a correctly classified Female Ruby Throated Hummingbird (top) and Male Ruby Throated Hummingbird (bottom) by ProtoPNet (left) and ProtoPNet-Concepus (right).
Figure 4 shows examples of how the two models correctly classify female and a male ruby- throated bummingbird. the top row, both of the models found the neck of female ruby-throated hummingbird from the given test image similar to the white feathers around neck from the training examples. On the other hand, although having red throat differentiates the male ruby-throated hummingbird from the female (28). the significance of a red-throat to correctly classify male ruby-throated hummingbird unclear. As shown the bottom left of Figure 4 the learned prototype from ProtoPNet is able to compare the prototype that contains red throat to that of the test image. However, the visualizations from ProtoPNet-Concept demonstrate that a red throat may not be all that the prototypical concept is representing. It appears to represent broader information about the bird's gray head and the transition its body, where the additional red or gray color located on the throat, and the white on the breast.

---

Table 2: Performance of ProtoPool-Concepts with different radius initializations using ResNet-
50(iNat) backbone
Radius
Acc.[%] before pruning Acc.[%] after pruning & finetuning
Prototype p
4.0
85.4 + 0.1
81.5 1.0
61
4.5
85.2 = 0.1
85.2 = 0.2
188 + 2
5.0
85.2 = 0.3
85.0 0.3
202
Table 3:
Accuracy comparison of ProtoPool-Concepts other shared prototype models the
Stanford Cars dataset
Arch.
Model
Prototype #
Acc.[%]
ProtoConcepts (ours)
150 6
89.4 0.5
Resnet 34 10
ProtoPSharc 25
480
86.4
Protopool [26
195
89.3 + 0.1
Densenet121
ProtoConcepts (ours)
194 +
87.1 + 0.4
Protopool [26
195
86.4 + 0.1
4.2 Case Study 2: Car Model Identification
In this case study, we implement our method for the Stanford Cars dataset 17 of 196 car types, using similar procedures and training algorithms as in the CUB-201 1-200 experiment. We trained a ProtoPool-Concene model with radius initialization 4.5, keeping other parameter settings the same as described in Sec. 1.1.1 We compare our method to self-reported accuracies of other shared prototype models ProtoPool [26] and ProtoPShare (25). Our model achieves similar test accuracy other shared prototype models, with the advantage of multiple visualizations for prototypical concepts.
More details, including visual explanations, can be found in the Appendix.
User Study
To show the reduction of ambiguity and resulting improvement in user interpretability, created distinction user study similar to HIVE [15] to compare our ProtoConcepts method with ProtoPNet; results are shown in Table We randomly picked ten samples from the test and calculated the top two predicted classes (i.e., the classes with the highest predicted probabilities according to the model) for each test sample. We then provided visual explanations from the most activated prototypes for these classes by ProtoPNet and ProtoPNet-Concepp choose which class the model actually predicting, looking only the visual explanations without the class probabilities. Examples of our user study are shown in Figure 7 We released our user study on Amazon Mechanical Turk and collected 50 responses from test takers with 98% survey approval rate to ensure the quality of responses, and removed response from both surveys after screening for nonsensical free response answers. We first ran a two-sided t-test on self-rated ML experience for the test takers from the ProtoPNet and ProtoPNet-Concepy The p-value is 1, and we are assured that there no statistically significant difference in machine learning experience between the two groups average. From the results of our survey, we are able to conclude that users with visual explanations from ProtoConcepts outperform those with visual explanations from ProtoPNet to statistically significant degree (p 0.003). Moreover, users given visual explanations from ProtoPNet were unable to outperform 50% random guessing to a statistically significant degree (p 0.289), whereas users given visual explanations from our model did (p 2.85 x 5)-5). Our survey results show not only that our model provides a notable improvement in user interpretability, but is able to improve non-expert user performance in difficult fine- ien-iirieed classification task whereas the previous ProtoPNet model cannot.
A test-taker was then asked to
6 Limitations
The interpretability of prototype-based models comes from visual explanations of classification decisions. Hence, the exact semantics (e.g.. "this bird has a long beak" or "this bird has spotted belly") underlying these visual explanations depend on the user's visual system to determine. We present a method for clarifying the user's semantic inference by offering multiple visualizations whose

---

Simulato the model
Simulate the modal
gloe un cnd
02
01 C20C000000 04
CRCR
C1 03
010200004
Or
Cf 0P
01
Of OECRUSE
Yed
Figure 5: User study example questions with architecture ProtoPNet (left), and ProtoConcepts (right). We showed each user 2 possible class explanations for given test image and asked the user to pick the correct class. We compared the results from two separate surveys, one in which visual explanations are generated by ProtoPNet, and one in which they are generated by ProtoConcepts.
commonalities can disambiguate our model's exact reasoning. While we find that our concept-based representation is helpful for this process, it is still unable provide explicit semantics explain its classifications. With recent progress in large vision-language hybrid models [22
16
4 34],
technique offering explicit semantics for visual explanations could be explored in future work. is important to note that in some visual domains, concepts may not have natural textual descriptions (e.g., in mammography or other radiology applications where it might look at a patch of particular type of breast tissue that is difficult to describe and has associated terminology).
7 Broader Impact
Interpretability an essential ingredient to trustworthy Al systems [3 improvement in interpretability to prototype-based networks, which are one of the leading techniques for interpretable neural networks computer vision. Thus, our technique can be used for important computer vision applications to discover new knowledge and create better human-Al interfaces for difficult, high-impact applications.
Our work presents major
Table Results of the user study comparing ProtoPNet to ProtoConcepts. We report the mean user accuracy for each model with a range of plus/minus 1.96 standard deviations. In addition, we conduct two I-sided t-tests on the results, with the alternative hypotheses that (1) users with ProtoConceptss visual explanations score higher average than with ProtoPNet and (2) Each model, respectively, achieves higher accuracy than random guessing. We conclude that users presented with visual explanations from ProtoConcepts outperform those with explanations from ProtoPNet to statistically significant degree.
Model ProtoConcepts (Ours) ProtoPNet 3
Mean Acc. [%]] Std. p-value (I) p-value (2)
62.1 +: 5.4
0.003
2.85 10
51.5 + 5.2
0.288
8
Conclusion
In this work, we present an interpretable method for image classification which incorporates proto- typical concepts with multiple visualizations to explain its predictions (this looks like those). Unlike previous works in prototype-basee method allows the user to use commonalities between visualizations of prototypical concepts to better infer the semantic meaning of prototypical concepts. We compare the visual explanations offered by our and previous prototype-basee methods and show that our model can achieve comparable accuracy to previous methods.
classification which offer only single prototype visualizations, our
10

---

Acknowledgements
We gratefully acknowledge support from the National Science Foundation under grants IIS- I1S22130550 RII T FEC-2218063
References
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps, Advances in Neural Information Processing Systems, 31, 2018.
[2] Alina Jade Barnett, Fides Regina Schwartz, Chaofan Tao, Chaofan Chen, Yinhao Ren, Joseph Lo, and Cynthia Rudin. IAIA-BL: A case-based interpretable deep learning model for classification of mass lesions in digital mammography. Nature Machine Intelligence, 3:1061-1070, 2021.
[3] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in Neural Information Processing Systems, 32, 2019.
[4] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Cheng, and Jingjing Liu. Uniter: Universal image-text text representation learning. In Computer Mision-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, pages 104-120. Springer, 2020.
[5] Zhi Chen, Yijic Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772 782, 2020. doi: 10.1088//42256-0000002666
[6] Jon Donnelly, Alina Jade Barnett, and Chaofan Chen. Deformable ProtoPNet: An interpretable image classifier using deformable prototypes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10265-10275, 2022.
[7] Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 2,41
[8] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2950-2958, 2019.
[9] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- explanations. Advances Neural Information Processing Systems, 32, 2019.
[10] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2015.
[11] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedingy of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4700-4708, 2017.
[12] Maksims Ivanovs, Roberts Kadikis, and Kaspars Ozols. Certurbation-based methods for explaining deep neural networks: A survey. Patlern Recognition Letlers, 150:228-234 2021.
[13] Mohd Javaid, Abid Haleem, Ravi Pratap Singh, Rajiv Suman, and Shanay Rab. Significance of machine learning in healtheare: Features, pillars and applications. International Journal of Intelligent Networks, 58 73, 2022.
[14] Been Kim, Martin Wattenberg, Justin Gilmer, Carric Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In International Conference Machine Learning, pages 2668-2677. PMLR, 2018.
[15] Sunnie SY Kim, Nicole Meister, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky. HIVE: Evaluating the human interpretability of visual explanations. In European Conference Computer Vision, pages 280-298. Springer, 2022.
[16] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and- languuge transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583-5594, PMLR, 2021.
[17] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine mne-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) 13)) Sydney, Australia, 2013.

---

[18] Thibault Laugel. Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard. and Marcin Detyniecki. The dangers of post-hoe interpretability: Unjustified counterfactual explanations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2801-2807, August 2019.
[19] Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine-grained im- age recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14933-14943, 2021.
[20] Mcike Nauta, Annemarie Jutte, Jesper Provoost, and Christin Seifert. This looks like that, because... explaining prototypes for interpretable image recognition. In Machine Learning and Principles and Practice of Knowledge Discovery in Databaxes: International Workshops of ECML PKDD 2021, Virtual Event, September 13-17, 2021, Proceedings, Part pages 441-456. Springer, 2022.
[21] Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in Neural Information Processing Systems, 29, 2016.
[22] OpenAL GPT-4 Technical Report. arXiv e-prints. art. arXiv:2303.08722 March 2023. doi: 10.48550/ arXiv.2303008774
[23] K Raju, B Chinna Rao, K Saikumar, and Nalajala Lakshman Pratap. An optimal hybrid solution to local and global facial recognition through machine learning. A Fusion of Artificial Intelligence and Internet of Things for Emerging Cyber Systems, pages 203-226, 2022.
[24] Schastian Ramos, Stefan Gehrig, Peter Pinggera, Uwe Franke, and Carsten Rother. Detecting unexpected obstacles for self-driving cars: Fusing deep learning and geometric modeling. In 2017 IEFE Intelligent Vehicles Symposiun (IV), pages 1025-1032. IEEE, 2017.
[25] Dawid Rymarczyk, Lukasz Struski, Jacek Tabor, and Bartosz Zieliński. Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1420-1430, 2021.
[26] Dawid Rymarczyk. Lukasz Struski, Michal Górszczak, Koryna Lewandowska, Jacek Tabor, and Bartosz Zieliñski. Interpretable image classification with differentiable protolypes assignment. In Computer Vision-ECCV 2022: 17th European Conference, Tel Avir, Ixrael, October 23-27, 2022, Proceedings, Part XII. pages 351-368. Springer, 2022.
[27] Ramprasaath R Selvaraju, Michael Cogswell, Ahhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient- -based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618 0266 2017.
[28] D. Sibley, The Sibley Guide to Birds. Audubon Society Nature Guides. Knopf, 2000. ISBN 9780679451228. URL httL.httts:///bbo - google, wtt.ccom//bbb.eee bttp//ooooiiidd-g-ll..-..-
[29] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[30] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Proceedings of International Conference on Learning Representations. 2013.
[31] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint 17Xiv:/706.000 03825,088558008 2017.
[32] Mukund Sundararajan. Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319 PMLR, 2017.
[33] Elham Tabassi. Artificial Intelligence Risk Management Framework (AI RMF 1.0). Technical report, National Institute of Standards and Technology, 2023.
[34] Hao Tan and Mohit Bansal. LXMERT:
Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100-5111, 2019.
[35] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltechucsd birds-200-2011 dataset. Technical Report CNS-TR TR-2011-000 California Institute of Technology, 2011.
12

---

[36] Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing. Interpretable image recognition by constructing transparent embedding space. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 895-904, 2021.
[37] Naoya Yoshimura, Takuya Mackawa, and Takahiro Hara. Toward understanding acceleration-base based activity recognition neural networks with activation maximization. In 2021 International Joint Conference on Newral Networks (IJNN), pages 1-8, IEEE, 2021.
[38] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, Hod Lipson. Understanding neural networks through deep visualization. In Deep Learning Workshop at the 32nd International Conference on Machine Learning, 2015.
13

---

A Ablation Study on Top-k Cluster Loss
To further demonstrate how top-k cluster loss. helps the model performance, we performed ablation study on ProtoPool-Concephe We trained ProtoPool-Concee by adding Leiss losses with the choices of & = 1,3, 5, 10 into the model with radius initialization 4.5 using ResNet50 (iNat) backbone. As shown Table although the classification performance of the model before pruning does not vary much when Lesa is introduced. the number of visualizable prototypes sharply increases, and the performance after pruning is significantly improved. While, discussed in the main paper, the Lebsk term ensures the quality of the latent space, encouraging prototypical concepts which enrich the interpretability the model.
Table 5: Model Performance ProtoPool-Conceppe under ResNet-50 (iNat) Backbone with different choices of losses. Acc. bf. prune refers to the performance of a model before the pruning procedure and Acc. finetuned refers to the performance after finetuning.
Model
Prototype p #
Acc.[%] bf. prune
Acc.[%]] finetuned
ProtoPool-Concepte + CCsstk kml
50 + 1
85,2 1 0.3
67.7 = 2.0
PootoPool-Concept. LCistk, ke3
85 + 8
85.3 + 0.3
83.5 + 0.5
ProtoPool-Concepee LCistk, k=5
108 +
85.5 + 0.1
84.4 + 0.3
ProtoPool-Concepts + LCist,. k=10
188
85.3 + 0.3
85.2 + 0.2
B
Training Parameters
In this section we present the training hyperparameters, objectives, and schedules used in our experiments for applying the ProtoConcepts module the ProtoPNet (3) ProtoPool [26] and TesNet prototype networks.
B.1 ProtoPNet
addition to the original losses ProtoPNet has, we trained ProtoPNet- ProtopNet-Coneqpts with Leisk, 1-10 and CRad. We set the corresponding weight for losses as demonstrated in Table/ Like ProtoPNet, we have 5 epochs of training as warmup where we trained the add-on layers, prototype layers, and the radius width, as shown in Table 7 Similar to the original ProtoPNet training schedule, We start training the convolution layer and last layer at the joint optimization stage for 10 epochs. Overall, 15 epochs trained hefore pruning and fine-tuning. Lastly, after pruning the prototypical balls that have no visualizations, we fine-tune last layer for 20 epochs.
Table 6: Parameter Settings for ProtoPNet-Concepte
Parameter
Weight
Cross Entropy Weights
1.0
Top-k Cluster Loss
0.8
Separation Loss
-..88
Radius Loss
0.01
Table 7:
Training Schedule for ProtoPNet-Concepts
Training Stage
Model Layers
Learning Rate
Duration
add on layers
3 10
Warmup Stage
prototype vectors
3 x 10-3
5 epochs
radius convolution f add on layers
0.5 x 10-4
1 10-4
3 x 10-3
Joint Stage
prototype vectors
3 x 10 3
10 epochs
last layer
1 10
Finetuning Stage
last layer
1 10-4
20 epochs
B.2 ProtoPool
We trained ProtoPool-Concee in addition to its original loss terms, with Leisk, 1-10 and Lxad. The corresponding weights for the losses shown in Table The training schedule is similar to that of the original ProtoPool.
14

---

Table 8: Parameter Settings for ProtoPool-Concepis
Parameter
Weight
Cross Entropy Weights
1.0
Top-k Cluster Loss
0.8
Separation Loss
0.08
Radius Loss
3 x 10-3
Table 9:
Training Schedule for ProtoPool-Coneepts
Training Stage
Model Layers
Learning Rate
Duration
add on layers
1.5 10
Warmup Stage
pool layer
1.5 10 10-3
10 epochs
radius
0.5 x 10-4
convolution f
x 10-3
Joint Stage
add layers
1.5 10-3
20 epochs
pool Layer
1.5 x 10
Finetuning Stage
last layer
1 10-4
15 epochs
We trained radius width only during the warm-up up stage. We set the first 10 epochs as the warm-up stage, and the following 20 epochs the joint optimization stage. Overall, the model is trained for 30 epachs before pruning and fine-tuning. We further trained additional 15 epochs to fine-tune the last layer weights after pruning the prototypical balls that not contain any visualizations.
shown in Table 9
B.3 TesNet
We trained ProtoPool-Concepp in addition its original loss terms, with LCkak, k=3 and LRad. The corresponding weights for the losses shown in Table TO The training schedule is similar that of the original TesNet, shown in Table [1] trained radius width during the warm-up stage and joint optimization stage. set the first 5 epochs as the warm-up stage, and the following 15 epochs as the joint optimization stage. Overall the model is trained for 20 epochs before pruning and fine-tuning, We further trained additional 15 epochs to fine-tune the last layer weights after pruning the prototypical balls that do not contain any visualizations.
Table 10: Parameters Setting for TesneetCo
Parameter
Weight
Cross Entropy Weights
1.0
Top-k Cluster Loss
0.8
Separation Loss
-0.2
Subspace Separation Loss
x 10-5
Orthogonality Loss
5 x 10-3
Radius Loss
3 x 10-5
Table 11:
Training Schedule for TesNet-Coneepts
Training Stage
Model Layers
Learning Rate
Duration
add on layers
3 x 10-3
Warmup Stage
Prototype Vectors layer
3 x 10-3
5 epochs
radius
1 10-4
convolution f add on layers
1 10-4
Joint Stage
3 x 10-3
15 epochs
Prototype Vectors Layer radius last layer
3 x 10-3
1 x 10-5
Finetuning Stage
1 X 10-4
15 epochs
C More Examples for ProtoPNet-Concepts Visualizations
In this section, we further demonstrate how multi-visualization solves the ambiguity of prototypes learned by ProtoPNet. The visualizations shown trained with a VGG-19 29 backbone for both ProtoPNet and
15

---

ProtoPNet:
ProtoPNet-Concepts:
Why is bird classified as Least Tern?
Why this bird classified a Tem?
Looks
Why is this bird classified as Chestnut sided Warbler?
Why is this bird classified as Chestnut sided Warbler?
Looks
Looks
Comes
Figure 6: Reasoning of a correctly classified Least Tern (top) and Chestnut-sided Warbler (bottom) by ProtoPNet (left) and ProtoPNet-Concepe (right). Some visualization examples of the Prototypical concepts are shown in the yellow bounding boxes, and compared with patches from the test image.
ProtoPNet-Concepts
how our method reduces the ambiguity inherent in the ProtoPNet architecture. The top row provides an example of how ProtoPNet and ProtoPNet-Concepe compare and classify a test image of Least Tem. ProtoPNet highlights the bird's leg the test image, which one can harely see in the model's learned prototypical features (e-g., it comparing the wings? Or is it comparing the feet, which comparing
too tiny see in the image?) the other hand, is clear ProtoPNet-Coneee even though the bounding boxes on the test image include the bird's feet. Similarly, the bottom row shows each model's 's reasoning for correctly classified test image of Chestnut-sides Warbler. It is not clear in this case barely-captured yellow crown feature. On the other hand,
wings,
whether ProtoPNet is comparing the background the many examples the yellow crown from the multi-visualizatiinn of ProtoPNct-Conceppe elarify that the prototype represents this crucial feature of a Chestnut-side sided Warbler (2%
More examples of unclear comparisons can found in Figure 7 As shown the plot, the ProtoPNet finds the green wing and spotted breast of the test image of a spotted catbird similar to a green wing with a black spot in the middle. On the other hand, the protolype learned by ProtoPNet-Concepe suggests that the prototypical concept focuses the texture. In Figure 7 it is also unclear if ProtoPNet finds the feathers the red feet of the Forster Tern be similar its learned prototype visualizations. However, we can clearly see from the ProtopNet-Conceps
Concepts multi-visuaii visualizations that the prototypical concept is actually the grey wing and white belly instead of the red feet.
Some types of ambiguities are specific to bird types. For instance, it is difficult to determine what the prototypes represent for birds that are black everywhere. For example, shown in Figure it is hard see what the prototypes learned by ProtoPNet ProtoPNet-Conee Conceppsssss suggest that color and possibly the red eye and shape of the head appear to be the second concept.
capturing except for the dark feathers. Multiple visualizations learned by shape the bird's belly are the first concept, whereas
Ambiguity of the learned prototypes can arise from low quality test images. For example, as shown in Figure 9 visualizations from ProtoPNet have low brightness/contrast on the prototype birds. It is hard to see any distinguishable features captured by those prototypes. On the other hand, although similar visualizations used by PoatoPNet-Conneppee other visualizations within the ball demonstrate the prototypical concepts (here, blue and white stripes apparently).
Lastly, having multiple visualizations can also help to draw inferences on misclassified cases. Figure T0 and Figure([] [[] cases when both ProtoPNet and ProtoPNet-Coneepe misclassify the test image. In Figure To it is unclear how ProtoPNet found the test image of Cedar Waxwing to be similar to Cardinal, except for the color of the fruit it has in its beak. On the other hand, multiple visualizations offer additional information such as trianglular silhouette the head, dark strip of feathers around eye, pointed features at the back of the head. Similarly, in Figure [1] ProtoPNet Coctt-pppss is able to offer more specific information than ProtoPNet, such as the grey and white color contrast, black head and the shape of the body. Here, the algorithms are showing why it difficult to tell these birds apart.
16

---

ProtoPNet:
ProtoPNet-Concepts:
Why is blrd classified Spotted Catbird?
Why is this bird classified a Spotted Catbird?
Looks
Looks
ProtoPNet:
ProtoPNet-Concepts:
Why classified as Tern?
Why this bird classified Tem?
Looks like
Looks
Comes
Figure 7: Reasoning of a correctly classified Spotted Catbird (top) and Forster Tem (bottom) by ProtoPNet (left) and ProtoPNet-Concppe (right).
ProtoPNet:
ProtoPNet-Concepts:
Why is this bird classified Bronzed Cowbird?
Why this bird classified Bronzed Cowbird?
Looks
Comes
Looks like
Looks
Looks like
Comes from
Figure 8: Reasoning of a correctly classified Bronzed Cowbird by ProtoPNet (left) and ProtoPNet- Concepts (right).
17

---

ProtoPNet:
ProtoPNet-Concepts:
Why is this blrd classified Florida Jay?
Why this bird classified a Florida Jay?
Comes
Looks
Looks like
Looks like
Figure 9: Reasoning of a correctly classified Florida Jay by ProtoPNet (left) and ProtoPNet -onccppsssss (right).
ProtoPNet:
ProtoPNet-Concepts:
Why is this bird falsely classified Cardinal?
Why this bird falsely classified Cardinal?
Looks like
Looks like
Figure 10: Reasoning of Misclassified Cedar Waxwing to Cardinal by ProtoPNet (left) and ProtoPNet-Concepss (right).
18

---

ProtoPNet:
ProtoPNet-Concepts:
Why falsely classified as Long tailed Jaeger?
Why this bird falsely classified Long tailed Jaeger?
from
Looks like
Looks like
Figure 11: Reasoning of Misclassified Black Tern to Long Tailed Jaeger by ProtoPNet (left) and ProtoPNet-Coneppe (right).
D More Examples for ProtoPool-Concepts Visualizations
this section, we present more examples
visualizations reasoning process for PrttoPoll-Concepe
are obtained from the model with ResNet 50(iNat) [T0] backbone. Because prototypes are shared among different classes, the information extracted by ProtoPool can be unintuitive, especially when the prototype docs not come from the same class given test image. This problem affects ProtoPool well ProtoPool-Coneee Col-CConcppts- while shared prototypes can, in some cases, make concept clearer borrowing good examples of that concept from other classes, our experience seems to make the concepts less clear. For example, in Figure 12 hard see how the beak of Cedar Waxwing is similar to the prototype visualization captured by ProtoPool. Moreover, similar to problem Waxwing from the top is also unclear. On the other hand, ProtoPool-Cool Concepts may be capturing color contrast throat and breast that is common to Great Crested Flycatcher and Cedar Waxwing, etc.. but since the contrast of the test image is not particularly good, the human might determine not to trust that
saw with ProtoPNet, whether the prototype is capturing the feet or the belly of the Cedar
ProtoPool:
ProtoPool-Concepts:
Why is bird classified Cedar
Why this bird classified Waxwing?
Waxwing?
Looks
Looks
Comes
Connupt Pesturs
Fiecsceer
Ssiceal Fasturss
Reasoning of a correctly classified Cedar Waxwing by ProtoPool (left) and ProtoPool- Concepts (right).
Figure 12:
19

---

ProtoPool:
ProtoPool-Concepts:
Why classified as headed Blackbird?
Why this bird classified
headed Blackbird?
Looks
Looks
Comas
Esscett Fasturss
hellonte taiabe
Crnning Grasbons
Looks
arnged Nechiind
Figure 13: Reasoning of correctly classified yellow-headed headed blackbird by ProtoPool (left) and ProtoPool-Concepls (right).
ProtoPool:
ProtoPool-Concets::
Why is bird classified Red Bellied Woodpecker?
Why this bird classified
Bellied Woodpecker?
Looks
Looks
Comas
Cancul Faskers
digy calare
Looks
Poclwes
drall Coorud
Flyantee
Figure 14: Reasoning of a correctly classified Red Bellied Woodpecker by ProtoPool (left) and ProtoPool-Coneept (right).
20

---

Training images where the Evidence for this bird being White breasted Kingfisher: visualizations come from Original image (box showing part that looks like prototype)
Similarity
Activation Map
Prototypical Concept visualizations
connection
contributed
3.83
x1.03
=3.93
3.72
x1.02
=3.80
Total Points
35.0
White broasted Kingfisher:
Figure 15: Reasoning of a correctly classified White- breastb Kingfisher by ProtoPool-Concepts
Evidence for this bird being Cedar Waxwing:
Training images where the Similarity Class
Points contributed
Original Image (box Activation Map showing part that looks like prototype)
Prototypical Concept visualizations
visualizations come from
Score
connection
3.82
x1.04
=3.97
Powesied
3.78
x1.02
=3.85
Doea Miewee Ovending Comboet
Total Points
Waxwing:
35.5
Figure 16: Reasoning of correctly classified Cedar Waxwing by ProtoPool-Concepts
comparison. ProtoPool-Concepe also appears to capture the yellow and black color pattern on the bird's head. Similarly, in Figure 13 the comparison made at the bottom left for ProtoPool is hard to identify with single visualization, while ProtoPool-Coneee the concept
offers more information such the yellow head and black body, though cohesive, since it includes bird with red and white stripes. might determine that this concept may not trustworthy, which a useful piece information. It is worth noting that the prototypical concept of the yellow-headed blackbird shown at the top of Figure [3 shared with shares
analyst sceing this concept
the Cedar Waxwing as shown in the bottom row of Figure [2] Figure 14 shows that ProtoPool-Cone part of prototype concept with the Cedar Waxwing: this concept appears indicate dark-and-light textural pattern that is similar the wing the Red Bellied Woodpecker. However, the comparison made by ProtoPool ambiguous. Figure 15 Figure 16 further demonstrate how the evidence layer and prototype activation are involved in the reasoning process. As explained in the main paper, a weighted score would be computed for each of the prototypical concepts, and total logit score would be calculated for the final prediction.
21

---

E More Examples for TesNet Visualizations for CUB dataset
this section, present some examples the reasoning process of TesNet- Canccp-Consepss and its comparison with TesNet. The visualizations were obtained from TesNet and TesNet-Concepts trained with a VGG19 backbone. Similar to ProtoPNet and ProtoPool, ambiguity arises from low quality images. In Figure 17 the prototype visualization of TesNet for Florida Jay shown at the top has low contrast. Whether the protolype is capturing the belly the feet is unclear. Although similar prototype visualization is also captured by the prototypical ball for TesNet-Concepls as shown the top right of the plot, the other visualizations reveal the prototypical concept as the white belly near the blue feathers. Similarly, the multi- cualti-visualization shown at the bottom suggests that the prototypical ball likely represents the bird's head and beak. Figure shows the prototype logic for identifying Cedar Waxwing by the color gradient the bird's throat. Some of the image patches chosen by Tesnet-Coneepts look almost identical the chosen patch of the image; this is more convincing than the comparisons from TesNet without concepts.
More Examples for Protopool-Concepts Visualizations and Model Performance for Car dataset
In this section, we present some of the visualizations for ProtoPool-Concepts on the Car dataset. The visualiza- tions are from the models trained with DenseNet121 [T]] backbone. Similar to ProtoPool-Concepts for the CUB 200-2011 dataset, the sharcable prototypes still make the comparison unintuitive. As shown in Figure T9 ProtoPool seems to compare the front a BMW to that an Audi. It appears find similarity between the kidney grille of the BMW to the front of the Audi, or could be the shape of the lights that are similar. On the other hand, the multiple visualizations from kidney grille of the BMW. The other visualizations further show that the PotooPoll-Cnneyy found part of the the hottom, just in front of the tires, is similar the same part of the other types of SUVs. Figure 20 shows that our model finds the wheel the Audi TT RS Coupe is similar to the wheels of different types of Audis.
suggest that the model is actually capturing the
Table [2] shows some additional performance experiments for PottoNettConneepe and ProtoPool-Concee on the Cars dataset. The training schedule and parameter settings are the same reported above. The initialized number of prototypes for ProtoPNet-Coneee radius initialization for ProtoPNet- Coneeett 7.5, and the radius initialization for ProtoPool-Coneye is 4.7. As shown in the table, ProtoPNet- Conceptsssssss under the DenseNet12) baseline has better performance with fewer visualizable prototypes than ProtoPNet, and its performance is very close to the baseline. On the other hand, ProtoPool- under a DenseNet161 backbone yields better performance than ProtoPNet with much fewer visualizable prototypes.
and ProtoPool-Coneeph are 1960 and 195 respectively. The
TesNet: Why is this bird classified as Florida Jay?
TesNet-Concepts: Why this bird classified Florida Jay?
Looks like
Comes
Looks like
Looks like
Looks like
Comes from
Figure 17: Reasoning of a correctly classified Florida Jay by TesNet (left) and TesNee-Concepts (right)
22

---

TesNet:
TesNet-Concepts:
Why is this bird falsely classified as Cedar Waxwing?
Why this bird falsely classifled Cedar Waxwing?
from
Looks like
Looks like
Looks like
Figure 18: Reasoning of a correctly classified Cedar Waxwing by TesNet (left) and TesNet- Conce--ss (right)
ProtoPool: Why is classifled BMW X3 SUV 2012?
Pronepool-Concepts: Why this classified BMW
SUV 2012?
Looks
Looks
Comas
Looks
Reasoning of a correctly classified BMW X3 SUV 2012 by ProtoPool (left) and ProtoPool- Concepts (right).
Figure 19:
23

---

ProtoPool:
ProtoPool-Concepts:
Why is this car classified as Audi TT RS Coupe 2012?
Why this car classified Audi
Coupe 2012?
Looks
Looks
Comes
Looks
Figure 20:
Reasoning of a correctly classified Audi TT RS Coupe 2012 by ProtoPool (left) and ProtoPool-Concepee (right).
Table 12:
Accuracy comparison of ProtoPool-Conceptt to other shared prototype models the Stanford Cars dataset
Arch.
Model
Prototype #
Acc.[%]
ProtopNet-Coneepts (ours)
1941 7
88.4 +
DenseNet121
ProtoPNet 3
1960
86.8 1.8:0.1
Baseline (given
N/A
8.17
ProtoPaol-Concepte (ours)
195
90.4
DenseNet161
ProtoPNet 3
1960
89.5
Baseline (given in 36
N/A
G
Training software and platform
We implemented ProtoConcepts ProtoPNet. ProtoPNet. and ProtoPool using PyTorch. The experiments were run on NVIDIA Tesla V100 NVIDIA RTX A5000. Using the ProtoConcept module with any base architecture has similar training times to using the base architecture alone.
24


Reviews:
[{"review_id": "L4Fhe3aH0Y", "reviewer": "Reviewer_8ZAP", "summary": "This paper improves interpretability for prototypical learning. Most previous methods use a single prototype as an interpretation -- and this is not as informative as we don't know what portion of the prototype corresponds to our target image. \n\nThe authors introduce a *prototypical ball* to interpret what prototypes are used for inference. Instead of selecting a single prototype, training samples from this prototypical ball are sampled.", "questions": "* Do you have an ablation on the loss scaling parameters in the supplementary (Tables 2,4)? How were these $\\lambda$ chosen? How does performance vary with this?\n\n* Why was $k=10$ chosen for $\\mathcal{L}_{\\text{clstk}}$? Do you have an ablation on this?\n\n* Continuing from the section above, I think a figure showing the ball, it's center, and radius with its prototypes would help readers understand this work more clearly. For example in Figure 1, the multiple visualizations of prototypical concepts doesn't clearly identify how those multiple visualizations are obtained (i.e. you could use the same figure if the results were the same using a k-NN algorithm) (same for figure 2)\n\n* In equation three, I think it's important that you show the full loss equation i.e. $\\mathcal{L} = \\mathcal{L}_{\\text{other terms}} + \\lambda_1 \\mathcal{L}_{\\text{clstk}} + \\lambda_2 \\mathcal{L}_{rad} $. It would be clear as to what the loss weights mean in Section 4.1.1\n", "limitations": "The authors have adequately addressed the negative societal impacts.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*Originality*: I am not completely informed on all prototypical learning methods, but it appears that this method is original. In addition to the novel use of a ball for multiple visualizations, they describe a training scheme specifically for their method. This contributes to the originality of their work. \n\n*Quality*/*Clarity*: The paper is generally presented well, with descriptions of previous work and their drawbacks, as well as what contributions improve the current work. I think some of the descriptions of previous work is not very clear, but I understand space constraints preclude a better description. I believe that the main contribution of this paper can be more clearly elucidated in the figures, which I describe in ``Weaknesses\" section.\n\n*Significance*: I think this work is important to the community. While it matches performance of previous methods, it presents a new way of interpreting the results of algorithms. Integrating this novelty into previous works is adds to its significance.\n", "weaknesses": "* A better explanation of interpretability would stregthen the paper. The results do improve interpretability, but the fundamental question (that is introduced in the paper) still remains: why are these selected as prototypes? There are more images and we humans can make connections and inferences, but it's still unclear as to why these are the prototypes within the ball.  I do see this explicitly addressed in Section 5, but I still think it is a weakness of the paper. I think an analysis like your reference [16] would strengthen your paper signficantly. I don't know if it is critical to this paper, but it would be a good follow-up paper.\n\n\n* Figure 2 is fairly unclear. It's difficult to understand what your method is actually doing. It's not clear how the logits are obtained either. Overall, if I were to look at that diagram without the help of the text, it would be quite unclear as to what your novelty is and how it is implemented. The ball that you introduce, which is your main contribution, is missing from the figure.\n\n\nA couple small things:\n\n* Please add reference for ProtoPool in Figure 1 and line 209.\n* Reference [16] and [18] look very similar -- are these the same?\n* The text \"ProtoConcepts Layer $g_p$\" in Figure 2 looks compressed compared to the rest of the text in that figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review comments. We are happy to make some clarifications as follows:\n> Why are these selected as prototypes?\n\n**A**: These prototypes are learned. The **data** chooses these prototypes. The interpretation of the prototypes as a concept is done by the human. The algorithm doesn't know what (e.g.) \"long beak'' is - it's supposed to discover a concept as a bunch of images of beaks. If you had only one example of a beak, it would just be a picture of a bird's head, and you wouldn't necessarily know what aspect of the beak was important (color? length? shape? combination of those?), but if you had several examples, you could detect the learned concept. Moreover, we performed post-hoc-analysis similar to [1] on the ProtoPool-Concepts examples, and the analysis can be found in the global rebuttal Figure 1. It is worth noting that the texture and shape, as shown in the result, are the two most important factors in learning the \"long beak\" concept for the given test image of Brown Thrasher. (Note that the ability to learn the prototypes from data is not a weakness; it is a strength since it would find concepts that may not be defined yet by a human. It also makes things a lot easier to assemble the data since the concepts do not need to be defined in advance.)\n>Figure 2 is fairly unclear. It's difficult to understand what your method is actually doing. It's not clear how the logits are obtained either. Overall, if I were to look at that diagram without the help of the text, it would be quite unclear as to what your novelty is and how it is implemented. The ball that you introduce, which is your main contribution, is missing from the figure.\n\n**A**: Thank you for your comment. Because we present a general method which can be added to a wide range of prototype-based models, each of which has a different architecture for calculating logits, we tried to create a figure which describes the prototype-based reasoning process in generality. However, we agree that it can be difficult to understand how the architecture works for a specific model given only the information in Figure 2. Therefore we will create separate architecture figures for each prototype-based model in our experiments and add them to the supplement, so readers can understand the prototype-based reasoning process for each model. \n\n>Do you have an ablation on the loss scaling parameters in the supplementary (Tables 2,4)? How were these chosen? How does performance vary with this?\n\n**A**: The parameters from Tables 2 and 4 are directly borrowed from the previous corresponding model, ProtopNet, and ProtoPool. These are obtained through finetuning by the previous works. For TesNet parameters and the weight of radius loss, we fine-tuned the weight to accommodate our method to the model by grid search. An ablation study on different choices of radius can be found in the Main paper Table 2.\n\n>Why was K=10 chosen for clstk? Do you have an ablation on this?\n\n **A**: K is a hyperparameter of our model and is selected through parameter tuning by grid search. The ablation of different choices of Ks on the ProtoPool-based model can be found in Supplementary material Table 1 .\n\n>A couple small things:\nPlease add reference for ProtoPool in Figure 1 and line 209.\nReference [16] and [18] look very similar -- are these the same?\nThe text \"ProtoConcepts Layer \n\" in Figure 2 looks compressed compared to the rest of the text in that figure.\n\n**A**:Thank you for catching these small errors. We will fix these issues in our final manuscript.\n\n[1] Meike Nauta, Annemarie Jutte, Jesper Provoost, and Christin Seifert. This looks like that, because ... explaining prototypes for interpretable image recognition. In Machine Learning and Principles and Practice of Knowledge Discovery in Databases, pages 441–456. Springer International Publishing, 2021. ISBN 978-3-030-93736-2."}, {"review_id": "lD45FdJ5bC", "reviewer": "Reviewer_p3JV", "summary": "This paper extends the patch comparison of prototypical part-based classifiers from one-to-one patch to one-to-multiple (e.g. from this looks like that to this looks like those). They compare their proposed network with Proto-PNet variants and show no classification accuracy improvements and little evidence of better interpretability. The proposed method was tested on CUB and Cars as previously done in ProtoPNet variants.\n\nI have read the author’s rebuttal and adjust the rating accordingly!", "questions": "The 1vsmany patch-wise comparison has been studies and they even conducted human studies on interpretability. How this work differs from them [4].\n\n[4] visual correspondence-based explanations improve ai robustness and human-ai team accuracy", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The research is well positioned in the literature of prototypical part-based classifiers.\n2. The extension from 1vs1 to 1vsmany is interesting and could help improve human understanding tasks that all prototypical part-based classifiers fall short on [1] \n\n[1] hive: evaluating the human interpretability of visual explanations", "weaknesses": "There are multiple weaknesses of this paper, mostly about the novelty.\n1. I see the extension is incremental though well motivated. There is no surprise in the classifier accuracy compared to other ProtoPNet variants. I always see (all of them) they perform roughly the same. I believe the \"big fish\" is to really make this prototypical part-based explanations usable for humans. \n2. Although it was motivated by the lack of interpretability of prototypical part-based classifiers, the analysis of model interpretability is superficial and not spot-on. I believe maybe this work also fails to help humans [1] as its siblings [2,3]. The authors could improve the interpretability analysis by running more extensive study (both automatic and human studies) rather than dissecting sample-wise visualizations that could be easily cherry-picked and misled.\n3. I think the paper lacks the analysis of computation cost (i.e. 1vs1 and 1vsmany comparison).\n\n[2] ProtoPNet\n[3] ProtoTree", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. However, we would like to make some clarifications as following:\n>I believe maybe this work also fails to help humans [1] as its siblings [2,3]. The authors could improve the interpretability analysis by running more extensive study (both automatic and human studies) rather than dissecting sample-wise visualizations that could be easily cherry-picked and misled.\n\n**A**: As per your suggestion, we evaluated the added interpretability of our ProtoPConcepts module using the **distinction** survey task in HIVE[1]. In this task, we asked participants to select the correct prediction out of two options based on provided visual explanations, and found that our ProtoPConcepts module allowed for a statistically significant increase in user interpretability. Please see the global rebuttal for exact study details and results. We thank for the suggestion of a human study and believe that our survey results objectively show the interpretability benefit from our method.\n\nIn terms of cherry-picking, we included a lot of examples in the appendix so the user can see whether we are cherry picking (we are not).\n\n>I think the paper lacks the analysis of computation cost\n\n**A**: Thank you for this input; we find that our method is actually more computationally efficient than previous prototype-based methods. Although our method allows for multiple visualizations, the only real computational overhead that is added comes when generating visual explanations after inference by calculating which training patches lie within each concept ball. However, this operation only needs to be performed once and is still relatively fast. During training and inference, the addition of our ProtoConcepts module amounts to a single thresholding layer after the distance calculation in latent space, which we have found to have negligible computational cost. In addition, our set representation for prototypical concepts allows us to skip the projection step of other prototype-based networks, which allows for faster training of a ProtoPConcepts network. We thank the reviewer again for this suggestion and will add further discussion of computational efficiency to the supplement.\n\n>The 1vsmany patch-wise comparison has been studies and they even conducted human studies on interpretability. How this work differs from them [4].\n\n**A**:There are some major differences between our work and [4]. [4] do not learn any concepts or prototypes. Instead, their study uses kNN-based modeling and has to compare a given test image with all training images to find the top 20 training images that are most similar to the test image. This is computationally expensive. On the other hand, our work does not use kNN-based modeling and instead learns a fixed set of relevant prototypes and concepts from the training set. Unlike [4], our model works by comparing a given test image with the learned prototypes/concepts, thereby avoiding having to compare each test image with the entire training set. We also benefit from learning concepts that are meaningful in the domain. Although the study by [4] also considers one vs. multi-patches algorithm, the multiple visualizations are ranked by their similarity scores to the given test image and thus contribute unevenly to the decision process. However, our approach finds multiple visualizations for each learned concept which contribute **equally** to the decision process.\n\n[1] Kim et al., HIVE: evaluating the human interpretability of visual explanations, ECCV 2022 \n\n[2] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in Neural Information Processing Systems, 32, 2019 \n\n[3] Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine-grained image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14933–14943, 2021. \n\n[4] Nguyen, Giang, Mohammad Reza Taesiri, and Anh Nguyen. \"Visual correspondence-based explanations improve AI robustness and human-AI team accuracy.\" Neural Information Processing Systems (NeurIPS) (2022).\n"}, {"review_id": "a0x9YDGz9Y", "reviewer": "Reviewer_wBiV", "summary": "The paper proposes a modification of the prototype layer of the existing prototype-based networks. Existing methods rely on a single training image patch as a prototype. The main drawback of this is that it is difficult for the user to understand the meaning of the prototype from a single visualization. The paper proposes to learn prototypical concepts which are visualized using multiple training image patches. In this way, the semantic of each prototype is less ambiguous.", "questions": "- what is the main advantage of your solution with respect to learning standard prototypes (e.g., as done in ProtoPNets) and visualizing it on multiple most activated training images? In other words, the original work on ProtoPNets visualizes the prototype on the most activated training image; what if you just take more images?\n- How is k set in the experiments? Why does the value of k change depending on the prototype-based model?", "limitations": "The authors address the limitations. I would like to stress a point that is also discussed in the limitations. The user expertise influences the impact of the method on identifying the concepts: given the same set of prototype visualization, an expert user has a different interpretation of them with respect to a non-expert.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "_Clarity_: the paper is generally well-written and structured clearly. The figures showing image examples are very helpful, especially for readers that are not experts on prototypical networks.\n\n_Significance_: the paper addresses a relevant problem which is well known to practitioners using prototypical networks. The problem is filling the gap between what the prototypes represent and the human understanding of prototype semantics. Previous works tried to tackle it. \nThe proposed changes do not badly impact the overall performance of the prototypical networks; indeed, the accuracy is comparable to previous methods.\n\n_Quality and originality_: the two novel losses are technically sound. The idea of representing the prototypes as a ball in the latent space is interesting. The solution can be applied to several types of prototypical networks.\n\n_Reproducibility_: the code will be made available upon acceptance (it was unavailable to the reviewers). The supplementary material reports the training parameters.", "weaknesses": "As reported in the introduction, the proposed solution should allow the user to determine the semantic meaning of each prototype with less ambiguity. However, the experiments do not report one or more metrics that show this reduction of ambiguity with respect to the plain prototypical networks. The paper and the supplementary material reports some example of concrete cases but not an overall evaluation. The optimal solution would be to run an evaluation with real users, but I understand that it is expensive.\nThe paper [1] proposes activation precision as an interpretability metric (section 4.2) (the segmentation masks are provided with CUB-200).\n\nGiven that Cub200 dataset provides part location annotations for the images, another metric could measure the degree to which the multiple visualizations of the prototype refer to the same bird part. \n\n[1] Barnett 2021, \"IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammography\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review comments. We are happy to make some clarifications as follows:\n\n>However, the experiments do not report one or more metrics that show this reduction of ambiguity with respect to the plain prototypical networks.\n\n**A**: As per your suggestion, we evaluated the added interpretability of our ProtoPConcepts module using the **distinction** survey task in HIVE[1]. In this task, we asked participants to select the correct prediction out of two options based on provided visual explanations, and found that our ProtoPConcepts module allowed for a statistically significant increase in user interpretability. Please see the global rebuttal for exact study details and results. We thank for the suggestion of a human study and believe that our survey results objectively show the interpretability benefit from our method.\n\n>What is the main advantage of your solution with respect to learning standard prototypes (e.g., as done in ProtoPNets) and visualizing it on multiple most activated training images? In other words, the original work on ProtoPNets visualizes the prototype on the most activated training image; what if you just take more images?\n\n**A**:Unfortunately, the type of calculation you're thinking about doesn't work for models such as ProtoPNet, and is part of the motivation for this work. The reason why models such as ProtoPNet are interpretable-by-design is that the prototypes learned by the model correspond **directly** to a single training patch in latent space. When models such as ProtoPNet perform inference on a test image, they calculate distances to these exact training patches in latent space. Therefore, the patches visualized by ProtoPNet are not just the **closest** training patch to that prototype, they are **exactly** the single training patch corresponding to that prototypical vector in latent space. If we tried to create a way to highlight other nearby training patches, it would yield unfaithful explanations, as the distances calculated in the model's reasoning process doesn't directly correspond to other nearby parts. To overcome this problem, we instead represent prototypes as sets for both the reasoning and explanation process, which allows us to create multiple visualizations for each prototype in a way that is still faithful to the model's exact reasoning process.\n>How is k set in the experiments? Why does the value of k change depending on the prototype-based model?\n\n**A**:In the top-k cluster loss for our optimization, the exact value of k is treated as a hyperparameter and is selected by cross-validation. Because the prototype-based models we analyze in our experiments have widely varying latent space geometries due to different distance metrics (e.g. L2 distance vs. cosine similarity), we tune the exact value of k separately for these different models along with other parameters such as the initial concept ball radius. \n\n[1] Kim et al., HIVE: evaluating the human interpretability of visual explanations, ECCV 2022"}, {"review_id": "G1pFOI3k6N", "reviewer": "Reviewer_x5Kg", "summary": "This paper proposes a new interpretable prototype-based classifier, called ProtoConcepts. Unlike existing prototype-based classifiers that use one-to-one comparisons, ProtoConcepts learns prototypical concepts using multiple image patches. This approach aims to make it easier to identify the underlying concept being compared, allowing for richer and more interpretable visual explanations. Experiments show that this modified \"this looks like those\" reasoning process can be applied to various prototypical image classification networks without affecting accuracy on benchmark datasets.\n", "questions": "See weaknesses.\n\n", "limitations": "1. The novelty of the paper appears to primarily lie in the combination of two distinct lines of research within XAI - concept-level explanations and prototype-based classifiers.\n\n2. The absence of user study results and the unavailability of source code make it difficult to be convinced about the interpretability of the explanations generated by the proposed method.\n\nSee more details in the part of \"Weaknesses\".", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe authors make a valid point that utilizing a single training image patch as a prototype can be insufficient for users to comprehend the concept the prototype represents. For example, when presented with a \"blue circle\" prototype, it is unclear whether the concept is related to the color \"blue\" or the shape \"circle.\"\n\n2.\tThe paper (Table 1) demonstrates that ProtoConcepts can be integrated with various prototype-based classifiers, such as ProtoPNet, TesNet, and ProtoPool, showcasing its adaptability across different methods.\n", "weaknesses": "1. It is definitely not a new idea in the XAI community to use multiple images/patches to visualize a concept. This is a common practice in the works of concept-level explanations, like [TCAV](http://proceedings.mlr.press/v80/kim18d/kim18d.pdf), [ACE]( https://arxiv.org/pdf/1902.03129.pdf), and so on. However, none of them are discussed in the paper;\n\n\n2. The main issue of the paper lies in the evaluation of interpretability. It is unclear whether these examples are cherry-picked. Also, given that the source code of the method is unavailable at this moment, it is unknown how good/bad the explanation results would be for many other cases.\n\nIf the examples are not cherry-picked, it would be better for the authors to clearly state this in the paper or use examples in a fixed order (e.g., always use the first example of each class).\n\nIt is hard to be convinced that the interpretability of ProtoConcepts is good by showing several separate examples. I would suggest that the authors conduct a carefully designed human user study. The user study results should better convince users that the method brings about a better understanding of the model's behavior. User studies are often conducted in the work of concept-level explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review comments. We are happy to make some clarifications as follows:\n>It is definitely not a new idea in the XAI community to use multiple images/patches to visualize a concept. This is a common practice in the works of concept-level explanations, like TCAV, ACE, and so on. However, none of them are discussed in the paper\n\n**A**:Thank you for bringing these relevant works to our attention; we agree that concept-level explanations are not a new idea in the field of XAI, and will add discussion of these works to our final manuscript. However, TCAV and ACE are from an entirely different genre than our approach. Those methods are **posthoc** and **supervised**. In other words, TCAV needs to be fed images of a **known, predefined** concept and used those to analyze an existing model. Unlike our method, **posthoc** methods are not involved in the model computation process. Any results produced **posthoc** are not always faithful to the classification decision. Instead our method is **inherently interpretable**, and the algorithm **discovers the concepts** by itself. It is similar to ProtoPNet and its variants, but none of those learn concepts; they use comparisons between **pairs** of images. The concepts that ProtoConcepts discovers could be totally new, whereas TCAV/ACE/etc. require **known** concepts.\n>I would suggest that the authors conduct a carefully designed human user study.\n\n**A**:As per your suggestion, we evaluated the added interpretability of our ProtoPConcepts module using the **distinction** survey task in HIVE[1]. In this task, we asked participants to select the correct prediction out of two options based on provided visual explanations, and found that our ProtoPConcepts module allowed for a statistically significant increase in user interpretability. Please see the global rebuttal for exact study details and results. We thank for the suggestion of a human study and believe that our survey results objectively show the interpretability benefit from our method.\n\n>If the examples are not cherry-picked, it would be better for the authors to clearly state this in the paper or use examples in a fixed order (e.g., always use the first example of each class).\n\n**A**:Thank you for this comment. We believe the results are representative. Luckily, you can judge yourself whether you believe the results are cherry picked because we included many additional examples of the reasoning process in the supplement. In addition, our user study shows that our method results in an objective increase in user interpretability of our visual explanations. \n\n[1] Kim et al., HIVE: evaluating the human interpretability of visual explanations, ECCV 2022\n"}]

Author rebuttal:
**User Study** We thank all the reviewers for their comments. To show the reduction of ambiguity (and resulting improvement in user interpretability), we created a **distinction** user study similar to HIVE[1] to compare our ProtoPConcepts method with ProtoPNet. We randomly picked ten samples from the test set and calculated the top two predicted classes (i.e., the classes with the highest predicted probabilities according to the model) for each test sample. We then provided visual explanations from the most activated prototypes for these classes by ProtoPNet and ProtoPNet-Concepts. A test-taker was then asked to choose which class the model is actually predicting, looking only at the visual explanations without the class probabilities. Examples of our user study are shown in the global rebuttal pdf Figure 2. We released our user study on Amazon Mechanical Turk and collected 50 responses from test takers with a 98% survey approval rate  to ensure the quality of responses, and removed 1 response from both surveys after screening for nonsensical free response answers. We first ran a two-sided t-test on self-rated ML experience for the test takers from the ProtoPNet and ProtoPNet-Concept. The p-value is 1, and we are assured that there is no statistically significant difference in machine learning experience between the two groups on average. We further performed a one-sided, two-sample Welch t-test with the alternative hypothesis that the ProtoConcept user study results in higher accuracy than the ProtoPNet on average. The p-value is 0.003, which means that **ProtoConcept exhibited a statistically significant improvement in model interpretability over ProtoPNet**. Moreover, users given visual explanations from ProtoPNet could not statistically beat a random guessing accuracy of 50 percent (p=0.289), which is consistent with the findings in the HIVE paper[1]. With our model, users' mean accuracy was able to beat random guessing by a statistically significant margin (p=$2.85*e^{-5}$). Our survey results show not only that **our model provides a notable improvement in user interpretability**, but is able to **improve non-expert user performance** in a difficult fine-grained classification task whereas the previous ProtoPNet model cannot. The detailed test statistics can be found in global rebuttal pdf Table 1. 

[1] Kim et al., HIVE: evaluating the human interpretability of visual explanations, ECCV 2022

Meta-review: Summary
This paper improves upon prototype-based learning for image classification. While most prior work use a one-to-one mapping, this paper proposes a way to use multiple training image patches per prototype. This makes the  visualization of the prototypes more interpretable. The method is evaluated on image classification using the CUB and the Cars dataset. 

Reviews & Justification
The paper received positive reviews overall and the reviewers were satisfied with the author responses. The reviewers find this paper provides a significant improvement over prior variants of the ProtoPNets.
Decision: Accept (poster)