Title: This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations
Venue: NeurIPS 2023
Accepted: yes
Keywords: deep learning

interpretability

prototype-based neural network

case-based reasoning
Abstract: We present ProtoConcepts, a method for interpretable image classification combining deep learning and case-based reasoning using prototypical parts. Existing work in prototype-based image classification uses a "this looks like that'' reasoning process, which dissects a test image by finding prototypical parts and combining evidence from these prototypes to make a final classification. However, all of the existing prototypical part-based image classifiers provide only one-to-one comparisons, where a single training image patch serves as a prototype to compare with a part of our test image. With these single-image comparisons, it can often be difficult to identify the underlying concept being compared (e.g., "is it comparing the color or the shape?''). Our proposed method modifies the architecture of prototype-based networks to instead learn prototypical concepts which are visualized using multiple image patches. Having multiple visualizations of the same prototype allows us to more easily identify the concept captured by that prototype (e.g., "the test image and the related training patches are all the same shade of blue''), and allows our model to create richer, more interpretable visual explanations. Our experiments show that our ``this looks like those'' reasoning process can be applied as a modification to a wide range of existing prototypical image classification networks while achieving comparable accuracy on benchmark datasets.

This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations

Abstract

We present ProtoConcepts, a method for interpretable image classification combining deep learning and case-based reasoning using prototypical parts. Existing work in prototype-based image classification uses a 'this looks like that' reasoning process, which dissects a test image by finding prototypical parts and combining evidence from these prototypes to make a final classification. However, all of the existing prototypical part-based image classifiers provide only one-to-one comparisons, where a single training image patch serves as a prototype to compare with a part of our test image. With these single-image comparisons, it can often be difficult to identify the underlying concept being compared (e.g., 'is it comparing the color or the shape?'). Our proposed method modifies the architecture of prototype-based networks to instead learn prototypical concepts which are visualized using multiple image patches. Having multiple visualizations of the same prototype allows us to more easily identify the concept captured by that prototype (e.g., 'the test image and the related training patches are all the same shade of blue'), and allows our model to create richer, more interpretable visual explanations. Our experiments show that our 'this looks like those' reasoning process can be applied as a modification to a wide range of existing prototypical image classification networks while achieving comparable accuracy on benchmark datasets.

1 Introduction

As machine learning models are increasingly adopted in high-impact, high-stakes domains such as healthcare [2, 13], self-driving cars [24], facial recognition [23], etc., the design of humaninterpretable models has become essential for ensuring accountability and transparency of their decisions. In particular, a class of models called prototype networks have combined the power of deep learning with the explainability of case-based reasoning to make accurate, human-interpretable decisions on fine-grained image classification tasks [3, 6, 19, 25, 26, 36]. These models make decisions by dissecting an image into informative feature patches, then comparing them to prototypical parts learned during training. Evidence of similarity to each prototypical part is then combined to make a final classification.

When evaluating the interpretability of an image classification network, it is important to consider not only the model's capability to explain its reasoning process, but also the quality of its explanations. The image features analyzed by prototype networks can encode a variety of visual properties, such as color, shape, texture, contrast, or saturation. Because each prototype in the existing prototype network corresponds exactly to a single patch of a single training image, it can be difficult for human users to understand the exact semantic content underlying a prototype's visualization. Although there has been some work in quantifying the importance of each of these feature categories in prototypes [20], the exact semantic content in an image patch can still remain ambiguous (e.g. 'Texture is

ProtoPool:

Why is this bird classified as a Brown

Thrasher?

ProtoPool-Concepts:

Why is this bird classified as a Brown

Thrasher?

Looks like

Looks like

Comes from

found in..

-Slaty backed Gull

- Boat tailed

Grackle

…

Concept Features

found in..

-Oven bird

- Fox Sparrow

…

Looks like

Comes from

Looks like

important in this prototype, but what kind of texture in particular?'). The ambiguity underlying prototype visualizations has been shown to cause a gap between visual explanations from prototype networks and human understanding of visual similarity [15]. Additionally, a variety of prototype networks have been proposed in which prototypical parts from a single image can serve as evidence for many different classes, thus reducing the total number of prototypes needed for the network to make classifications [19, 26, 25]. The rationale behind this prototype sharing is that certain distinctive features (e.g. 'long beak, solid black head, spotted belly') can be found in multiple classes, and thus do not need to be represented as distinct class-specific copies. However, while reducing the number of prototypes can simplify visual explanations by reducing the amount of considered evidence, these methods can often produce seemingly nonsensical explanations when a training patch from an image is used spuriously as evidence for a different class.

Inspired by the difficulty of identifying the meaning behind single-image prototypes, we propose ProtoConcepts, a novel modification to prototype geometry enabling visualizations of prototypical concepts from multiple training images. This allows human users to better understand the conceptual content of a prototype by observing the features shared among its visualizations. An example of a visual explanation produced by our ProtoConcepts is shown in Figure 1 (right). Unlike previous prototype networks where a prototype is visualized using a single training image patch (Figure 1, left), our ProtoConcepts visualizes each prototype with multiple visualizations. Because our model offers many visual examples for each prototype, the semantic meaning of each prototype can be determined with less ambiguity - we can look at the commonality between the visualized training image patches to infer the semantic meaning of each prototype. Our method can be applied to a wide range of prototype-based networks, resulting in better human interpretability while achieving comparable classification accuracy. Our code is available at https://github.com/Henrymachiyu/This-looks-like-those_ProtoConcepts

2 Related Work

Attempting to understand deep neural networks, posthoc explanation techniques such as activation maximization [7, 21, 37, 38], image perturbation [8, 12], and saliency visualizations [30-32, 27] fail to explain a network's reasoning process, and their results can be risky and unreliable [18, 1]. On the other hand, prototype-based networks compare learned latent feature representations, called

prototypes, to the latent representations of an image to perform classification. These models are constrained to be intrinsically interpretable. The Prototypical Part Network (ProtoPNet) [3] uses class-specific prototypes, where a pre-determined number of prototypes are assigned to each class. Each prototype is trained to be similar to feature patches from images of its own class and dissimilar to patches from images of other classes. This gives such class-specific models high discriminative power suitable for fine-grained image classification. In the end, prototype similarity scores are aggregated as positive evidence for each class in a 'scoresheet.' The Transparent Embedding Space Network (TesNet) [36] improved on the class-specific ProtoPNet architecture by transforming the latent space geometry into a hypersphere, and representing prototypes as near-orthogonal basis vectors specific to a class subspace in the Grassmannian manifold.

Other prototype-based networks found that abandoning class-specific prototype constraints allowed for a significant reduction in the number of prototypes, making explanations simpler and allowing prototypes to represent similar visual concepts shared in images of different classes. In [25], a ProtoPNet is first trained with enforced class specificity, and then a novel data-based similarity metric is used to 'merge' highly similar prototypes across classes into a single prototype representing multiple classes. In Neural Prototype Trees (ProtoTree) [19], the 'scoresheet' of the ProtoPNet is replaced by a binary decision tree, in which the scored presence or absence of a prototype in each node determines the traversal path of the tree. Thus, each prototype in the tree represents either positive or negative evidence for the classes at the leaf nodes of the subtree for which the prototype is in the root node. Finally, in the ProtoPool network [26], the soft assignment of prototypes to classes is learned throughout training, and is eventually transformed into a hard assignment at test time.

Our work also relates to works that try to understand the semantic meaning behind a learned prototype of a prototype-based network. For example, in [20], Nauta et al. developed a method to quantify the importance of various visual characteristics relevant to each prototype. Our approach is complementary to, yet differs from theirs in that we illuminate the concept behind each prototype by presenting multiple visualizations of the same prototype, thereby allowing one to better infer the semantic meaning behind each prototype.

Finally, our work is related to posthoc methods in concept visualization such as TCAV [14] and ACE [9]. Posthoc methods do not show any part of the model's actual reasoning process. Worse, their visual explanations may not be faithful to the model's reasoning process. Sometimes, multiple concepts have the same concept vector [as discussed in 5], and concepts may not be clustered in the latent space (since they are not trained to be that way), leading to meaningless concept vectors. Instead, our method is inherently interpretable, and derives its visual concept-based explanations directly from the decision-making process.

3 Methods

We begin with a brief general review of traditional prototype-based models, then propose our method, which represents prototypes as interpretable concepts with multiple visual explanations. We show how our method can be applied as a modification to existing prototype-based networks, and introduce a novel training algorithm to learn meaningful prototypical concepts. Details for the implementation of specific prototype-based classification models are in the Experiments section.

3.1 Prototype Architectures

Existing prototype-based networks [3, 19, 25, 26, 36] can be understood architecturally as the combination of three components. First is a convolutional neural network (CNN) f parameterized by weights w conv, which acts as a latent feature extractor on input images x . Next the prototype layer p considers the convolutional output f ( x ) as H × W 'patch' vectors z i ∈ R D , each of which corresponds to features from a patch of the original image space. Each patch is compared to m learned prototype vectors P = { p j } j m =1 , where each prototype p j ∈ R D lies in the same space as the image's latent feature patches. The prototype layer p calculates a similarity score g p j ◦ f ( x ) for each prototype p j which is monotonically decreasing with respect to the distance between p j and the closest image patch ˜ z ∈ f ( x ) in the model's feature space. Finally, the prototype layer g p is followed by a prototype class assignment mechanism h which assigns evidence logits to each class based on the calculated prototype similarity values g p j ◦ f ( x ) and class assignment weights w h . The evidence

CNN

…

…

= 3.67

=3.52

…

…

Ranked

Predictions

By logits

Top 1

Prediction:

Golden winged

…

=3.36

…

p

1

p

2

p

m

Logit value:37.9

Logit value:27.0

Convolution Layer

ProtoConcepts

Layer

9p

Evidence Layer

logits are then normalized by a softmax to yield predicted probabilities for a given image belonging to each class.

In the final model, each prototype vector p j is restricted so that it is equal to a latent patch ˜ z ∈ f ( x i ) from some training image x i . Thus, the model's reasoning process can be interpreted visually as an evidence scoresheet of comparisons between test image patches and salient features underlying previously-seen training image patches. We propose to modify the geometry of prototypes p j by instead representing them as a ball in the latent space encompassing multiple training image patches. Mathematically, each prototype is a ball p j = B ( c j , r j ) around a central vector c j ∈ R D with radius r j ∈ R + . This ball representation has the property that the similarity scores of any latent patch vector z i ∈ R D and any vector d j ∈ p j in the prototypical ball are constrained to be the same. Hence, with this design, as long as a prototypical ball p j contains a number of latent training patches, we can visualize the prototype p j by visualizing any of those latent training patches it contains. This set representation allows our prototypes to represent concepts found in all latent training patches contained within a prototypical ball, thereby providing richer semantic explanations than that of visualizing just a single latent training patch. Implementing our concept geometry on a prototypebased model entails minor changes to the architecture and loss; model-specific implementation details are described in Sec. 4.1.1.

3.2 Prototypical Concept Representations

The prototype layer g p j of previous prototype-based classification models uses one of two activation functions for calculating similarity scores between prototypes and image patches: a log-based activation on inverse ℓ 2 distance, and a cosine similarity score. In this section, we formalize our ball representation of prototypical concepts with respect to the two corresponding latent spaces.

Log-based Activation: Models such as ProtoPNet [3] and ProtoPool [26] use a log-based function g log to compute a similarity score, which is monotonically decreasing with respect to the ℓ 2 distance. In particular, for the latent representation z = f ( x ) of an input image x , this function computes g log p j ( z ) = max ˜ z ∈ patches ( z ) log(( ∥ ˜ z -p j ∥ 2 2 +1) / ( ∥ ˜ z -p j ∥ 2 2 + ϵ )) with a small constant ϵ to avoid numerical overflow. For prototypical concepts, we propose an analogous similarity function

corresponding to the distance to center c j of prototypical ball p j = B ( c j , r j ) , bounded below by r j :

Cosine Similarity: Models such as TesNet [36] and Deformable ProtoPNet [6] have found that representing prototypes as unit vectors on a hypersphere can improve the accuracy of prototype-based models. In these models, the prototype layer compares prototypes to latent image patches via the cosine similarity function g cos p j ( z ) = max ˜ z ∈ patches ( z ) ˜ z T p j / ( ∥ ˜ z ∥ 2 ∥ p j ∥ 2 ) . This cosine similarity can be interpreted as a cosine activation function on the angular distance between normalized vectors on the hypersphere. For models with a hypersphere-based latent space, we replace the cosine similarity function g cos p j by calculating the cosine similarity to the ball B ( c j , r j ) on the hypersphere :

Both Eqs. (1) and (2) ensure that for any patch vector z i ∈ R D , if z i lies outside the ball p j , its distance to p j is the distance to its center c j , and if z i lies inside p j , its distance to p j is kept the same as the radius of the ball. During training, a pass-through estimator of each activation function is used to avoid a zero gradient when training patches lie within the prototypical ball.

3.3 Training Algorithm

The training of prototype-based networks is typically divided into three stages: (1) stochastic gradient descent (SGD) of layers before the last layer; (2) projection of prototypes; (3) optimization of the last layer h . Because of the ball representation of prototypical concepts, the second stage can be skipped as prototypical balls are visualized by the training patches they contain at the end of the first step.

SGD of layers before last layer: The first training stage prototype-based networks aims to learn a meaningful latent space that clusters important training patches near semantically similar prototypes. To do this, a joint optimization problem is solved to optimize the network parameters using SGD while keeping the last layer weight matrix fixed. To learn semantically meaningful prototypical concepts, we introduce two novel loss terms, L Clstk and L Rad:

̸

Here d is used as the distance function corresponding to the respective latent space geometry of the ProtoConcepts module. The minimization of L Clstk encourages k prototypical concepts of the correct class to be close to at least one latent patch of each training image. The specific value of k is treated as a hyperparameter and is chosen by cross-validation. As opposed to the traditional prototype cluster loss [3], which only penalizes the distance of the closest prototype to a latent training patch, we find that our topk formulation is essential for ensuring that prototypical concept balls contain multiple training patches. The minimization of L Rad encourages prototypical concept balls to be compact. This allows learned prototypical concepts to encapsulate more specific semantic features important for classification.

Prototypical Concept Visualisation: In previous prototype-based networks, the trained prototype vector p j is projected (pushed) to the nearest training patch to produce a single patch visualization. However, since we use a set representation for prototypical concepts, it is no longer reasonable to project the prototypical ball or its central vector to the training patches. Instead, we visualize each prototypical concept by the training patches that are contained within each ball at the end of training. Thus, we can produce multiple visualizations from each prototypical concept and skip the projection step altogether.

Pruning and Fine-tuning: Because the number of prototypical concepts is fixed at the beginning of training, it is possible for a trained ProtoConcepts model to have a small number of prototypical

concepts that do not contain any latent training image patches. In other words, these prototypical concepts do not capture any visualizable latent features, and should not be considered when making predictions. To prune the non-visualized prototypical concepts, we design a 0/1 mask on the prototype class assignment mechanism so that the given non-visualized prototypical concepts do not provide evidence for any image classes. After pruning these prototypical concepts, we finally perform a sparse convex optimization on the last layer weight matrix w h as in previous prototype-based methods [3].

4 Experiments

4.1 Case Study 1: Bird Species Identification

To demonstrate and examine the effectiveness of our method, we implement the ProtoConcepts module on the ProtoPNet, ProtoPool, and TesNet networks for the Caltech-UCSD Birds-200-2011 (CUB 200-2011) dataset [35]. The dataset contains 5,994/5,794 images for training and testing across 200 different bird species. We perform offline data augmentation, and crop training and testing images using bounding boxes provided with the dataset.

4.1.1 Implementation

In this section, we describe the implementation changes made to each prototype-based model for our experiments to incorporate prototypical concepts. A full list of exact parameter settings and training schedule for each model can be found in the Appendix.

ProtoPNet: The ProtoPNet model [3] consists of a CNN backbone, a log-based prototype layer g log , and a fully connected layer h [3]. Each of 200 classes is assigned 10 class-specific prototypes, which are constrained to provide evidence only for their corresponding class through the last layer weighting. We train a ProtoPNet-Concepts model by substituting the prototype layer as in Eq. (1), and then training with additional losses L Rad and L Clstk with k = 10 , radius initialization 7 . 5 , and loss weights 0 . 01 and 0 . 8 , respectively. The training schedule and other hyperparameters are unchanged from the original paper.

TesNet: The TesNet model [36] uses a cosine similarity function in its prototype layer to represent the latent space as a hypersphere and incorporates several hypersphere-based loss functions during training. After modifying the activation function as in Eq. (2), we train TesNet-Concepts with a radius initialization of 8 . 05 , and a topk cluster loss L Clstk with k = 3 . We set the weight for the radius loss as 3 × 10 -5 and the weight for topk cluster loss as 0 . 8 with a learning rate of 1 × 10 -4 during the warmup stage, and reweight the loss functions described in its original paper with a new training schedule.

ProtoPool: The ProtoPool architecture [26] employs a log-based activation on the prototype layer, and adds a prototype class assignment mechanism that allows sharing prototypes among multiple classes, thereby reducing the number of prototypes. After modifying the prototype layer as in Eq. (1), we train ProtoPool-Concepts using a radius initialization of 4 . 5 and topk cluster loss L Clstk with k = 10 . We set a radius loss weight 3 × 10 -3 , a topk cluster weight 0 . 8 , and learning rate for the radius as 0 . 5 × 10 -3 . The training schedule and other hyperparameters are left unchanged from the original paper.

For baseline comparisons, we retrain CNN backbones and previous prototype-based models using publicly available code. As shown in Table 1, for the same CNN architecture, our ProtoConcepts network achieves similar test accuracy to corresponding baseline models.

Radius Study: We performed an ablation on ProtoPool-Concepts with a different set of radius initialization and topk cluster loss with k = 10 under the same training settings described above. Intuitively, a larger radius would encourage more prototypical concepts to capture information and an infinitesimal radius would be equivalent to a single visualization. As shown in Table 2, the number of visualizable prototypical concepts increases with the radius. In addition, radius also influences the accuracy of our ProtoConcepts model after pruning and fine-tuning. If the radius is too small, the number of visualizable prototypical concepts also tends to be small - this means that pruning will remove too many prototypes, and consequently adversely impact performance. On the other hand, if

the radius is too large, each prototypical ball would capture too many latent representations whose semantic meanings may be too diverse, thereby making the prototypes less meaningful. Hence, we need to tune the radius to be within a reasonable range.

4.1.2 Interpretability

The reasoning process of a ProtoConcepts network mirrors that of a traditional prototype network, but provides multiple visualizations of the prototypes used in its explanations. Figure 3 shows how our model classifies a testing image of a Baltimore Oriole into the correct class. It is worth noting that because of the shareable prototypes mechanism inherited from ProtoPool, the visualizations of a learned prototypical concept now come from different classes. The most representative features that an ornithologist would use to classify an adult male Baltimore Oriole are the bright orange and black plumage and the slender beak [28]. As shown in Figure 3, our model is able to compare the test image patch containing orange and black plumage with the learned prototypical concept that captures similar features from its own class. Additionally, our model compares the patch containing the wings to the prototypical concept shared among Baltimore Orioles and other similar birds such as Orchard Orioles. Lastly, our model compares the part of image that contains a slender beak along with its black head to the prototypical concept that captures the slender beak and black head of birds such as Orchard Orioles and Eastern Towhees.

Figure 1 shows an example of how a test image of a Brown Thrasher is correctly classified by ProtoPool and ProtoPool-Concepts. Because of the prototype sharing mechanism, it is frequently the case that training image patches from birds of one class are used as evidence of a test image belonging to another class. As a result, the similarities presented by ProtoPool's visual explanations can be unintuitive. As shown in the plot, it is unclear how ProtoPool found the head of a Brown Thrasher to be similar to the blue head of a Barn Swallow. Moreover, it is hard to determine whether the model is comparing the background or the belly of the Brown Thrasher, as shown at the bottom. However, comparisons made by our method are much clearer thanks to multi-visualizations. Although both models attempt to compare the lower part of the bird to the learned prototypes, it is clear that our ProtoPool-Concepts is looking at the pattern on the belly, which is an identifying feature of the Brown Thrasher [28].

Evidence for this bird being a Baltimore Oriole:

Original Image (box

showing part that

looks like prototype)

Prototypical Concept

visualizations

Training images where the

visualizations come from

Similarity

Score

Class

connection

Points

contributed

3.72

x1.052

=3.91

3.59

x1.051

= 3.77

3.51

x1.040

=3.65

Total Points to

33.24

ProtoPNet:

Why is this bird classified as a Female

Ruby Throated Hummingbird?

ProtoPNet-Concepts:

Why is this bird classified as a Female Ruby

Looks like

Looks like

Comes from

Looks like

Looks like

Comes from

ProtoPNet:

Why is this bird classified as a Male

Ruby Throated Hummingbird?

ProtoPNet-Concepts:

Why is this bird classified as a Male Ruby

Figure 4 shows examples of how the two models correctly classify a female and a male rubythroated hummingbird. In the top row, both of the models found the neck of a female ruby-throated hummingbird from the given test image similar to the white feathers around neck from the training examples. On the other hand, although having a red throat differentiates the male ruby-throated hummingbird from the female [28], the significance of a red-throat to correctly classify a male ruby-throated hummingbird is unclear. As shown in the bottom left of Figure 4, the learned prototype from ProtoPNet is able to compare the prototype that contains a red throat to that of the test image. However, the visualizations from ProtoPNet-Concepts demonstrate that a red throat may not be all that the prototypical concept is representing. It appears to represent broader information about the bird's gray head and the transition to its body, where the additional red or gray color is located on the throat, and the white on the breast.

4.2 Case Study 2: Car Model Identification

In this case study, we implement our method for the Stanford Cars dataset [17] of 196 car types, using similar procedures and training algorithms as in the CUB-2011-200 experiment. We trained a ProtoPool-Concepts model with radius initialization 4 . 5 , keeping other parameter settings the same as described in Sec. 4.1.1. We compare our method to self-reported accuracies of other shared prototype models ProtoPool [26] and ProtoPShare [25]. Our model achieves similar test accuracy to other shared prototype models, with the advantage of multiple visualizations for prototypical concepts. More details, including visual explanations, can be found in the Appendix.

5 User Study

To show the reduction of ambiguity and resulting improvement in user interpretability, we created a distinction user study similar to HIVE [15] to compare our ProtoConcepts method with ProtoPNet; results are shown in Table 4. We randomly picked ten samples from the test set and calculated the top two predicted classes (i.e., the classes with the highest predicted probabilities according to the model) for each test sample. We then provided visual explanations from the most activated prototypes for these classes by ProtoPNet and ProtoPNet-Concepts. A test-taker was then asked to choose which class the model is actually predicting, looking only at the visual explanations without the class probabilities. Examples of our user study are shown in Figure 7. We released our user study on Amazon Mechanical Turk and collected 50 responses from test takers with a 98 % survey approval rate to ensure the quality of responses, and removed 1 response from both surveys after screening for nonsensical free response answers. We first ran a two-sided t -test on self-rated ML experience for the test takers from the ProtoPNet and ProtoPNet-Concept. The p -value is 1, and we are assured that there is no statistically significant difference in machine learning experience between the two groups on average. From the results of our survey, we are able to conclude that users with visual explanations from ProtoConcepts outperform those with visual explanations from ProtoPNet to a statistically significant degree ( p = 0 . 003) . Moreover, users given visual explanations from ProtoPNet were unable to outperform 50% random guessing to a statistically significant degree ( p = 0 . 289) , whereas users given visual explanations from our model did ( p = 2 . 85 × 10 -5 ) . Our survey results show not only that our model provides a notable improvement in user interpretability, but is able to improve non-expert user performance in a difficult fine-grained classification task whereas the previous ProtoPNet model cannot.

6 Limitations

The interpretability of prototype-based models comes from visual explanations of classification decisions. Hence, the exact semantics (e.g., 'this bird has a long beak' or 'this bird has a spotted belly') underlying these visual explanations depend on the user's visual system to determine. We present a method for clarifying the user's semantic inference by offering multiple visualizations whose

Simulatethemodel

Simulatethemodel

Given a bird photo, the ProtoPNet model predicts the species based on prototypes it has learned from previously seen photos. Specifically for

Given a bird photo, the ProtoPNet model predicts the species based on prototypes it has learned from previously seen photos. Specifically for

each prototype, the model identifies a region in the photo that looks the most similar to the prototype and rates their simllarity.

each prototype, the model identifies a region in the photo that looks the most similar to the prototype and rates thelr similarity.

For a given photo, we show explanations of how the model reasons for 2 bird specles. For each bird specles, rate how similar each

For a given photo, we show explanations of how the model reasons for 2 bird specles. For each bird specles, rate how similar each

prototype is to the photo region. Note that the (region, prototype) pairs are presented in random order. At the end, choose the bird

prototype is to the photo region. Note that the (region, prototype) pairs are presented in random order. At the end, choose the bird

species you think is correct.

species you think iscorrect.

Random guessing will get you 50% accuracy. You will receive a reward based on your performance beyond this 50% random chance.

Random guessing will get you 50% accuracy. You will receive a reward based on your performance beyond this 50% random chance.

Photo

Task: Rate the simlarity of each prototype-region pair on a scale of 1-4.

2: Someehat Not tinlar

f: Not Smillar

2: Someehat Similar

: Sinlar

Photo

2: Bomehu Not tonltr

Task: Rate the simlarity of each prototype-region pair on a scale of 14.

1: Not Slmder

3: Bomewhat Sinilar

4; (lniler

the specified species.

prototype’s photos from

Prototypes and their

source photos are from

That is, Species 1

sujetuoo (juo uogeuejdxe

the prototypes and the

Species 1.

Clk sn'Speois 1° and "tpeoes 2'10 r

Species 1

A-A

O1O2O3O4

O1O2O3O4

O1O2O3O4

O1O2O3O4

O1O2O3O4

O1O2O3O4

specified

Prototypes

and their

source

photes are

from the

That is.

Species 1

explanation

only contains

the prototypes

and the

prototype/s

species.

lpeies 1

O1O2O3O4

O1O2O3O4

O1O2O3O4

pecis 1

O1O2O3O4

O1O2O3O4

O1O2O3O4

phoftos from

Species 1.

Q. Choose the bird species you think is correct, then click *Next Photo.*

Q. Choose the bird species you think is correct, then click “Next Photo."

OSpecies 1 O Species 2

O Species 1 O Species 2

commonalities can disambiguate our model's exact reasoning. While we find that our concept-based representation is helpful for this process, it is still unable to provide explicit semantics to explain its classifications. With recent progress in large vision-language hybrid models [22, 16, 4, 34], a technique offering explicit semantics for visual explanations could be explored in future work. It is important to note that in some visual domains, concepts may not have natural textual descriptions (e.g., in mammography or other radiology applications where it might look at a patch of a particular type of breast tissue that is difficult to describe and has no associated terminology).

7 Broader Impact

Interpretability is an essential ingredient to trustworthy AI systems [33]. Our work presents a major improvement in interpretability to prototype-based networks, which are one of the leading techniques for interpretable neural networks in computer vision. Thus, our technique can be used for important computer vision applications to discover new knowledge and create better human-AI interfaces for difficult, high-impact applications.

8 Conclusion

In this work, we present an interpretable method for image classification which incorporates prototypical concepts with multiple visualizations to explain its predictions ( this looks like those ). Unlike previous works in prototype-based classification which offer only single prototype visualizations, our method allows the user to use commonalities between visualizations of prototypical concepts to better infer the semantic meaning of prototypical concepts. We compare the visual explanations offered by our and previous prototype-based methods and show that our model can achieve comparable accuracy to previous methods.

References

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in Neural Information Processing Systems , 31, 2018.

Alina Jade Barnett, Fides Regina Schwartz, Chaofan Tao, Chaofan Chen, Yinhao Ren, Joseph Y. Lo, and Cynthia Rudin. IAIA-BL: A case-based interpretable deep learning model for classification of mass lesions in digital mammography. Nature Machine Intelligence , 3:1061-1070, 2021.

Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in Neural Information Processing Systems , 32, 2019.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings , pages 104-120. Springer, 2020.

Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence , 2(12):772-782, 2020. doi: 10.1038/s42256-020-00265-z.

Jon Donnelly, Alina Jade Barnett, and Chaofan Chen. Deformable ProtoPNet: An interpretable image classifier using deformable prototypes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10265-10275, 2022.

Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal , 1341(3):1, 2009.

Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2950-2958, 2019.

Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems , 32, 2019.

Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2015.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4700-4708, 2017.

Maksims Ivanovs, Roberts Kadikis, and Kaspars Ozols. Perturbation-based methods for explaining deep neural networks: A survey. Pattern Recognition Letters , 150:228-234, 2021.

Mohd Javaid, Abid Haleem, Ravi Pratap Singh, Rajiv Suman, and Shanay Rab. Significance of machine learning in healthcare: Features, pillars and applications. International Journal of Intelligent Networks , 3: 58-73, 2022.

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCA V). In International Conference on Machine Learning , pages 2668-2677. PMLR, 2018.

Sunnie SY Kim, Nicole Meister, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky. HIVE: Evaluating the human interpretability of visual explanations. In European Conference on Computer Vision , pages 280-298. Springer, 2022.

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning , pages 5583-5594. PMLR, 2021.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) , Sydney, Australia, 2013.

Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki. The dangers of post-hoc interpretability: Unjustified counterfactual explanations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages 2801-2807, August 2019.

Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine-grained image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14933-14943, 2021.

Meike Nauta, Annemarie Jutte, Jesper Provoost, and Christin Seifert. This looks like that, because... explaining prototypes for interpretable image recognition. In Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2021, Virtual Event, September 13-17, 2021, Proceedings, Part I , pages 441-456. Springer, 2022.

Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in Neural Information Processing Systems , 29, 2016.

OpenAI. GPT-4 Technical Report. arXiv e-prints , art. arXiv:2303.08774, March 2023. doi: 10.48550/ arXiv.2303.08774.

K Raju, B Chinna Rao, K Saikumar, and Nalajala Lakshman Pratap. An optimal hybrid solution to local and global facial recognition through machine learning. A Fusion of Artificial Intelligence and Internet of Things for Emerging Cyber Systems , pages 203-226, 2022.

Sebastian Ramos, Stefan Gehrig, Peter Pinggera, Uwe Franke, and Carsten Rother. Detecting unexpected obstacles for self-driving cars: Fusing deep learning and geometric modeling. In 2017 IEEE Intelligent Vehicles Symposium (IV) , pages 1025-1032. IEEE, 2017.

Dawid Rymarczyk, Łukasz Struski, Jacek Tabor, and Bartosz Zieli´ nski. Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1420-1430, 2021.

Dawid Rymarczyk, Łukasz Struski, Michał Górszczak, Koryna Lewandowska, Jacek Tabor, and Bartosz Zieli´ nski. Interpretable image classification with differentiable prototypes assignment. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII , pages 351-368. Springer, 2022.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision , pages 618-626, 2017.

D. Sibley. The Sibley Guide to Birds . Audubon Society Nature Guides. Knopf, 2000. ISBN 9780679451228. URL https://books.google.com/books?id=L-4gzAK1NBwC .

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Proceedings of International Conference on Learning Representations . 2013.

Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 , 2017.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning , pages 3319-3328. PMLR, 2017.

Elham Tabassi. Artificial Intelligence Risk Management Framework (AI RMF 1.0). Technical report, National Institute of Standards and Technology, 2023. https://doi.org/10.6028/NIST.AI.100-1.

Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 5100-5111, 2019.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltechucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.

Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing. Interpretable image recognition by constructing transparent embedding space. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 895-904, 2021.

Naoya Yoshimura, Takuya Maekawa, and Takahiro Hara. Toward understanding acceleration-based activity recognition neural networks with activation maximization. In 2021 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2021.

Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In Deep Learning Workshop at the 32nd International Conference on Machine Learning , 2015.

A Ablation Study on Topk Cluster Loss

To further demonstrate how topk cluster loss helps the model performance, we performed an ablation study on ProtoPool-Concepts. We trained ProtoPool-Concepts by adding L Clstk losses with the choices of k = 1 , 3 , 5 , 10 into the model with radius initialization 4 . 5 using ResNet50 (iNat) backbone. As shown in Table 5, although the classification performance of the model before pruning does not vary much when L Clstk is introduced, the number of visualizable prototypes sharply increases, and the performance after pruning is significantly improved. While, as discussed in the main paper, the L Clstk term ensures the quality of the latent space, encouraging prototypical concepts which enrich the interpretability of the model.

B Training Parameters

In this section we present the exact training hyperparameters, objectives, and schedules used in our experiments for applying the ProtoConcepts module to the ProtoPNet [3], ProtoPool [26], and TesNet [36] prototype networks.

B.1 ProtoPNet

In addition to the original losses ProtoPNet has, we trained ProtoPNet-Concepts with L Clstk, k=10 and L Rad. We set the corresponding weight for losses as demonstrated in Table 6. Like ProtoPNet, we have 5 epochs of training as warmup where we trained the add-on layers, prototype layers, and the radius width, as shown in Table 7. Similar to the original ProtoPNet training schedule, We start training the convolution layer and last layer at the joint optimization stage for 10 epochs. Overall, 15 epochs are trained before pruning and fine-tuning. Lastly, after pruning the prototypical balls that have no visualizations, we fine-tune the last layer for 20 epochs.

B.2 ProtoPool

Wetrained ProtoPool-Concepts, in addition to its original loss terms, with L Clstk, k=10 and L Rad. The corresponding weights for the losses are shown in Table 8. The training schedule is similar to that of the original ProtoPool,

shown in Table 9. We trained radius width only during the warm-up stage. We set the first 10 epochs as the warm-up stage, and the following 20 epochs as the joint optimization stage. Overall, the model is trained for 30 epochs before pruning and fine-tuning. We further trained additional 15 epochs to fine-tune the last layer weights after pruning the prototypical balls that do not contain any visualizations.

B.3 TesNet

We trained ProtoPool-Concepts, in addition to its original loss terms, with L Clstk, k=3 and L Rad. The corresponding weights for the losses are shown in Table 10. The training schedule is similar to that of the original TesNet, shown in Table 11. We trained radius width during the warm-up stage and joint optimization stage. We set the first 5 epochs as the warm-up stage, and the following 15 epochs as the joint optimization stage. Overall the model is trained for 20 epochs before pruning and fine-tuning. We further trained additional 15 epochs to fine-tune the last layer weights after pruning the prototypical balls that do not contain any visualizations.

C More Examples for ProtoPNet-Concepts Visualizations

In this section, we further demonstrate how multi-visualization solves the ambiguity of prototypes learned by ProtoPNet. The visualizations shown are trained with a VGG-19 [29] backbone for both ProtoPNet and

ProtoPNet:

Why is this bird classified as a Least

Tern?

ProtoPNet-Concepts:

Why is this bird classified as a Least

Tern?

Looks like

Looks like

Comes from

Why is this bird classified as a

Why is this bird classified as a

Looks like

Looks like

Comes from

ProtoPNet-Concepts. Figure 6 demonstrates how our method reduces the ambiguity inherent in the ProtoPNet architecture. The top row provides an example of how ProtoPNet and ProtoPNet-Concepts compare and classify a test image of a Least Tern. ProtoPNet highlights the bird's leg on the test image, which one can barely see in the model's learned prototypical features (e.g., is it comparing the wings? Or is it comparing the feet, which are too tiny to see in the image?) On the other hand, it is clear that ProtoPNet-Concepts is comparing the wings, even though the bounding boxes on the test image include the bird's feet. Similarly, the bottom row shows each model's reasoning for a correctly classified test image of a Chestnut-sided Warbler. It is not clear in this case whether ProtoPNet is comparing the background or a barely-captured yellow crown feature. On the other hand, the many examples of the yellow crown from the multi-visualization of ProtoPNet-Concepts clarify that the prototype represents this crucial feature of a Chestnut-sided Warbler [28].

More examples of unclear comparisons can be found in Figure 7. As shown in the plot, the ProtoPNet finds the green wing and spotted breast of the test image of a spotted catbird similar to a green wing with a black spot in the middle. On the other hand, the prototype learned by ProtoPNet-Concepts suggests that the prototypical concept focuses on the texture. In Figure 7, it is also unclear if ProtoPNet finds the feathers or the red feet of the Forster Tern to be similar to its learned prototype visualizations. However, we can clearly see from the ProtoPNet-Concepts multi-visualizations that the prototypical concept is actually the grey wing and white belly instead of the red feet.

Some types of ambiguities are specific to bird types. For instance, it is difficult to determine what the prototypes represent for birds that are black everywhere. For example, as shown in Figure 8, it is hard to see what the prototypes learned by ProtoPNet are capturing except for the dark feathers. Multiple visualizations learned by ProtoPNet-Concepts suggest that color and possibly the shape of the bird's belly are the first concept, whereas the red eye and shape of the head appear to be the second concept.

Ambiguity of the learned prototypes can arise from low quality test images. For example, as shown in Figure 9, visualizations from ProtoPNet have low brightness/contrast on the prototype birds. It is hard to see any distinguishable features captured by those prototypes. On the other hand, although similar visualizations are used by ProtoPNet-Concepts, other visualizations within the ball demonstrate the prototypical concepts (here, blue and white stripes apparently).

Lastly, having multiple visualizations can also help to draw inferences on misclassified cases. Figure 10 and Figure 11 show cases when both ProtoPNet and ProtoPNet-Concepts misclassify the test image. In Figure 10, it is unclear how ProtoPNet found the test image of a Cedar Waxwing to be similar to a Cardinal, except for the color of the fruit it has in its beak. On the other hand, multiple visualizations offer additional information such as the trianglular silhouette of the head, dark strip of feathers around the eye, and pointed features at the back of the head. Similarly, in Figure 11, ProtoPNet-Concepts is able to offer more specific information than ProtoPNet, such as the grey and white color contrast, black head and the shape of the body. Here, the algorithms are showing us why it is difficult to tell these birds apart.

ProtoPNet:

Why is this bird classified as a Spotted

Catbird?

ProtoPNet-Concepts:

Why is this bird classified as a Spotted

Catbird?

Looks like

Looks like

Comes from

Looks like

Looks like

Comes from

ProtoPNet:

Why is this bird classified as a orster

Tern?

ProtoPNet-Concepts:

Why is this bird classified as a Forster

Tern?

ProtoPNet:

Why is this bird classified as a Bronzed

ProtoPNet-Concepts:

Why is this bird classified as a Bronzed

Looks like

Looks like

Comes from

Looks like

Looks like

Comes from

ProtoPNet:

Why is this bird classified as a Florida

ProtoPNet-Concepts:

Why is this bird classified as a Florida

Looks like

Looks like

Comes from

Looks like

ProtoPNet:

Why is this bird falsely classified as a

Cardinal?

ProtoPNet-Concepts:

Why is this bird falsely classified as a

Cardinal?

Looks like

Looks like

Comes from

ProtoPNet:

Why is this bird falsely classified as a

Long tailed Jaeger?

ProtoPNet-Concepts:

Why is this bird falsely classified as a

Long tailed Jaeger?

Looks like

Looks like

Comes from

D More Examples for ProtoPool-Concepts Visualizations

In this section, we present more examples of the reasoning process for ProtoPool-Concepts. The visualizations are obtained from the model with ResNet 50(iNat) [10] backbone. Because prototypes are shared among different classes, the information extracted by ProtoPool can be unintuitive, especially when the prototype does not come from the same class as a given test image. This problem affects ProtoPool as well as ProtoPool-Concepts - while shared prototypes can, in some cases, make a concept clearer by borrowing good examples of that concept from other classes, in our experience it seems to make the concepts less clear. For example, in Figure 12, it is hard to see how the beak of a Cedar Waxwing is similar to the prototype visualization captured by ProtoPool. Moreover, similar to a problem we saw with ProtoPNet, whether the prototype is capturing the feet or the belly of the Cedar Waxwing from the top row is also unclear. On the other hand, ProtoPool-Concepts may be capturing a color contrast of the throat and breast that is common also to the Great Crested Flycatcher and Cedar Waxwing, etc., but since the contrast of the test image is not particularly good, the human might determine not to trust that

ProtoPool:

Why is this bird classified as a Cedar

Waxwing?

ProtoPool-Concepts:

Why is this bird classified as a  Cedar

Waxwing?

Looks like

Looks like

Comes from

Concept Features

found in..

-Great Crested

Flycatcherl

-Cedar Waxwing

…

Concept Features

found in..

-Cedar Waxwing

-Yellow headed

Blackbird

- Evening Grosbeak

…

Looks like

Comes from

Looks like

ProtoPool:

Why is this bird classified as a Yellow

ProtoPool-Concepts:

Why is this bird classified as a  Yellow

Looks like

Looks like

Comes from

Concept Features

found in..

-Cedar Waxwing

-Yellow headed

Blackbird

- Evening Grosbeak

…

Looks like

Comes from

Looks like

Concept Features

found in..

-Red winged

Blackbird

-Yellow headed

Blackbird

…

ProtoPool:

Why is this bird classified as a Red

ProtoPool-Concepts:

Why is this bird classified as a  Red

Looks like

Looks like

Comes from

Concept Features

found in..

-Red bellied

Woodpecker

-Northern Flicker

- Clay colored

Sparrow …

Looks like

Comes from

Looks like

…

Concept Features

found in..

-Great Crested

Flycatcherl

-Cedar Waxwing

-Pileated

Woodpecker

Evidence for this bird being a White breasted Kingfisher:

Original Image (box

showing part that

looks like prototype)

Similarity

Score

3.83

Class

connection

x1.03

Points

contributed

=3.93

Total Points to

35.0

Prototypical Concept

visualizations

Training images where the

visualizations come from

3.72

x1.02

=3.80

Evidence for this bird being a Cedar Waxwing:

Original Image (box

showing part that

looks like prototype)

Similarity

Score

3.82

3.78

Class

connection

x1.04

x1.02

Points

contributed

=3.97

=3.85

Total Points to

Cedar Waxwing:

35.5

Prototypical Concept

visualizations

Training images where the

visualizations come from

Concept Features

found in..

-Great Crested

Flycatcherl

-Cedar Waxwing

…

Concept Features

found in..

-Cedar Waxwing

-Yellow headed

Blackbird

- Evening Grosbeak

口

comparison. ProtoPool-Concepts also appears to capture the yellow and black color pattern on the bird's head. Similarly, in Figure 13, the comparison made at the bottom left for ProtoPool is hard to identify with a single visualization, while ProtoPool-Concepts offers more information such as the yellow head and black body, though the concept is not cohesive, since it includes a bird with red and white stripes. An analyst seeing this concept might determine that this concept may not be trustworthy, which is a useful piece of information. It is worth noting that the prototypical concept of the yellow-headed blackbird shown at the top of Figure 13 is shared with the Cedar Waxwing as shown in the bottom row of Figure 12. Figure 14 shows that ProtoPool-Concepts shares part of a prototype concept with the Cedar Waxwing; this concept appears to indicate a dark-and-light textural pattern that is similar to the wing of the Red Bellied Woodpecker. However, the comparison made by ProtoPool is ambiguous. Figure 15, Figure 16 further demonstrate how the evidence layer and prototype activation are involved in the reasoning process. As explained in the main paper, a weighted score would be computed for each of the prototypical concepts, and a total logit score would be calculated for the final prediction.

E More Examples for TesNet Visualizations for CUB dataset

In this section, we present some examples of the reasoning process of TesNet-Concepts and its comparison with TesNet. The visualizations were obtained from TesNet and TesNet-Concepts trained with a VGG19 backbone. Similar to ProtoPNet and ProtoPool, ambiguity arises from low quality images. In Figure 17, the prototype visualization of TesNet for a Florida Jay shown at the top has low contrast. Whether the prototype is capturing the belly or the feet is unclear. Although a similar prototype visualization is also captured by the prototypical ball for TesNet-Concepts as shown in the top right of the plot, the other visualizations reveal the prototypical concept as the white belly near the blue feathers. Similarly, the multi-visualization shown at the bottom suggests that the prototypical ball likely represents the bird's head and beak. Figure 18 shows the prototype logic for identifying a Cedar Waxwing by the color gradient on the bird's throat. Some of the image patches chosen by TesNet-Concepts look almost identical to the chosen patch of the test image; this is more convincing than the comparisons from TesNet without concepts.

F More Examples for ProtoPool-Concepts Visualizations and Model Performance for Car dataset

In this section, we present some of the visualizations for ProtoPool-Concepts on the Car dataset. The visualizations are from the models trained with a DenseNet121 [11] backbone. Similar to ProtoPool-Concepts for the CUB 200-2011 dataset, the shareable prototypes still make the comparison unintuitive. As shown in Figure 19, ProtoPool seems to compare the front of a BMW to that of an Audi. It appears to find a similarity between the kidney grille of the BMW to the front of the Audi, or it could be the shape of the lights that are similar. On the other hand, the multiple visualizations from ProtoPool-Concepts suggest that the model is actually capturing the kidney grille of the BMW. The other visualizations further show that the ProtoPool-Concept found a part of the car on the bottom, just in front of the tires, is similar to the same part of the other types of SUVs. Figure 20 shows that our model finds the wheel of the Audi TT RS Coupe is similar to the wheels of different types of Audis.

Table 12 shows some additional performance experiments for ProtoPNet-Concepts and ProtoPool-Concepts on the Cars dataset. The training schedule and parameter settings are the same as reported above. The initialized number of prototypes for ProtoPNet-Concepts and ProtoPool-Concepts are 1960 and 195 respectively. The radius initialization for ProtoPNet-Concept is 7.5, and the radius initialization for ProtoPool-Concept is 4.7. As shown in the table, ProtoPNet-Concepts under the DenseNet121 baseline has better performance with fewer visualizable prototypes than ProtoPNet, and its performance is very close to the baseline. On the other hand, ProtoPool-Concepts under a DenseNet161 backbone yields better performance than ProtoPNet with much fewer visualizable prototypes.

TesNet:

Why is this bird classified as a Florida

TesNet-Concepts:

Why is this bird classified as a Florida

Looks like

Looks like

Looks like

Comes from

Looks like

Comes from

只

TesNet:

Why is this bird falsely classified as a

Cedar Waxwing?

TesNet-Concepts:

Why is this bird falsely classified as a

Cedar Waxwing?

Looks like

Looks like

Comes from

Looks like

KUL

KUL

ProtoPool:

Why is this car classified as a BMW X3

SUV 2012?

ProtoPool-Concepts:

Why is this car classified as a BMW X3

SUV 2012?

Looks like

Looks like

Comes from

…

Looks like

Comes from

Looks like

…

PE722

ProtoPool:

Why is this car classified as a Audi TT

RS Coupe 2012?

ProtoPool-Concepts:

Why is this car classified as a Audi TT

RS Coupe 2012?

Looks like

Looks like

Comes from

…

Looks like

Comes from

Looks like

…

T

CD

G Training software and platform

We implemented our ProtoConcepts on ProtoPNet, ProtoPNet, and ProtoPool using PyTorch. The experiments were run on 1 NVIDIA Tesla V100 or 1 NVIDIA RTX A5000. Using the ProtoConcept module with any base architecture has similar training times to using the base architecture alone.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new interpretable prototype-based classifier, called ProtoConcepts. Unlike existing prototype-based classifiers that use one-to-one comparisons, ProtoConcepts learns prototypical concepts using multiple image patches. This approach aims to make it easier to identify the underlying concept being compared, allowing for richer and more interpretable visual explanations. Experiments show that this modified \"this looks like those\" reasoning process can be applied to various prototypical image classification networks without affecting accuracy on benchmark datasets.\n\nSTRENGTHS:\n1.\tThe authors make a valid point that utilizing a single training image patch as a prototype can be insufficient for users to comprehend the concept the prototype represents. For example, when presented with a \"blue circle\" prototype, it is unclear whether the concept is related to the color \"blue\" or the shape \"circle.\"\n\n2.\tThe paper (Table 1) demonstrates that ProtoConcepts can be integrated with various prototype-based classifiers, such as ProtoPNet, TesNet, and ProtoPool, showcasing its adaptability across different methods.\n\nWEAKNESSES:\n1. It is definitely not a new idea in the XAI community to use multiple images/patches to visualize a concept. This is a common practice in the works of concept-level explanations, like [TCAV](http://proceedings.mlr.press/v80/kim18d/kim18d.pdf), [ACE]( https://arxiv.org/pdf/1902.03129.pdf), and so on. However, none of them are discussed in the paper;\n\n\n2. The main issue of the paper lies in the evaluation of interpretability. It is unclear whether these examples are cherry-picked. Also, given that the source code of the method is unavailable at this moment, it is unknown how good/bad the explanation results would be for many other cases.\n\nIf the examples are not cherry-picked, it would be better for the authors to clearly state this in the paper or use examples in a fixed order (e.g., always use the first example of each class).\n\nIt is hard to be convinced that the interpretability of ProtoConcepts is good by showing several separate examples. I would suggest that the authors conduct a carefully designed human user study. The user study results should better convince users that the method brings about a better understanding of the model's behavior. User studies are often conducted in the work of concept-level explanations.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\n1. The novelty of the paper appears to primarily lie in the combination of two distinct lines of research within XAI - concept-level explanations and prototype-based classifiers.\n\n2. The absence of user study results and the unavailability of source code make it difficult to be convinced about the interpretability of the explanations generated by the proposed method.\n\nSee more details in the part of \"Weaknesses\".", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a modification of the prototype layer of the existing prototype-based networks. Existing methods rely on a single training image patch as a prototype. The main drawback of this is that it is difficult for the user to understand the meaning of the prototype from a single visualization. The paper proposes to learn prototypical concepts which are visualized using multiple training image patches. In this way, the semantic of each prototype is less ambiguous.\n\nSTRENGTHS:\n_Clarity_: the paper is generally well-written and structured clearly. The figures showing image examples are very helpful, especially for readers that are not experts on prototypical networks.\n\n_Significance_: the paper addresses a relevant problem which is well known to practitioners using prototypical networks. The problem is filling the gap between what the prototypes represent and the human understanding of prototype semantics. Previous works tried to tackle it. \nThe proposed changes do not badly impact the overall performance of the prototypical networks; indeed, the accuracy is comparable to previous methods.\n\n_Quality and originality_: the two novel losses are technically sound. The idea of representing the prototypes as a ball in the latent space is interesting. The solution can be applied to several types of prototypical networks.\n\n_Reproducibility_: the code will be made available upon acceptance (it was unavailable to the reviewers). The supplementary material reports the training parameters.\n\nWEAKNESSES:\nAs reported in the introduction, the proposed solution should allow the user to determine the semantic meaning of each prototype with less ambiguity. However, the experiments do not report one or more metrics that show this reduction of ambiguity with respect to the plain prototypical networks. The paper and the supplementary material reports some example of concrete cases but not an overall evaluation. The optimal solution would be to run an evaluation with real users, but I understand that it is expensive.\nThe paper [1] proposes activation precision as an interpretability metric (section 4.2) (the segmentation masks are provided with CUB-200).\n\nGiven that Cub200 dataset provides part location annotations for the images, another metric could measure the degree to which the multiple visualizations of the prototype refer to the same bird part. \n\n[1] Barnett 2021, \"IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammography\"\n\nQUESTIONS:\n- what is the main advantage of your solution with respect to learning standard prototypes (e.g., as done in ProtoPNets) and visualizing it on multiple most activated training images? In other words, the original work on ProtoPNets visualizes the prototype on the most activated training image; what if you just take more images?\n- How is k set in the experiments? Why does the value of k change depending on the prototype-based model?\n\nLIMITATIONS:\nThe authors address the limitations. I would like to stress a point that is also discussed in the limitations. The user expertise influences the impact of the method on identifying the concepts: given the same set of prototype visualization, an expert user has a different interpretation of them with respect to a non-expert.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the patch comparison of prototypical part-based classifiers from one-to-one patch to one-to-multiple (e.g. from this looks like that to this looks like those). They compare their proposed network with Proto-PNet variants and show no classification accuracy improvements and little evidence of better interpretability. The proposed method was tested on CUB and Cars as previously done in ProtoPNet variants.\n\nI have read the author’s rebuttal and adjust the rating accordingly!\n\nSTRENGTHS:\n1. The research is well positioned in the literature of prototypical part-based classifiers.\n2. The extension from 1vs1 to 1vsmany is interesting and could help improve human understanding tasks that all prototypical part-based classifiers fall short on [1] \n\n[1] hive: evaluating the human interpretability of visual explanations\n\nWEAKNESSES:\nThere are multiple weaknesses of this paper, mostly about the novelty.\n1. I see the extension is incremental though well motivated. There is no surprise in the classifier accuracy compared to other ProtoPNet variants. I always see (all of them) they perform roughly the same. I believe the \"big fish\" is to really make this prototypical part-based explanations usable for humans. \n2. Although it was motivated by the lack of interpretability of prototypical part-based classifiers, the analysis of model interpretability is superficial and not spot-on. I believe maybe this work also fails to help humans [1] as its siblings [2,3]. The authors could improve the interpretability analysis by running more extensive study (both automatic and human studies) rather than dissecting sample-wise visualizations that could be easily cherry-picked and misled.\n3. I think the paper lacks the analysis of computation cost (i.e. 1vs1 and 1vsmany comparison).\n\n[2] ProtoPNet\n[3] ProtoTree\n\nQUESTIONS:\nThe 1vsmany patch-wise comparison has been studies and they even conducted human studies on interpretability. How this work differs from them [4].\n\n[4] visual correspondence-based explanations improve ai robustness and human-ai team accuracy\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper improves interpretability for prototypical learning. Most previous methods use a single prototype as an interpretation -- and this is not as informative as we don't know what portion of the prototype corresponds to our target image. \n\nThe authors introduce a *prototypical ball* to interpret what prototypes are used for inference. Instead of selecting a single prototype, training samples from this prototypical ball are sampled.\n\nSTRENGTHS:\n*Originality*: I am not completely informed on all prototypical learning methods, but it appears that this method is original. In addition to the novel use of a ball for multiple visualizations, they describe a training scheme specifically for their method. This contributes to the originality of their work. \n\n*Quality*/*Clarity*: The paper is generally presented well, with descriptions of previous work and their drawbacks, as well as what contributions improve the current work. I think some of the descriptions of previous work is not very clear, but I understand space constraints preclude a better description. I believe that the main contribution of this paper can be more clearly elucidated in the figures, which I describe in ``Weaknesses\" section.\n\n*Significance*: I think this work is important to the community. While it matches performance of previous methods, it presents a new way of interpreting the results of algorithms. Integrating this novelty into previous works is adds to its significance.\n\nWEAKNESSES:\n* A better explanation of interpretability would stregthen the paper. The results do improve interpretability, but the fundamental question (that is introduced in the paper) still remains: why are these selected as prototypes? There are more images and we humans can make connections and inferences, but it's still unclear as to why these are the prototypes within the ball.  I do see this explicitly addressed in Section 5, but I still think it is a weakness of the paper. I think an analysis like your reference [16] would strengthen your paper signficantly. I don't know if it is critical to this paper, but it would be a good follow-up paper.\n\n\n* Figure 2 is fairly unclear. It's difficult to understand what your method is actually doing. It's not clear how the logits are obtained either. Overall, if I were to look at that diagram without the help of the text, it would be quite unclear as to what your novelty is and how it is implemented. The ball that you introduce, which is your main contribution, is missing from the figure.\n\n\nA couple small things:\n\n* Please add reference for ProtoPool in Figure 1 and line 209.\n* Reference [16] and [18] look very similar -- are these the same?\n* The text \"ProtoConcepts Layer $g_p$\" in Figure 2 looks compressed compared to the rest of the text in that figure.\n\nQUESTIONS:\n* Do you have an ablation on the loss scaling parameters in the supplementary (Tables 2,4)? How were these $\\lambda$ chosen? How does performance vary with this?\n\n* Why was $k=10$ chosen for $\\mathcal{L}_{\\text{clstk}}$? Do you have an ablation on this?\n\n* Continuing from the section above, I think a figure showing the ball, it's center, and radius with its prototypes would help readers understand this work more clearly. For example in Figure 1, the multiple visualizations of prototypical concepts doesn't clearly identify how those multiple visualizations are obtained (i.e. you could use the same figure if the results were the same using a k-NN algorithm) (same for figure 2)\n\n* In equation three, I think it's important that you show the full loss equation i.e. $\\mathcal{L} = \\mathcal{L}_{\\text{other terms}} + \\lambda_1 \\mathcal{L}_{\\text{clstk}} + \\lambda_2 \\mathcal{L}_{rad} $. It would be clear as to what the loss weights mean in Section 4.1.1\n\nLIMITATIONS:\nThe authors have adequately addressed the negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}