Interpretable Image Classification with Adaptive Prototype-based Vision Transformers
Authors: Chiyu Ma
Jon Donnelly
Wenjun Liu
Soroush Vosoughi
Cynthia Rudin
Chaofan Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: deep learning
interpretability
prototype-based neural network
case-based reasoning
Primary area: interpretability_and_explainability

Abstract: We present ProtoViT, a method for interpretable image classification combining deep learning and case-based reasoning. This method classifies an image by comparing it to a set of learned prototypes, providing explanations of the form ``this looks like that.'' In our model, a prototype consists of **parts**, which can deform over irregular geometries to create a better comparison between images. Unlike existing models that rely on Convolutional Neural Network (CNN) backbones and spatially rigid prototypes, our model integrates Vision Transformer (ViT) backbones into prototype based models, while offering spatially deformed prototypes that not only accommodate geometric variations of objects but also provide coherent and clear prototypical feature representations with an adaptive number of prototypical parts. Our experiments show that our model can generally achieve higher performance than the existing prototype based models. Our comprehensive analyses ensure that the prototypes are consistent and the interpretations are faithful.

Interpretable Image Classification with Adaptive
Prototype-based Vision Transformers
Chiyu Ma
Jon Donnelly Duke University
Wenjun Liu
Dartmouth College chiyu.ma.gr gra.eaa...rrr.eo eaayyumaagrddatttuuuueellel dartmouth.edu
Dartmouth College wenjun lig dartmouth.edl
jon.donnelly @duke.edu
Soroush Vosoughi
Cynthia Rudin
Chaofan Chen
Dartmouth College
Duke University
University of Maine chaofan.chen @ mainee.du
soroush. vosoughi @osooughiddattttutt........ dartmouth.edu
cynthia@ess crutiia@cs.duke.chl duke.edu
Abstract
We present ProtoViT, method for interpretable image classification combining deep learning and case-based reasoning. This method classifies an image by comparing it to set of learned prototypes. providing explanations of the form "this looks like that." In our model, a prototype consists of parts, which can deform over irregular geometries create better comparison between images. Unlike existing models that rely on Convolutional Neural Network (CNN) backbones and spatially rigid prototypes, model integrates Vision Transformer (ViT) backbones into prototype based models, while offering spatially deformed prototypes that not only accommodate geometric variations of objects but also provide coherent and clear prototypical feature representations with an adaptive number of prototypical parts. Our experiments show that model can generally achieve higher performance than the existing prototype based models. Our comprehensive analyses ensure that the prototypes are consistent and the interpretations are faithful. Our code is available at https: :etps://er //gi github...... com/ wtt.ecoo/////// Hew.ymacchiy//PPooootii.
Introduction
With the expanding applications machine learning models in critical and high-stakes domains like healthcare [4, 14, 44, 52, 51], autonomous vehicles [32], finance [47] and criminal justice [5], it has become crucial to develop models that are not only effective but also interpretable by humans. This need for clarity and accountability has led to the emergence of prototype networks. These networks combine the capabilities of deep learning with the clarity of rase-based reasoning, providing understandable outcomes in fine-grained image classification tasks. Prototype networks operate by dissecting an image into informative image patches and comparing these with prototypical features established during their training phase. The model then aggregates evidence of similarities these prototypical features to draw a final decision on classification.
Existing prototype-based models [10, 4, 13, 26, 35, 36, 46, 27, 45, 8, 7]. which are inherently interpretable [34] (i.e., the learned prototypes are directly interpretable, and all calculations can be visibly checked), mainly developed with convolutional neural networks (CNNs). As vision transformers (ViTs) [15, 40, 41] gain popularity and inspire extensive applications, becomes crucial to investigate how prototype-base based architectures Though few attempts to develop a prototype-based vision transformer have been made [50, 49, 22), these methods do not provide inherently interpretable explanations of the models' reasoning because they do not project the learned prototypical features to examples that actually exist in the dataset.
be integrated with vision transformer backbones.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Why is this bird classified as a Male Ruby Throated Hummingbird?
ProtoPNet:
image
Protonype
Image
Protetypa
broud, explerations
protehypas
Looks like
Looks like
antiguous.
Test image
Protetype
Test Image
Protocype
udapt piclotypical
festare
Looks like
Looka like
represented by boane, They what pronotypa capturee.
Test
ProtoVIT (Ours):
Looks like
Looka like
shipa and have comantic
Each feature
Teat Image
Looks like
Prototyse
Teat
Looka like
Prototype
Figure 1: Reasoning process of how prototype based models identify a test image of male Ruby- Throated Hummingbird as the correct class. Prototypes are shown in the bounding boxes.
Table 1:
Table of contributions of PiotoVit (Ours) compared to existing works.
Supoon Rucibooe??
Deformable Protetypes?
Coberem Protetypes?
Adoptovo Sizee?
lnlio Intespicahii?
ProtoPNer
Maybe
Deformable ProtaPNes [13]
Yes
Prouftoemer (50)
Yes
No
Maybe Yes
No
No
ProtoVIT (Ours)
Vis
Yis
Yes
Vei
Thus, the cases that these models use in their reasoning have no well defined visual representation, making it impossible know what exactly each prototype represents.
The coherence and clarity of the learned prototypes are important. Most prototype-based models, such ProtoPNet [10], TesNet [46] and ProtoConcept [26], spatially rigid prototypical features (such as a rectangle), which cannot account for geometric variations of an object. Such rigid prototypical features can be ambiguous as they may contain multiple features in one bounding box (see top row of Fig. 1 for an example). Deformable ProtoPNet [13] deforms the prototypes into multiple pieces adjust for the geometric transformation. However, Deformable ProtoPNet utilizes deformable convolution layers that rely on a continuous latent space to derive fractional offisets for prototypical parts to move around, and thus are not suitable for models like ViTs which output discrete tokens as latent representations. Additionally, the prototypes learned by Deformable ProtoPNet tend to be incoherent (see Fig. 1, middle row).
Addressing the gaps in current prototype-based methods (see Table 1), we propose the prototype- based vision transformer (ProtoViT)) novel architecture that incorporates ViT backbone and can adaptively learn inherently interpretable and geometrically variable prototypes of different sizes, without requiring explicit information about the shape or size of the prototypes. We do so using novel greedy matching algorithm that incorporates an adjacency mask and an adaptive slots mechanism. We provide global and local analysis, shown in Fig. that empirically confirm the faithfulness and coherence of the prototype representations from ProtoViT, We show through empirical evaluation that Proto VIT achieves of-the-arr-- accuracy well excellent clarity and coherence.
Related Work
Posthoc explanations for CNNs like activation maximization [16, 28), image perturbation [17, 20], and saliency visualizations [3, 38, 39] fall short in explaining the reasoning process of deep neural networks because their explanations are not necessarily faithful [33, 2]. In addition, numerous efforts have been made to enhance the interpretability of Vision Transformers (ViTs) by analyzing attention weights [1, 42), and other studies focus understanding the decision-making process through gradients [18, 19], attributions [9, 53], and redundancy reduction techniques [30]. However, because of their posthoc nature, the outcomes of these techniques can be uncertain and unreliable [2, 25].

---

In contrast, prototype- prosotype-based based approaches offer a transparent prediction process through case-based reasoning. These models compare small set of learned latent feature representations called prototypes (the "cases") with the latent representations of test image to perform classification. These models are inherently interpretable because they leverage comparisons to only well-defined cases in reasoning. The original Prototypical Part Network (ProtoPNet) [ [10] employs class-specifie prototypes, allocating a fixed number of prototypes to each class. Each prototype is trained to be similar to feature patches from images of its own class and dissimilar to patches from images of other classes. Each prototype is also latent patch from a training image, which ensures that "cases" are well-defined in the reasoning process. The similarity score from the test image to each prototype is added as positive evidence for each class in a "scoresheet," " and the class with the highest score is the predicted class. The Transparent Embedding Space Network (TesNet) [46] refines the original ProtoPNet by utilizing a cosine similarity metric to compute similarities between image patches and prototypes in latent space. It further introduces new loss terms encourage the orthogonality among prototype vectors of the same class and diversity between the subspaces associated with each class. Deformable ProtoPNet [13] aims decompose the prototypical parts into smaller sub-pat patches and use deformable convolutions [12, 54] to capture pose variations. Other works [35, 27, 36] move away from class-specific prototypes, which reduces the number of prototypes needed. This allows prototypes to represent a similar visual concept shared in different classes. Our work is similar to these works that define "cases" as latent patch representations by projecting the trained class-specific prototypes to the closest latent patches.
As alternative to the closest training patches, ProtoConcept [26] defines the cases group of visualizations in the latent spaces bounded by a prototypical ball. Although they are not projected, the visualizations of ProtoConcept are fixed to the set of training patches falling in each prototypical ball, which establishes well defined cases for each prototype. On the other hand, works such as ProtoPFormer [50] do not project learned prototypes the closest training patches because they observed a performance degradation after projection. The degradation is likely caused by the fact that the prototypes are not sufficiently close to any of the latent patches. The design of a "global branch" that aims to learn prototypical features from class tokens also raises concerns about interpretability. as visualizing an arbitrary trained vector (class token) does not offer any semantic connection the input image. On the other hand, work such as ViTNet [22] also lack details about how the "cases" are established, yielding concerns about the interpretability of the model. Without a mechanism like prototype projection [10] or profotypical concepts [26] to enable visualizations, prototypes are just arbitrary learned tensors the latent space of the network, with no clear interpretations. Visualizing nearby examples alone cannot explain the model's reasoning, since the closest patch to prototype can be arbitrarily far away without the above mechanisms. Simply adding a prototype layer to an architecture without well defined "cases" in the reasoning process does not make the new architecture more interpretable.
Our work is also related to INTR [31], which trains a class specific attention query and inputs it to a decoder localize the patterns in an image with cross-attention. In contrast, our encoder-only model, through a different reasoning approach, learns patch-wise deformed features that are more explicit, semantically coherent, and provides more detail about the reasoning process than attention heatmaps [21, 48, 6] it shows how the important pixels are used in reasoning. not just where they are.
3 Methods
We begin with a general introduction of the architecture followed by an in-depth exploration of its components. We then delve into our novel training methodology that encourages prototypical features to capture semantically coherent attributes from the training images. Detailed implementations on specific datasets are shown in the experiment sections.
3.1 Architecture Overview
Fig. 2 shows an overview of the ProtoVit architecture. Our model consists of a feature encoder layer f. which pretrained ViT backbone that computes a latent representation of an image; a greedy matching layer y. which compares the latent representation to learned prototypes to compute prototype similarity scores; and an evidence layer h, which aggregates prototype similarity scores into classification using a fully connected layer. We explain each of these in detail below. Let
3

---

sub-prototypes
Similarity Comparison and calculation Input Images
Summed Similarity Score
Cla token
Protetypes
(Tsss)
Groody
(csss)
Logit Values
Gready Matching
Greedy
Transformar Encoder
Adjacent Maak
[|] Adjacency
@ Motcodjacenyy
Input Images
Sigmoid plons Indicator prot--pototppss
epait
patchee clarity. manty the image
Sub prototype included
spli
Sub protopyottpp not included
Encoded Putch tokees
Matching sub-protstypes pes with the most similar latent feature tokens, reality, the reshaped encoded patch tokens are in shape 14 14 larger
Feature Encoder Layer
Greedy Matching Prototype Layer g
Evidence
Figure Architecture of Proto VIT. The feature encoder layer f can be any kind of vision transformer encoders such as DeiT and CaiT. The greedy matching and prototype layer deforms the prototypes into sub- prototyrottssss and finds the closest non-ou overlappepppng feature patches from the test image. Our adaptive slots mechanism filters out some number of sub-prototypes, and the sum of the remaining similarity scores (with a correction to avoid down-weighting prototypes with more sub-prototypes filtered out) is returned. The evidence layer h is fully connected layer that computes the logit predictions based on the summed similarity scores.
H W x C' be the shape of an input image where H, W, C' are the height, width and number of channels of the image respectively. The feature encoder layer ff is ViT backbone, which first splits the input images into N units of L L C sized patches (such the Chestnut Sided Warbler in Fig. 2), where I. is the height and width of the image patch. The feature encoder layer f then flattens and encodes the image patches into set of latent feature tokens zy defined later in Eq. where each latent feature token 2 E Rd and d is the hidden dimension of the model. The network aims learn m prototypes P {D, )m where each Dj E k xd is composed of sub-prototype vectors p Rd. The greedy matching layer g then finds the most similar latent feature token ziy. measured by cosine similarity, of the input image each sub-prototype P without replacement (i.e., if p) matches z). P cannot match with z1)). We further introduce an adjacency mask to ensure that, for each prototype Pj. its sub-prototyped P are geometrically contiguous. We then introduce our adaptive slots mechanism, which decides whether each prototype Py should include each sub-prototype P based on coherence to the other sub- sub-prototypes and its influence on model performance, allowing the model to learn prototypes with dynamic number of sub-prototypes. Finally, the evidence layer computes the logit values for each class based on the summed cosine similarity score g8, function to make predictions. Intuitively, the prototypes, which later projected the closest latent feature tokens, can be viewed as the most representative features of each class that the model found among the training examples. The model performs classification based the input's similarity to these key features.
greedy for each prototype Py. The logit values are then normalized by a softmax
3.2 Feature Encoder Layer
Given an input image x E XXXXXC the feature encoder layer first splits the image into N unit patches Xpatch each of shape L x L C. It then flattens the unit patches and projects them

---

to the embedding space as XEmbed R which is then prepended with a trainable class to- ken Xelass E Rd and passed into the sition embedding Epos I The ViT backbone then outputs the encoded tokens as z == Ieelassi paach "panchi.... : pucch where z E ([++1)xd In this sense, each of the output patch tokens Spucch is the latent representation of the corresponding unit patch X The class token can be viewed as a way to approximate the weighted sum over patch tokens, enabling an image-level repre- sentation. It is, thus, difficult to visualize the class token and therefore unsuitable for comparisons to prototypes. Drawing inspiration from the idea of focal similarity [36], which involves calculating the maximum prototype activations minus the mean activations achieve more focused representations, we take the difference between patch-wise features and the image-level representation. In doing so, we aim to similarly produce more salient visualizations. Specifically, define the feature token / by taking the difference between each patch token and the image-level representation. Thus, latent feature tokens can be written as:
Attention layers along with a learnable po-
f 1 z 2z where 2)- - Zelass and 2 E IR.
(I)
By this design, not only encode richer semantic information within the latent feature tokens, but also enable the application of prototype layers to various ViT backbones that contains class token. An ablation study shows that model performance drops substantially when removing the class token from the feature encoding (see Appendix Sec. E.I).
3.3 Greedy Matching Layer
The greedy matching layer g integrates three key components: greedy matching algorithm, adjacency masking, and an adaptive slots mechanism, as illustrated in Fig. 2.
Similarity Metric:
The greedy matching layer contains prototypes P= {p>) where each Py consists of K sub- sub-prototypes P that have the same dimension as each latent feature token z Each sub-prototype pf is trained to be semantically close to at least one latent feature token. To measure the closeness of the sub-prototype P and latent feature token zzy. we use cosine similarity
x;" p'
cf)( E - which has range -1 to 1. The overall similarity between prototype p; and the latent feature tokens 2 is then computed as the sum across the sub-prototypes of the cosine similarities, which has a range from - K to K.
Greedy Matching Algorithm: We have not yet described how the token z to which P is compared is selected. In contrast to prior work, we do not restrict that sub-prototypes have fixed relative locations. Rather, we perform greedy matching algorithm to identify and compare each of the K sub-prototypes D of Pj to any K non-overlapping latent feature tokens. To be exact, for given prototype Pj with K sub-prototypes, feature token z that are closest in cosine similarity, "match" ziz with pf. and remove z and p from the next iteration, until all K pairs are found. This is illustrated Fig. 2, and more details can be found in Appendix Sec. C.
iteratively identify the sub-prototype p and latent
Without any restrictions, this greedy matching algorithm can lead to many sub-prototypes that are geometrically distant from each other. Assuming that the image patches representing the same feature are within r E R spatial units of one another in horizontal, vertical and diagonal directions, we introduce the Adjacency Mask A to temporarily mask out latent feature tokens that are more than r' positions away from a selected sub- bub-prototype/latent feature token pair in all directions. Within each iteration & of the greedy matching algorithm, the next pair only be k- - };zy;r), r), that are within r positions
Adjacency Masking:
selected from the latent feature tokens, AĄt = A(Ip) 8-1 of the last selected pair. An example of how the adjacency mask with r = 1 works is illustrated in Fig. 2. Incorporating adjacency masking with the greedy matching algorithm, we thus find K non-overlapping and adjacent cubb-rototype-ltteee patch pairs.
Adaptive Slots Mechanism: Because not all concepts require K sub-prototypes to represent, we introduce the the adaptive slots mechanism S. The adaptive slots mechanism consists of learnable
5

---

vectors V R and a sigmoid function with a hyperparameter temperature We chose sufficiently large value for ensure that the sigmoid function has steep slope. The vectors are sent the sigmoid function to approximate the indicator function as [Include pÿ} Sigmoid (v)) F).
Each 1 [Include ) } is an approximation of the indicator for whether the k-th sub-prototype will be included the j-th prototype Pj. More details can be found in Appendix Sec. D.
Computation of Similarity: As described above, measure the similarity of selected latent- mutch-sub-potooyyp pair using cosine similarity. We use the summed similarity across sub-prototypes to measure the overall similarity for prototype Pj with K selected non-overlapping latent fea- ture tokens. As pruning out some sub-prototypes for a prototype Pj reduces the range of the summed cosine similarity for μj. we rescale the summed similarities back to their original range by K K/Ex k (include }. We formally define the reweighted summed similarity function obtained from the greedy matching layer as:
K
garosd (zz)
max cos(z), p}} z ZyEu,g
K E
finclude p
finclude p
R=1
3.4 Training Algorithm
Training of ViT has four stages: (1) optimizing layers before the last layer by stochastic gradient descent (SGD); (2) prototype slots pruming; (3) projecting the trained prototype vectors to the closest latent patches; (4) optimizing the last layer h. Note that well-trained first stage is crucial to achieve minimal performance degradation after prototype projection. procedure plot is illustrated in Appendix Fig. 5.
Optimization of layers before last layer: The first training stage aims to learn a latent space that clusters feature patches from the training set that are important for class near semantically similar prototypes of that class. This involves solving a joint optimization problem over the network parameters via stochastic gradient descent (SGD). We initialize every slot indicator i (Include nt) as
1 to allow all sub-prototypes to learn during SGD. We initialize the last layer weight similarly to ProtoPNet [10]. The slot indicators and the final layer are frozen during this training stage. The slot indicator functions are not involved in computing cluster loss or separation loss because each sub-prototype should remain semantically close to certain latent feature tokens, regardless of its inclusion in the final computation. Since we deform the prototypes, we propose modifications to the original cluster and separation loss defined in [10] to:
1
LC1st
Σ max i=1
max cos(z), P"); Eppp 2n
p, E
n
Lsep
max max max cos(z), p/). Ex P/EP; 2/
n1
The minimization of this new cluster loss encourages each training image to have some unit latent feature token z that is close at least sub-prototypes p of its own class. Similarly, by minimizing the new separation loss, we encourage every latent feature token of training image to stay away from the nearest aub-prototype from any incorrect class. This similar to the traditional cluster and separation loss from [10] if we define the prototypes to consist of only We further introduce a novel loss term, called coherence loss, based on the intuition that, the sub- prototypes collectively represent the same feature (c.g., the feet of bird), these sub-prototypes should be similar under cosine similarity. The coherence loss is defined as:
uub-prototype.
1 m
LCoh
(1 cos(p); p3)) (Include p il [Include Dz)- P
(4)
m
Intuitively, the coherence loss penalizes the sub-prototypes that are the most dissimilar to the other sub-prototypes out of the K slots for prototype P,- The slots indicator function is added to the

---

coherence loss term to prune sub- prototypes that are semantically distant from others in prototype. Moreover, we use the orthogonality loss, introduced in previous work [46, 13], to encourage each prototype Pj to learn distinctive features. The orthogonality loss is defined as:
Corth Σ 1=1
1pp Ipll
(5)
where C' is the number of classes. For each class I with p assigned prototypes, p() RRx is matrix obtained by flattened the prototypes from class I, and is the number of prototypes P, for each class. L an identity matrix in the shape of p x p. Overall, this training stage aims minimize the total loss as:
Ciotal LCE Cc + Xee + AzhCoh A4horth
(6)
where Lee is the cross entropy loss for classification and λ1 A2, A3, da are hyper-parameters.
Slots pruning: In this stage, goal is to prune the sub-prototypes P that dissimilar to the other sub-prototypes for each prototype. Intuitively, we aim to remove these sub-prototypes because sub- coob-proootpps that are not similar will lead prototypes with an inconsistent, unintuitive semantic meaning. We freeze all of the parameters in the model, except the slot indicator vectors v. During this stage, we jointly optimize the coherence loss defined in Eq. along with the cross entropy loss. We lower the coherence loss weight to avoid removing all the slots. Since the slot indicators approximations of step functions using sigmoid functions with a sufficiently high temperature parameter т, the indicator values during this phase are fractional but approach binary values close to 0 or The loss in the stage is defined as:
L LCE + AsLCon-
Projection: In this stage, we first round the fractional indicator values to the nearest integer (either I or 0), and freeze the slot indicators' values. Then, we project each prototype Dj to the closest training image patch measured by the summed cosine similarity defined in Eq. 2, as in [10]. Because the latent feature tokens from the ViT encoder correspond to image patches, we not need to up-sampling techniques for visualizations, and the prototypes visualized in the bounding boxes represent the exact corresponding latent feature tokens that the prototypes are projected to.
As demonstrated in Theorem 2.1 from ProtoPNet [10], if the prototype projection results in minimal movement, the model performance is unlikely to change because the decision boundary remains largely unaffected. This is ensured by minimizing cluster and separation loss, as defined in Eq. 3. In practice, when prototypes are sufficiently well-trained and closely clustered to certain latent patches, the change in model performance after the projection step should be minimal. Additionally, this step ensures that the prototypes are inherently interpretable, as they are projected to the closest latent feature tokens, which have corresponding visualizations and provide a pixel space explanation of the "cases" in the model's ' reasoning process.
Optimization of last layers: Similar to other existing works [10, 46], we performed convex optimization on the last evidence layer after performing projection, while freezing all other parameters. This stage aims to introduce sparsity to the last layer weights W by penalizing the 1 1-norm of layer weights WA,1 (initially fixed as -0.5) associated with the class for the 1-th class prototype p', where I & By minimizing this loss term, the model is encouraged to use only positive evidence for final classifications, The loss term is defined as:
La LCE + Ao
Σ 1
Σ
[bblll
(8)
BE bE(D,..,C))bbf
4 Experiments
4.1 Case Study 1: Bird Species Identification
To demonstrate the effectiveness of Pit we applied it to the cropped Caltech-UCSD Birds-200- 2011 (CUB 200-2011) dataset [43]. This dataset contains 5,994/5,794 images for training and testing
7

---

Table 2: Comparison of ProtoVit implemented with DeiT and CaiT backbones to other existing works. Our model is not only inherently interpretable but also superior in performance compared to other methods using the same backbone. We also include models with a CNN backbone (Densenet-16) for reference in the top section. The reported accuracies are the final results after all training stages. Arch.
Model
CUB Acc.[%]
Car Acc.[%]
ProtoPNet (given in [TO])
80.1 + 0.3
89.5 + 0.2
Densenet-161
Def. ProtoPNet(2x2) (given in (13))
80.9 =0.22
88.7 = 0.3
~88.68M params
ProtoPool (given in [36])
80.3 + 0.3
90.0 + 0.3
TesNet (given in [46])
81.5 + 0.3
92.6 + 0.3
Base (given in [50]
80.57
86.21
Diny-Tiny
ViT-Net (given in [50])
81.98
88,41
-MM params
Proto (given in [50])
82.26
88.48
Proto ViT(K44 r=1) (ours)
82.92 + 0.5
89.02 + 0.1
Baseline (given in [50]
84.28
90.06
DeiT-Small
ViT-Net (given in [50]
84.26
91.34
-2M params
ProtoPFormer (given in [50])
84.85
90.86
Proto (oooo, r=1) (ours)
85.37 + 0.13
91.84 + 0.3
Baseline (given in [50)
83,95
90.19
CaiT-XXS 24
ViT-Net (given in [50])
84.51
91.54
11.9M params
ProtoPFormer (given in [50])
84.79
91.04
Proto Vi=( r=1) (ours)
85.82 + 0.15 92.40 + 0.1
from 200 different bird species. We performed similar offline image augmentation to previous work [10, 46, 26] which used random rotation, skew, shear, and left-right flipping. After augmentation, the training set had roughly 1,200 images per class. We performed prototype projections on only the non-augmented training data. Additionally, we performed ablation studies on the class token (See Appendix Sec. 1) coherence loss (See Appendix Sec. E.3), and adjacency mask (See Appendix Sec. E.2). The quantitative results for the ablation can be found in Appendix Table. 6. We assigned the algorithm to choose 10 class-specific prototypes for each of the 200 classes. Each the prototypes is composed of 4 uub-prototypes. Each set of sub-prototypes was encouraged to learn the 'key" features for its corresponding class through only the last layer weighting, and the sub-prototypes were designed to collectively represent one key feature for the class. More discussion different choices of K can be found in Appendix F. For more details about hyperparameter settings and training schedules, please refer to Appendix Sec. A and Appendix Sec. B respectively. Details and results of user studies regarding the interpretability of Proto ViT can be found in Appendix. Sec. K.
4.1.1 Predictive Performance Results
The performance of our Proto VIT with CaiT and DeiT backbones is compared that of existing work in Table 2, including results from prototype- based models using DenseNet161 (CNN) backbones. Integrating ViT (Vision Transformer) backbones with smaller number of parameters into prototype- based algorithms significantly enhances performance compared to those using CNN (Convolutional Neural Network) backbones, particularly more challenging datasets. Compared with other prototype-based models that utilize ViT backbones, our model produces the highest accuracy, outperforming even the black box ViTs used as backbones, while offering interpretability. We provide accuracy using an ImageNet pretrained Densenet- 161 to match the ImageNet pretraining used in all transformer models.
4.1.2 Reasoning Process and Analysis
Fig. 3 shows the reasoning process of Pit for a test image of a Barn Swallow. Example visualizations for the classes with the top two highest logit scores are provided in the figure. Given this test image x, our model compares its latent feature tokens against the learned sub- uu-protos P through the greedy matching layer g. In the decision process, our model uses the patches that have the most similar latent feature tokens to the learned prototypes evidences. In the example, our model correctly classifies a test image second most likely class based on prototypical features. In addition to this example reasoning process, we conducted local and global analyses (shown in Fig. 4) to confirm the semantic consistency of prototypical features across all training and testing images, where local and global analyses
Barn Swallow and thinks the Tree Swallow is the
8

---

Reasoring for this bird being Barn Swallow:
Reasoning bird being Tree Swallow:
Text Trogs
Protosypes
Saneigeed sinlarty
Lad
Toor irage
Protoppos.
Semeiged similartt
Lad
Tote
seight
2666 6.901
3.28
2.62
0.040
0.002
0.841
0.004
1.77 X 0.541 1.67
:
:
Total Points Barn Swallow:
Total Points Tree Swallow:
31.8
15.6
Figure Reasoning process of how ViT classifies test image of Barn Swallow using the learned prototypes. Examples for the top two predicted classes are provided. We use the Deil-Small backbone with r=1 and h=4 for the adjacency mask.
Protohypas Images
Protolype
Nearest Training patches grotetaps
Neareet Test patches the prototype bobeiiiing
and Iriage
Figure 4: Nearest prototypes to test images (left), and the nearest image patches to prototypes (right). We exclude the nearest training patch, which is the prototype itself by projection.
defined in [10]. The left side of Fig. 4 displays local analysis examples, visualizing the most semantically similar prototypes to each test image. The right side of Fig. 4 shows global analysis examples, presenting the top three nearest training and testing images the prototypes. Our local analysis confirms that, across distinct classes and prototypes, the comparisons made are reasonable. For example, the first prototype compared to the white breasted kingfisher seems to identify the bird's blue tail, and is compared to the tail of the bird in the test image. Further, our global analysis shows that each prototype consistently activates prototype in the first row of the global analysis consistently highlights the black face of the common yellowthroat at variety of scales and poses. Taken together, these analyses show that the prototypes of ProtoViT VIT have strong, consistent semantic meanings. More examples of the reasoning process and analysis can be found in Appendix Sec. H and Sec. respectively.
single, meaningful concept. For example, the
4.1.3 Location Misalignment Analysis
Vision Transformers (ViT') use an attention mechanism that blends information from all image patches, which may prevent the latent token at a position from corresponding to the input token at that

---

Table Experimental results on the Location Misalignment Benchmark. We compare our Proto ViT (Deit-Small) with other prototype- based models with CNN backbones (ResNet34). We found that our model performs similarly to or better than the existing models with CNN backbones terms of the misalignment metrics and test accuracy.
Method
PI.C
PRC
PAC
Acc. Before
Acc. After
AC.
ProtoPNet
24.0) + 1.7
13.5 = 3.1
23.7 = 2.8
76.4 + 0.2
68.2 + 0.9
8.2 1.1
TesNet
16,0 + 0.0
2.9
3.4
81.6 + 0.2
75.8
5,8
0.6
ProtoPool
31.8 0.8
4.5
0.9
11.2 = 1.3
80.8 +: 0.2
4.8 + 0.1
ProtoTree
27.7 + 0.3
13.5
13.5=3.1
3.1
23.7 = 2.8
76.4 + 0.2
68.2 + 0.9
8.2
1.1
ProtoViT(ours)
21.68 3.1
1.28
0.1
2.92 = 0.1
85.4 + 0.1
82.8 0.3
2.6
+ 0.4
position. To assess whether ViT backbones can be interpreted as reliably as Convolutional Neural Networks (CNNs) in ProtoPNets, we conducted experiments using the gradient-based adversarial attacks described in the Location Misalignment Benchmark [37]. As summarized in Table 3, our method, which incorporates ViT backbone, consistently matched or outperformed leading CNN- based prototype models such as ProtoPNet [10], ProtoPool [36], and Proto Tree [27] across key metries: Percentage Change in Location (PLC), Percentage Change in Activation (PAC), and Percentage Change in Ranking (PRC), as defined in the benchmark. Lower values for these metrics indicate better performance. This shows that Vit is at least as robust as CNN-based models. Moreover, as observed in the Location Misalignment [enchmark[37], the potential for information leakage also exists in deep CNNs, where the large receptive fields of deeper layers would encompass the entire image same as attention mechanisms. While our model is not entirely immune to location misalignment, empirical results indicate that its performance is on par with other state-of-the- art CNN-based models. Qualitative comparisons that further support this point can be found in Appendix. Sec. G, and more results and discussions on PLC for ablated models can be found in Appendix. Sec. E
4.2 Case Study 2: Cars Identification
In this case study, we apply our model to car identification. We trained our model on the Stanford Car dataset[24] of 196 car models. More details about the implementation can be found in Appendix. Sec. L. The performance with baseline models can be found in Table. 2. Example visualizations and analysis can be found in Appendix L. We find that Pio again produces superior accuracy and strong, semantically meaningful prototypes on the Cars dataset.
5 Limitations
While we find our method can offer coherent and consistent visual explanations, it is not yet able to provide explicit textual justification to explain its reasoning process. With recent progress in large vision-language hybrid models [23, 11, 29]. technique offering explicit justification for visual explanations could be explored in future work. it important to note that some visual domains such as mammography other radiology applications, features may not have natural textual descriptions thus, explicit semantics are not yet possible, and larger domain-specific vocabulary would first need to be developed. Moreover, as discussed in Sec. 4.1.3, our model is not completely immune to location misalignment, meaning the learned prototypical features may be difficult to visualize in local patches. However, this issue is not unique to our approach; CNN-based models face the same challenge. As layers deepen, the receptive field often expands to cover the entire image, leading to the same problem encountered with attention mechanisms in vision transformers.
6 Conclusion
In this work, we presented an interpretable method for image classification that incorprates VIT backbones with deformed prototypes to explain its predictions (this looks like that). Unlike previous works prototype-based classification, our method offers spatially deformed prototypes that not only account for geometric variations of objects but also provide coherent prototypical feature representations with an adaptive number of prototypical parts. While offering inherent interpretability, model empirically outperform the previous prototype based methods in accuracy.
10

---

7 Acknowledgement
We would like to acknowledge funding from the National Science Foundation under grants HRD- 2222336 and 01A-2218063 and the Department of Energy under DE-SC0021358.
References
[1] Samira Abnar and Willem Zuidema. Quantifying Attention Flow in Transformers. arXiv preprint arXiv:2005.00922, 2020.
[2] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity Checks for Saliency Maps. Advances in Neural Information Processing Systems, 31, 2018.
[3] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On Pixel- Wise Explanations for Non-Linear Classifier Decisions by Layer-Wiss Wise Relevance Propagation. PloS one, 10(7))e0130144 2015.
[4] Alina Jade Barnett, Fides Regina Schwartz, Chaofan Tao, Chaofan Chen, Yinhao Ren, Joseph Y
Lo, and Cynthia Rudin. A case-based interpretable deep learning model for classification of mass lesions digital mammography. Nature Machine Intelligence, 3(12):0611-1070 2021.
[5] Richard Berk, Drougas Berk, and D Drougas. Machine Leurning Risk Assessments in Criminal Justice Settings. Springer, 2019.
[6] Adrien Bibal, Rémi Cardon, David Alfier, Rodrigo Wilkens, Xiaoou Wang. Thomas François, and Patrick Watrin. Is Attention Explanation? An Introduction the Dehate. Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2889-3900, 2022.
[7] Andrea Bontempelli, Stefano Teso, Katya Tentori, Fausto Giunchiglia, and Andrea Passerini. Concept-level debugging of part-prototype networks. arXiv preprint arXiv:2205.15769 2022.
[8] Zachariah Carmichael, Suhas Lohit, Anoop Cherian, Michael J Jones, and Walter J Scheirer. Pixel-grounded prototypical part networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4768 4779, 2024.
[9] Hila Chefer, Shir Gur, and Lior Wolf. Transformer Interpretability Beyond Attention Visualiza- tion. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 782-791, 2021.
[10] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This Looks Like That: Deep Learning for Interpretable Image Recognition. Advances in Neural Information Processing Systems, 32, 2019.
[11] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: Universal Image-TExt Representation Learning. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, pages 104- 120. Springer, 2020.
[12] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang. Han Hu, and Yichen Wei. Deformable Convolutional Networks. In Proceedings of the IEEE international conference on computer vision, pages 764-773, 2017.
[13] Jon Donnelly, Alina Jade Barnett, and Chaofan Chen. Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10265-10275, 2022.
[14] Jon Donnelly, Luke Moffett, Alina Jade Barnett, Hari Trivedi, Fides Schwartz, Joseph Lo, and Cynthia Rudin. AsymMirai: Interpretable Mammography- Deep Learning Model for 1-5-ycar Breast Cancer Risk Prediction. Radiology, 310(3)) 2024.
11

---

[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x 16 words: Transformers for Image Recognition at Scale. arXiv preprint arXivv:010. / 1929, 2020.
[16] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing Higher-Layer Features of Deep Network. University of Montreal, (3):1 2009.
[17] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding Deep Networks via Extremal Perturbations and Smooth Masks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2950-2958, 2019.
[18] Wei Gao, Fang Wan, Xingjia Pan, Zhiliang Peng, Qi Tian, Zhenjun Han, Bolei Zhou, and Qixiang Ye. TS-CAM: Token Semantic oupled Attention Map for Weakly Supervised Object Localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2886-2895, 2021.
[19] Saurav Gupta, Sourav Lakhotia, Abhay Rawat, and Rahul Tallamraju. ViTOL: Vision Trans- former for Weakly Supervised Object Localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4101-41100 2022.
[20] Maksims Ivanovs, Roberts Kadikis, and Kaspars Ozols. Certurbaton-bbase methods for explaining deep neural networks: A survey. Pattern Recognition Letters, 150:228-234, 2021.
[21] Sarthak Jain and Byron C Wallace. 2019.
Attention is not Explanation. arXiv preprint
[22] Sangwon Kim, Jaeyeal Nam, and Byoung Chul Ko. ViT-NeT: Interpretable Vision Transformers with Neural Tree Decoder. In International Conference on Machine Learning, pages 11 112- 11172. PMLR, 2022.
[23] Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-and- -anguuga Transformer Without Convolution Region Supervision. In International Conference on Machine Learning, pages 5583-5594, PMLR, 2021.
[24] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
[25] Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin De- tyniecki. The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2801-2807, 2019.
[26] Chiyu Ma, Brandon Zhao, Chaofan Chen, and Cynthia Rudin. This Looks Like Those: Illumi- nating Prototypical Concepts Using Multiple Visualizations. Advances in Neural Information Processing Systems, 36, 2024.
[27] Meike Nauta, Ron Van Bree, and Christin Seifert. Neural Prototype Trees for Interpretable Fine-grained Image Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19933-1493,, 2021.
[28] Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in Neural Information Processing Systems, 29, 2016.
(29) Giang Nguyen, Mohammad Reza Taesiri, and Anh Nguyen. Visual correspondence-based ex- planations improve Al robustness and human-AI team accuracy. Neural Information Processing Systems (NeurIPS), 2022.
[30] Bowen Pan, Rameswar Panda, Yifan Jiang. Zhangyang Wang, Rogerio Feris, and Aude Oliva. la-red2: Interpretabillty-Aaaa Redundancy Reduction for Vision Transformers. Advances in Neural Information Processing Systems, 34:24898-2499 2021.
12

---

[31] DIPANJYOTI Paul, Arpita Chowdhury, Xinqi Xiong, Feng-Ju Chang, David Edward Carlyn, Samuel Stevens, Kaiya L Provost, Anuj Karpatne, Bryan Carstens, Daniel Rubenstein, et al. A Simple Interpretable Transformer for Fine-Grane GrainGr Image Classification and Analysis. In The Twelfih International Conference on Learning Representations, 2023.
[32] Sebastian Ramos, Stefan Gehrig, Peter Pinggera, Uwe Franke, and Carsten Rother. Detecting Unexpected Obstacles for Self-Driving Cars: Fusing Deep Learning and Geometric Modeling. In 2017 IEEE Intelligent Vehicles Symposium (IV), pages 1025-1032. IEEE, 2017.
[33] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.
[34] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16:1-85, 2022.
Haiyang
[35] Dawid Rymarczyk, Lukasz Struski, Jacek Tabor, and Bartosz Zieliński. ProtoPShare: Pro- totyical Parts Sharing for Similarity Discovery Interpretable Image Classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1430- 2021.
[36] Dawid Rymarczyk, tukasz Struski, Michat Górszezak, Koryna Lewandowska, Jacek Tabor, and Bartosz Zieliñssi.. Interpretable Image Classification with Differentiable Prototypes Assignment. In European Conference on Computer Vision, pages 35 351-368. Springer, 2022.
[37] Mikolaj Sacha, Bartosz Jura, Dawid Rymarczyk, Lukasz Struski, Jacek Tabor, and Bartosz Zieliński. Interpretability benchmark for evaluating spatial misalignment of prototypical parts explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 21563- 21573, 2024.
[38] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. arXiv preprint arXiv: 131.66444, 2013.
[39] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks. In International Conference Machine Learning, pages 3319-3328. PMLR, 2017.
[40] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficiemt image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[41] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going Deeper with Image Transformers. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 32 42, 2021.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. Advances in Neural Information Processing Systems, 30, 2017.
[43] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD birds-200-201 Dataset. Technical Report 10--T-2-2110001 California Institute of Technology, 2011.
[44] Chong Wang, Yuanhong Chen, Yuyuan Liu, Yu Tian, Fengbei Liu, Davis J McCarthy, Michael Elliot, Helen Frazer, and Gustavo Carneiro. Knowledge distillation to ensemble global and interpretable proiotype-based mammogram classification models. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 14-24. Springer, 2022.
[45] Chong Wang, Yuyuan Liu, Yuanhong Chen, Fengbei Liu, Yu Tian, Davis McCarthy, Helen Frazer, and Gustavo Carneiro. Learning support and trivial prototypes for interpretable image classification. Proceedings of the IEFF/CVE International Conference on Computer Vision, pages 2062-2072, 2023.
13

---

[46] Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing, Interpretable Image Recognition by Constructing Transparent Embedding Space. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 895-904, 2021.
[47] Thierry Warin and Aleksandar Stojkov. Machine learning in finance: a metadata-based system- atic review of the literature. Journal of Risk and Financial Management, 14(7):302, 2021.
[48] Sarah Wiegreffe and Yuval Pinter. Attention is not not Explanation. In 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Confer- ence on Natural Language Processing, EMNLP-IJCNLP 2019, pages 11 11-20. Association for Computational Linguistics, 2019.
[49] Yang Xu and Zuqiang Meng. Interpretable vision transformer based on prototype parts for COVID-19 detection. IET Image Processing, 2024.
[50] Mengqi Xue, Qihan Huang, Haofei Zhang, Lechao Cheng, Jie Song, Minghui Wu, and Min- gli Song. ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition. arXiv preprint arXiv:2208. 10431, 2022.
[51] Haoming Yang, Steven Winter, Zhengwu Zhang, and David Dunson. Interpretable ai for relating brain structural and functional connectomes. arXiv preprint arXiv:2210.05672, 2022.
[52] Haoming Yang, Pramod KC, Panyu Chen, Hong Lei, Simon Sponberg, Vahid Tarokh, and Jeffrey Riffell. Neuron synchronization analyzed through spatial-temporal attention. bioRxiv, pages 0024-07.40 2024.
[53] Tingyi Yuan, Xuhong Li, Haoyi Xiong, Hui Cao, and Dejing Dou. Explaining Information Flow Inside Vision Transformers uUing Markov Chain. In eXplainable Al approaches for debugging and diagnosis., 2021.
[54] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable ConvNets v2: More Deformable, Better Results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9308 9066,931616000 2019.
14

---

Appendix Table of Contents
A Training Hyperparameters
16
B Training schedule
16
C More discussion on the greedy matching layer
17
D More discussions on adaptive slots mechanism
17
E
Ablation studies
18
E.1
Class-token
18
E.2
Adjacency mask
20
E.3 Coherence loss
21
E,4
Ablated Misalignment.
21
Choices of K sub-prototypes
21
G Qualitative examples of robustness against perturbations
22
H More examples of reasoning process
25
I More examples of reasoning process for misclassification
30
J More examples of analysis
30
K Details on User Studies
30
Details on Car dataset
36
M Broader Impact
36
N Computational Cost
41
Training software and platform
41
15

---

A Training Hyperparameters
This section documents the specific hyper-parameters used train ProtoViT, shown in Table 4. We used identical parameter settings across all the backbones. We used the values suggested in prior work for terms that already existed, and used a small grid search select the other values. No dedicated tuning procedure is necessary for these coefficients.
B
Training schedule
This section documents the specific training schedule used to train ProtoViT. As discussed in Sec. 3.4, the training stages are generally divided into four steps as shown in Appendix Fig. 5. Training schedule settings can be found in Appendix Table 5. Warm-up optimization is performed by training the feature encoder with extremely small learning rate while training the prototype vectors for epochs. Then, we increase the learning rate for the feature encoder layer for the following 10 epochs. As mentioned in Appendix Sec. A, the number sub-prototypes being pruned during the slots pruning stage depend on the learning rate of the slots parameters and the weight of the coherence loss in the slots pruning stage.
Initialized ProtoVIT
Stage Optimizing layers
layer
Phase Warm-up training: train feature encoder with an extremely amall leaming maa
Phase 2 Join Optimization: train feature encoder with larger learning rate
Objective function:
lotal CE
+
cist
Cist
Sep
7.g2 Con + 2 24 Orto
Stage 2: Slots Pruning
Objective
L
L prune CE +
Cah
Stage Prototype Projection
Objective: Round the fractional approxlmation of Indicator integer (1/0) Project prototypes to the nearest feature tokens
Stage 4: Last Layer Fine-Tuning
Objective function:
L L
CE
Sparsity
Trained ProtoVIT
Figure The procedure for training Pio,oViT, The objectives for each stage are shown.
Table 4: Parameter Settings for Proto VIT
Parameter
Weight
Cross Entropy Weights
1.0
Cluster Loss
-0.8
Separation Loss
0.09
Li Norm Loss
x 10
Orthogonality Loss
1 x 10-3
Coherence Loss
x 10
Coherence Loss for Slots Pruning
10-5
Sigmoid Temperature
100
16

---

Table 5: Training Schedule for Proto ViT
Training Stage
Model Layers
Learning Rate
Duration
Optimization Warm-up Stage
feature encoder
1 x 10-7
prototype layer p
3 10
epochs
Optimization Joint Stage
feature encoder f
5 x 10-5
prototype layer p
3 x 10-3
10 epochs
Slots Pruning Stage Fine-tuning Stage
slots parameters
8 x 10-5
10 epochs
last layer
1 10-2
15 epochs
C More discussion on the greedy matching layer
Intuitively, a non-deformed prototype P, consists K non-overlapping sub-prototypes P with rigid adjacency (i.e., rectangular shape). These non-deformed prototypes are compared with K non-overlapping latent feature tokens x in the same geometric shape (i.e., in a rectangle). To deform such prototype, could treat each sub-prototype vector P as independent "prototype" to train. Then, the prototype P, naturally deformed into K' independent sub-prototypes P that move freely in the latent space. However, this naïve approach could lead to significant overlap among the sub- prototypes. Although this could be mitigated by incorporating a unit-wise orthogonality loss during the training phase encourage dissimilarity among the sub-prototypes, similar to the orthogonality loss outlined Eq. such unit-wise orthogonality loss is in conflict with the objectives of the coherence loss defined in Eq. 4, which encourages the sub-prototypes P to remain neighboring in cosine distance to preserve the semantic coherence of each prototype Py- Thus, an algorithm that could provide unit-wise and non-on non-overlapped matching is ideal, and the greedy matching algorithm with adjacency masking meets the needs. An summary of the algorithm for greedy matching with adjacency masking is shown in Alg. 1.
Algorithm 1 Greedy Distance Matching Algorithm
Require: Input latent feature tokens z, and prototypes p.
1: Compute cosine similarity between latent feature tokens and sub-protort cotbprrtotypes.
2: Initialize masks in shape of latent feature tokens, and sub- prototypes to include all possible pairs
3: for in K slots of sub-proe uub-prototypes do
4: Mask out non-adjacent and selected patches by replacing their similarity scores with a high
negative value
5: Identify the closest latent feature token for each remaining sub-prototype
6: Select the pair that has the highest cosine similarity out of all identified pairs
7: Update all masks based on the selected pair
8: Track the sequence of selected sub-prototypes for reordering
9: end for
10: Sort the pairs and the corresponding cosine similarity based on its original sub-prototype order
D More discussions on adaptive slots mechanism
As discussed in the Sec. 3.3, The adaptive slots mechanism consists of learnable vectors v E m and sigmoid function with hyper-parameter temperature The vectors are sent to the sigmoid function to approximate the indicator function as (Inciude }²} = Sigmoid r). Each (Incuuee p an approximation of the indicator for whether the k-th sub-prototype will be included in the j-th prototype P,- To be specific, define the approximated indication function i (Include pt) with temperature as:
[Include p.)
1 I+ IVE-
99)
By the range of the sigmoid function, we are able to approximate the indicator function with a high temperature т. Fig. 6 shows an example of different choices of r. As the temperature value goes up. the sigmoid function more closely approximates the behavior of the indicator function. Thus,
17

---

picked sufficiently large for approximation, and later rounded the values to the closest integer (i.e 1) to serve as the actual slot indicator for each sub-prue cult-potttypp.
Sigmold Function with Different Temperature Values
Temper dusto
Tempeneuue
Temperature
Tenperature
0.8
SgmeidM
0.6
Figure 6: Plot of sigmoid function with different choices of temperature value
E
Ablation studies
This section details ablation studies on the class token, coherence loss, and adjacency mask. We performed the ablation studies with backbone DeiT-Small with and K for the adjacency mask and number of sub-protote protootot respectively.
Table 6: Ablation study on Proto VIT. Model is compared with K 4, and r 1 for slots and adjacency mask settings. We use Deit-Small as the backbone, and the model is trained on CUB- 200-2011 1 dataset. For the current setting of Proto we included class token, coherence loss, and adjacency mask along with the greedy matching algorithm.
greedy matching
class token coherence loss adjacency mask
Accuracy [%]
PLC [%]
Yes
No
No
No
84.32 = 0.10
31.02 0,1
Yes
Yes
No
No.
85.54 + 0.20
35.88 0.8
Yes
Yes
Yes
85.13 + 0.17
32.98 0.6
Yes
Yes
No
Yes
85.45 = 0.30
31.67
0.2
Yes
No
Yes
Yes
84.12 + 0.05
17.83 1.5
Yes
Yes
Yes
Yes
85.37 = 0.13
21.68 3.1
E.1
Class-token
In this section, we performed an ablation study on the class token. In the ablated model Pit (patch tokens only), the latent feature tokens 2 are defined the latent patch tokens Zpatch. The class token Xclass and its corresponding component in position embedding Epus are excluded in training. As shown in Appendix Table. 6, incorporating the class token into the latent feature tokens, the performance of the model generally increased by 1% with and without the adjacency mask and coherence loss. Though observe drop in performance by excluding the class token and the corresponding component in the position embedding, the ablated model still achieves comparable performance the other prototype-based ViT's and the black-box backbone.
Subtracting out the class token may have interesting implications for the class specificity of our prototypes. expect subtracting the class token out operates as a kind of focal similarity, where each token (after taking the difference) represents the unique information available that position which relevant to the target class. Prototypes will, then, learn to represent unique positional information that is relevant to the predicted class. As such, we might expect prototypes
Saliency test:
18

---

Histogram of mean correct class activation for ProtoViT with and without class token
Count
Without Cia token (count)
With Cls token (count)
500
300
100
0.00
0.32
0.48
0.64
0.00
0.99
1.12
1.28
1.60
1.70
1.02
2.00
2.40
2.50
2.88
3.20
3.30
3.52
3.68
3.84
Prototype Activation
Histogram of the largest mean incorrect class activation for ProtoViT with and without class token
Count
token (count)
(cnunt)
800
600
200
1.00
1.24
1.36
1.60
12
1.84
1.96
2.20
7,32
2.44
2.56
2.80
2.92
3.04
16
3.28
40
3.52
3.64
3.76
3,00
Prototype Activation
Histogram of the difference between mean correct class activation and largest mean incorrect class activation for ProtoViT with and without class token
Without Cis token [count]
With Cis token (count)
Count 60C
200
50
3.24
2 98
-2 /2
2.46
-2 20
1 19
68
1242
90
64
12
14
0.40
06
1.18
14
1.70
1996
2.22
2.48
274
3.00
Prototype Activation
Figure 7: Histogram analysis of mean activation for Proto VIT with and without class token.
19

---

to be more tightly tied to their class than they would be if they were learning with simple arbitrary tokens. We thus expect an improvement in class-specific saliency of the prototypes.
To test if class token improved the saliency of prototype to its assigned class, we performed the analysis on correct and incorrect class similarity scores. If prototypes are more specific to their assigned class when including the class token, we would expectthe gap between the mean activation for prototypes of the correct class (summed cosine similarity scores as defined in Eq. 2) and the largest mean activation for prototypes of an incorrect class to be larger. To test this, we randomly pick a model with the class token and one without the class token and evaluate each model on the test set of CUB200-201 I [43]. For each test image, we average over the correct class prototypes activations, and denote this average as Seer for the model trained with patch-tokens only, and gels for the model trained with class tokens. Similarly, we denote the largest mean activations from the incorrect class prototypes as gels and gibl for the models with and without class token respectively. We use jols and gubl to denote the difference between the two measures for the two models. As shown in Appendix Fig. 7, there is clear shift in distribution between the two models for all three measures. When we include the class token, prototypes tend to activate more highly on the correct class, and the gap between the mean correct class activation and the highest mean incorrect class activation tends to be larger.
We further perform one sided t-tests to test whether goor gabl and gels مabl are significantly greater than O. As shown in Appendix Table. 7, including the class token statistically significantly increases each measure relative to a model trained with patch token only. That is, including the class token improves the saliency of encoded features.
E.2 Adjacency mask
As introduced in Sec. 3.3, the adjacency mask is designed to ensure that sub-prototypes are ge- ometrically adjacent. Without the adjacency mask, sub-prototypes are able to match with latent feature tokens from anywhere in the image.As indicated in Appendix Table 6, removing the adjacency mask typically results in a slight improvement in performance. This outcome is expected since prototypes are learned for the maximum possible performance with fewer constraints. However, removing the adjacency mask tends to damage the semantics of the model's prototypes. For instance, as demonstrated in Appendix Fig. 8, without the adjacency mask, the model might correctly identify the beak of a least tern in one image patch but erroneously relate other prototypical sub-parts to the bird's feet in another patch. Although the slim feet of the least tern indeed looks similar to the slim beak on the scale of the sub-prototype, such comparisons are not meaningful. In contrast, implementing the adjacency mask in ViT effectively prevents such issues by enforcing geometric adjacency among the sub-prototyper Furthermore, without the adjacency mask, the model may learn sub- uub-prototypes that, though visually similar to other sub-prototypes. are fundamentally different, introducing noisy representations and disrupting the coherence of prototypical features. For instance, as demonstrated in Appendix Fig. 8, both models misclassified test image of black tern pigeon guillemot which also has red feet. Without the adjacency mask, the prototypical feature capturing the red feet of the pigeon guillemot also mistakenly includes reflections of the red feet in the water. Although the water reflection looks similar to the actual red feet captured by the other sub- sub- protootyss,s this misrepresentation undermines the coherence of the feature representation. On the other hand, Proto compares the feet of the black tern to the red feet of the pigeon guillemot. Thus, the use of an adjacency mask is essential for preserving the coherence of prototypical feature representations.
Table 7: Results of the t-test over correct class activation with and without class token, and the difference between the correct and incorrect class activation with and without class token. The p-values for both tests are O.
Target
Mean 11.6 Std.
T-stats against O
Correct Class Activation
0.625 = 0.011
116.3
clss gabl )
Correct and Incorrect Class Activation Difference (8els -gabl
0.375 + 0.008
47.1
20

---

ProtoViT without adjacency mask: Why is this bird classified as a Least Tern?
ProtoViT:
Why is this bird classified a Least Tern? Image
Prototypes
Test Image
Prototype
Looks like
ProtoVIT without adjacency mask: Why is Ter classified Pigeon Guillemot?
Test Image
Prototype
ProtoViT:
Why is this Black Tern classified
Pigeon Guillemot?
Looks like
Test
Figure 8: How Pio without adjacency mask makes predictions (left) vs. how Proto ViT makes predictions. The test image of least tern correctly classified by the two models, and the test image of black tern is classified as pigeon guillemot by both of the models.
E.3 Coherence loss
As explained Eq. 4, coherence loss used to ensure that the sub-prototypes within each prototypical feature are semantically similar to each other. By design, this loss term helps aub-prototypes to collectively represent a coherent feature. By removing the coherence loss, the prototypical features would still contain diverse parts of features in one prototypical feature, similar problem that the non-deformed prototypes have, even though the sub-prototypes remain geometrically adjacent. For example, shown in Appendix Fig. 9, model trained without coherence loss tends to mix the wing and belly of the Myrtle Warbler into one prototypical feature, while it mixes the head and wing for the prototypical features for Painted Bunting. In comparison, training Proto ViT with coherence loss ensures that all the sub-prototypes collectively represent one prototypical feature, in this case containing the striped belly of the Myrtle Warbler, the wing and the red lower part of the Painted Bunting.
E.4 Ablated Misalignment
We computed the Prototype Location Change (PLC) for each ablated model using greedy matching algorithms. Following the PLC metric defined in Sacha et al. (2024) [37], we measured the shift in the 90th percentile of prototype activations across test images. As shown in Table 6, the combination of coherence loss and adjacent masks effectively reduces changes in prototype locations after adversarial attacks. This indicates that these mechanisms promote greater stability in the learned prototypes. Interestingly, incorporating the class token into the latent representations increases PLC. This is expected, the class token contains more global information, which, while improving overall model performance, leads to larger shifts in prototype locations.
F Choices of K sub-prototypes
Why K=4: By the design of the vanilla ProtoPNet 10] and the other non-deformed CNN based prototype models|46, 15, 26], the input images are encoded into latent features by CNN backbones, where 2 E 7 and d denotes the latent dimension varying by different choices of backbones. These models learn prototypical features of shape x x d from the latent features. On the other hand, ViT backbones such as CaiT and DeiT encode the input images into 14 14 x latent feature tokens. To be consistent with existing models, we design the model to learn prototypes of shape 2 x d from the R Mx14xd latent features, so that each prototype corresponds to the same proportion of the input image. Using the greedy matching algorithm, we move away from
21

---

ProtoViT without coherence loss:
ProtoViT:
Why is this bird classified as a Myrtle Warbler?
Why is this bird classified as Myrtle Warbler? Prototypes
Image
Test Image
Prototype
Looks like
ProtoViT without coherence loss: Why is this bird classifled as a Painted Bunting?
ProtoVIT:
Why is this Black Ter classified as a Painted Bunting?
Teat Image
Prototypss
Test Image
Prototype
Loail Bio
Looks like
Looks like
Figure 9: How Proto ViT without coherence loss make predictions (left) vs. how ProtoViT makes predictions. The given test image of Myrtle Warbler and Painted Bunting are correctly classified by both models.
learning fixed 2 x 2 rectangular prototypes and instead have adaptively shaped prototypes consisting of four uub-prototypes. It worth noting that prototype-based models with CNN backbones heavily rely on up-sampling for prototype visualizations. This process can introduce errors, leading to the of the top 5% most similar regions for visualization, which results in irregularly sized bounding boxes, as shown in the top row of Fig. 1. In contrast, with ViT backbones, we are able to visualize the exact image patches that the prototype projected to, and thus produce more precise and accurate prototype visualizations.
Other choices of K : Although we selected K= maintain consistency with other existing prototype-baset models, alternative values of K are also viable. The model performance for K and K = 6 settings can be found Appendix Table. 8. We observed that the model performs best with = 2 when K 5 is used, and with r = 3 when K = 6. As shown in the table, the models with different choices of K perform similarly to the setting with 1l show examples of reasoning process for ProtoViT(K 5, r = 2) and ProtoViT(L respectively. As shown in the figures, though having more cub-prototypes is helpful to capture larger featuires such as the tail of the Lincoln Sparrow and Forster Tern in Fig. 10 and Fig. 11 respectively, because of variations scale, many features such as the beak and the head in 11 do not always need that many sub- poo-pooottysss... Local and global analysis for Proto and ViTTo = 6, 3) are shown in Fig. 13 and Fig. 13 respectively. As shown the figures, having more sub-prototypes is advantageous in representing features that are large in scale such as the white belly of Sayornis shown in the second row of global analysis in Fig. 12, and brown pattern of Eastern Towhee in the bottom row of Fig. 13. Both the local and global analysis again show that the prototypes of Proto ViT have strong, consistent semantic meanings.
Appendix Fig. 10 and Fig.
6, 3)
5, 2)
It is worth noting that prior prototype-based models could not easily support prototypes with 5 or 6 sub-prototypes. Since methods like Deformable ProtoPNet treat prototypes as convolutional features, the only way to handle prototypes with 5 sub-prototypes would be to form each prototype by dimensional convolutional filter. This would allow the model to have sub- protttos,,,,s, but would enforce very strange shape prototypes (a horizontal line).
G Qualitative examples of robustness against perturbations
We provide several instances of the perturbation examples, in which we mask out the region selected by each prototype, as shown in Fig. 14. In each row, we mask out all matched locations for a prototype (middle-left column) using a white mask the black wings and a black mask the red parts,
22

---

Table 8: Proto ViT performance with = 5 and 6 using DeiT-Small backbone
# of sub-prototypes Adjacency mask range It
Accuracy [%]
85.37 0.13
85.33 + 0.20
85.26 + 0.15
Reasoning this bird being Lincoln Sparrow:
Reasoning for bird being Henslow Sparrow:
Reweighed Cumubative
Total weight
Rewsighed
Last
Total
Test Image
Prototypes
amilarity
Layer waight
Test
Prototypes
Cumudative simtarity
weight
Layar
seight
Lodk
3.07
0.980 3.00
3,11 X 1.016 3.16
3.05 x 0.900 2.99
2.16 1.029 2.22
Points
Total Sparrow: 26.0
Lincoin Sparrow:
Figure 10: How Proto ViT with = 5 and r using DeiT-Small backbone classifies test image of Lincoln Sparrow to the correct class (left), and the second most likely class Henslow Sparrow (right).
Reasoning for this bird being a Forster Term:
Reasoning for this bird being a Least Term:
Reweighed
Total weight
Rewnighed
Total weight
Teat Image
Prototypes
sinilarity
Layer
Teat Image
Prototypes
sinllarity
Layer
5.95
X 0.999
5.94
5.84
1.010 5.90
0.980 4.35
4,29 X 1.027 4.41
2.89 X 1.054 3.05
X 1.017 4.24
2.30 1.038 2.39
Total Points to Forster Term:
Total Points to Least Tern:
38.5
27.9
Figure 11: How Proto with K = 6 and 3 using DeiT-Small backbone classifies test image of Forster Tern to the correct class (left), and the second most likely class Least Tern (right).
and check where that prototype activates after masking (shown in the leftmost column). We then confirm that the activated region for other prototypes remains reasonable when the mask from another
23

---

Nearest Protokypes images boundinboo)
Prototype bounding bod)
Nearest Training patches to the prototype (in beussiing box)
Neareet petches the prototype (In besnding
Original bext Image
Warbier
Figure 12: Examples of local analysis (left) and global analysis (right) of PiotoViT with DeiT-Small backbone with K 5 and
2.
Nearest Prototypes Images
Prototype besending box)
Nearest Training patches to the prototype (n hounding bea)
Nearest patches the prototype (nn boundingas)
besendegghhe hox)
Original tast Image
Figure 13: Examples of local analysis (left) and global analysis (right) of ProtoViT with DeiT-Small backbone with K 6 and r 3.
24

---

prototype is upplied (right two columns), We observed that after removing the preferred region by each prototype, activates on another reasonable alternative (e.g.. red belly prototype might activate on red back as second choice).
Experiment on masking prototype activation area:
Modified Image
Target Prototype Modified Image
Other Prototype
Looks
Looks
Looks like
Looks like
c.
Looks like
Looka like
Prototype location change experiment similar Location Misalignment
Benchmark figure by altering background color:
Test Image
Prototype
Observation made in Location Misalignment Benchmark paper Prototype based model
with CNN backbones:
orgiat imuge
mod Bed mmago
Prototype activation: 3.835
prololiioa 2nd
Modifled Image
Prototype
Prototype
similerty
Figure 14: Perturbations (top) and background change (bottom). As shown the top, our prototypes identify reasonable alternative regions on which to activate after masking the originally most activated regions. As shown in the bottom, altering the background does substantially impact the activation location of our ViT-based prototype. This shows that our Proto VIT has a better location alignment than the CNN-based prototype models.
H More examples of reasoning process
This section provides more examples for the model reasoning process. Fig. 15, Fig. 16, and Fig. 17 demonstrate more examples of reasoning process of ViT with Deil-Small backbone. Fig. 18, Fig. 19 and 20 are the examples of reasoning process of ViT with CaiT-backones. Fig. 21, Fig. 22 and Fig. 23 are the examples of reasoning process of Proto VIT with Deit-Tiny backbone. In each case, again intuitive reasoning from VIT.
25

---

Reasonlng being Mourning Warbler:
Reasoning for bird being a Nashville Warbler:
Reweighed
Reweighed sinilarity
Test
similarity
weight
weight
Protolypes
seight
Lide
3.45 X 1,013 3.49
2.72 X 0.388 2.42
Looka
3.18 X 1.015 3.22
2.42 X 0.884 1.98
Total Points to Mouriing Warbler: $0.4
Total Points to Nasivville Warbler: 19.6
Figure 15: How PiT with DeiT-Small backbone classifies a test image of a Mourning Warbler the correct class (left), and the second most likely class Nashville Warbler (right).
Reasoning being Myrtie
Reasoning for
being a Warbler:
Reweighed
Rownighed sinilority
Test
Prototypes
similarity
weight
weight
Test
Protolypes
weight
Lodka
3.70 X 0.908 3.36
2.53 X 1.027 2.60
2.44
Lodks
3.49 X 0.903
3:15
2.25 X
1.821 2.30
1.84
:
Total Points to Myrtla Warbler:
Total Points to Palm Warbler:
31.5
19.4
Figure 16: How Proto VIT with DeiT-Small backbone classifies a test image of Myrtle Warbler to the correct class( left), and the second most likely class Palm Warbler (right).
26

---

Reasoning being an Eared Grebe:
Reasoning for this bird being a Horned Grebe:
Reweighed similarity
Reweighed
Test
Prototypes
weight
Test
Protolypes
sinilarity
seight
Londa
3.54 X 0.983
3.48
1.93 X 1.014 1.96
1.67
Looka
3.46 X 0.963
3.40
1.65 X 1.013 1.87
X
Total Points to Eared Grabe:
Total Points to Homed Grebe: 15.8
33.7
Figure 17: How Pit with DeiT-Small backbone classifies a test image of an Eared Grebe to the correct class (left), and the second most likely class Horned Grebe (right).
Reasoning being an Common Tem:
Reasoning for being Aratic Tern:
Reweighed
Test
Prototypes
similarity
weight
weight
Test
Protolypes
Cumilatie sinilority
weight
Lodi
3.66 X 0.983 3.63
3.55 X 0.946
3.36
2.51
Lodks
Looksiee
2.26 X 0.994
2.25
2.34 X
0.95 2.22
2.05
:
Total Points to Common Torm:
Total Points to Arctic Torn:
24.8
24.2
Figure 18: How ViT with CaiT-xxs backbone classifies a test image of a Common Tern to the correct class (left), and the second most likely class Arctic Tem (right).
27

---

Reasonlng being an Lincoin Sparrow:
Reasoning for bird being Henslow Sparrow:
Reweighed
Reweighed
Test Prototypes
similarity
weight
Test
Protolypes
similarity
seight
3.74 X 0.839 3.14
3.45
X 0.843 2.90
Looka
5.15 X 0.841 2.65
2.99 X 0.854 2.55
Total Points to Lincoln Sparrow: 25.9
:
Total Points to Henslow Sparrow: 22.0
Figure 19: How Proto ViT with CaiT-xxs backbone classifies a test image of a Lincoln Sparrow to the correct class (left), and the second most likely class Henslow Sparrow (right).
Reasoning
being Mockingbird:
Reasoning for being a Western Pewee:
Reweighed similarity
Test
Prototypes
weight
weight
Test
Protolypes
Cumilative sinilority
weight
Lodk lhe
Looks
3.89
0.984 3.83
3.20
X 0.940 3.01
Lodks
3.25 X 0.988
3.21
2.53 X 0.940 2.38
3.20
Total Points to Mockingbird:
:
Total Points
30.6
Westemn Wood Pawes: 19.7
Figure 20: How Proto with CaiT-xxs backbone classifies test image of Mockingbird to the correct class (left), and the second most likely class Western Wood Pewee (right).
28

---

Reasoning for this bird being Scarlet Tanager:
Reasoning for this bird being a Summer Tanager:
Reweighed similartty
Total Image
Rewsighed
Test
Prototypes
weight
Prototyses
sinflaity
Looks
Looks
3.65 X 0.096 3.27
3.71 X 0.859 3.19
Looks
3.33 X 0.809
2.96
2.90 X 0.868
2.52
Looka
Total Points to Scarfet Tanager:
:
Total Points to Summer Tanager:
29.6
25.6
Figure 21: How Proto VIT with Deit- Tiny backbone classifies test image of a Scarlet Tanager to the correct class (left), and the second most likely class Summer Tanager (right).
Reasoning for this bird being Pied Kingfisher:
Reasoning for this bird being White-breasted Kingfisher:
Reweighed
Last
Total weight
Reweighed Cumilative
Lest
Tosal
Test Image
Prototypes
Camulative
Layer
Tost Image
Prototypes
Layer
weight
3,97 X 0.965 3.51
3.63 X 0.904 3.28
Lodka le
Looks
3.56
0.884 3.15
3.50 X 0.920 3.22
3.52 X 0.887 2.94
3.32 X 0.954 3.17
Lodka like
2.86
Lookslie
3.08 X 0.953 2.94
Total Points to Pied Kingfisher:
Total Points to
29.5
White-breasted Kingfisher: 20.9
Figure 22: How Proto with Deit-Tiny backbone classifies test image of Pied Kingfisher to the correct class (left), and classifies test image of a White-breasted Kingfisher to the correct class (right).
29

---

Reasoning for this bird being Parakeet Auklet:
Reasoning for this bird being a Baltimore Oriole:
Beweighed Cumulative
Last
Total
Reweighed Cumuiaaive
Last
Tosal Hoiht
Teat Image
Prototypes
Layer
weight
Image
Prototypes
Layer
3.97
X 0.894 3.55
3.96 X
1.08
4.27
Lodo
Loebs
3.01
3.58 X 1.09 3.90
3.58 X 0.931 3.53
3.19 X 1.12 3.57
3.46 X 0.968
3.45
Lobil
1.19
1:11 1.32
Total Points to Parakeet Auklet:
Total Points to Baltimore Oriole: 30.8
31.7
Figure 23: How ProtoViT with Deit- Tiny backbone classifies test image of Parakeet Auklet the correct class (left), and classifies a test image of Baltimore Oriole to the correct class (right).
I More examples of reasoning process for misclassification
In this section, provide the reasoning process of how model misclassified test image of summer tanager and a slaty backed gull in Fig. 24. We found that the misclassification the given summer tanager example may be because of data mislabeling the original dataset. A summer tanager does not have a black colored wing- That test image should indeed belong to the scarlet tanager class the model predicted. Similar mislabeling cases also happen for Red Headed Woodpecker with image ID Red__eee Headed_ Woordpecker_One Sttpppecceer00188...11 00018118455 183455 and Red_ _ Woordpecker_(N0 L..00000003000 as we found when randomly selecting examples to present for the paper. On the other hands, misclassifications can be contributed by the similarity between different classes. As shown in the bottom of Fig. 24, the West Gull looks very similar Slaty Gull. And the model's reasoning process indeed shows that the model believes that these two classes are the top two most likely classes for prediction.
These examples showcase the ability our method to help us understand the reasoning process of the model, not just when it is right, but also when it wrong.
More examples of analysis
This section provides more examples for the local and global analysis. Fig. 25, Fig. 26, and Fig. 27 are the examples for analysis for Pio with Deit-Small, CaiT-xxs24 and DeiT-tiney backbones respectively. The visualizations demonstrate that the prototypes of Protovit exhibit consistent, strong semantic meanings across different VIT backbones.
K Details on User Studies
This section provides details on the user study we conducted. Through our user study, we show that Proto improves both the clarity of reasoning process and coherence of prototypical features relative to ProtoPNet[10] and Deformable ProtoPNe([13]. We randomly selected 10 bird images from test set, and show the comparison with the three most similar prototypes from top-1 predicted class of the three models. We then presented the test to 10 participants, all of whom attend college in the U.S. and have some background in Machine Learning. We instructed participants to rate how well they understood the models' reasoning process through the presented examples,
Overview:
30

---

Reasoning for this Summer Tanager_aaaaa 1070_137714 is misclassified as Scarlet Tanager:
Text Image
Prototypes
Conristive Lajer wiiht wight
sendarty
Gursdative Lager wight amitartty
neight
Tanager:
3.89 X 0.920 3.58
Looaa Ba
3.85 0.919 3.54
Text linige nureple,. of Sunvesar Tanager:
Lodka
3.77 X 0.919 3.46
3.76 0.920 3.46
The soaal poket to Scaret Tarager: 35.59
Note: The Surveer Tanageria the thasd theel dres sa
Reasoning for this Slaty Backed Gull is misclassified as Western Gull:
Tool
Text Image
Prototypea
Consrntive Lajer wiight wight
sendarty
Prototypes
smitartty weight
Ourndetive Lager wight
Bocked Gull:
2.74 X 1.064 - 2.92
Looa
2.72 1.066 2.90
Text lesige nocepple of Wasdem
Looia Ba
X 1.006 2.42
2.08 1.074
The sotal poket to Staty Backed Gullt 24.37
Note: The Wastem Gull second livaly class as they look very
Figure 24: Misclassification examples. Reasoning process of how our model misclassified a test image of a summer tanager (top) and a slaty backed gull (bottom). The top misclassification due to mislabeling.
and asked them to rate their confidence in their evaluation. Intuitively, the user should easily be able to distinguish which feature the model is comparing to, if the model has superb clarity. At the end of the experiment, we asked participants to rank the three models based on overall clarity of reasoning as well as coherence of prototypical features Iearned. Coherence of feature refers when the visualization represents one, and only one feature. Some sample questions are shown in Appendix. Fig. 28. Although this task does not pose any risk to the participants, they were nevertheless informed of their rights, and were asked for consent before data collection. The participants took on average 10 20 minutes to complete their assigned tasks, and were compensated $30 per hour.
To quantify the user study result on model clarity, we assign a score from 4 to I as the participant rated their understanding of the model's reasoning from best to bad. Similarly, assign a score from 4 to for the confidence rating from completely confident to not confident. We then multiply the confidence score and the understanding score to have the total score the clarity of the model reasoning. The maximum total score for a question is 16, which indicates that the participant has very clear and confident understanding of the model's reasoning. The minimum total score for a question is 1, which indicates that the participant does not understand the model's 's reasoning at all. As shown in Table. 9, average, the participants believe that they understand the reasoning process of Pit the best with the highest confidence. The result of the one-sided t-test on understanding score and total score further illustrate that there is a statistically significant improvement in model clarity of PiT to the vanilla ProtoPNet. Fig. 29 shows the result of participants ranking on the three models based on coherence of prototypical features and the clarity of the reasoning process. Again, Pit is mostly ranked as the best in providing coherent prototypical features and clear reasoning process. It is worth noting that the majority of the participants commented that the prototypical features by ProtoPNet are usually too broad to understand what specific part the model looking at. Moreover, the prototypical features by Deformable ProtoPNet are less accurate in pinpointing the specific parts. On the other hand, the prototypical features by Protovit can provide more accurate and specific visualizations. It easy to tell if the prototypical features are representing the head of birds or the feet of the birds.
Result:
31

---

Nearest Protolypes to lest images beunddggbea boa)
Prototype bounding bos)
Nearest Training petches prototype (n hounding boo)
Neareat Test petches the prototype bounding
Original
Mackagiind
Nearest Prototypes to test images beunddogihen box)
Prototype bounding bes)
Nearest Training patchas prototype bounding
Nearest Test patches the bourdingbai
Original image
Maci
Werber
Figure 25: More examples of local analysis(left) and global analysis (right) of Pro with DeiT- small backbone.
32

---

Nearest Protolypes to lest images becnddigibea boa)
Prototype bounding bos)
Nearest Training petches to (nn bounding boo)
Neareat Test petches the prototype Inn bounding
Original
Fynncher
Maccard Warber
Nearest Prototypes to test images besnndiggieee box)
Prototype bounding boo)
Nearest Training patchas prototype bounding
Nearest Test patches prototype
Original image
Magal Tam
Figure 26: More examples of local analysis (left) and global analysis (right) of Pio with CaiT-xxs24 backbone.
33

---

Nearest Protolypes to lest images bsseddggbba
Prototype bounding bos)
Nearest Training petches to prototype hounding
Neareat Test petches the prototype bounding
Original
heige Burting
Nearest Prototypes to test images beundiogihen box)
Prototype bounding beot
Nearest Training patchas prototype bounding
Nearest Test patches prototype Inn bounding
Original Image
Madives
Figure 27: More examples of local analysis (left) and global analysis (right) of ProtoViT with DeiT-Tiny backbone.
34

---

tepetinuge
Fastura
Looker
well gurunderrarddyg undentard model remening:
monditin meauing
ki contor about undintendiig ring of aove feras rorner in messuiin proois
Model
Model
Pronehpica
Irput
your understeed my trp. undertaand ienzoiing:
2 Seel
4 Ja0
Yoar wonh Somee in mode's mater for the ohove Model
Compaes elaer oe moce tee monsoveiinn mesoeiie seores
condert doad risato aboo seserionn mosurmy
abou tandrn divs febo da poosss
can manoar Micuu teaetion ressting proves.
Please rankinng the modets based the ocherenceje g. one and only festure)
Oet tther Hurrge lovee. for Whre iino....
Bod Spen 2
Model 1
Photetypical
Inpult
Lookss lloe
Hon well your undentanding jug. you undder sodeddddddddddd the model ressoning::
Bnc
Your confidence models procictton above Model:
Terty confocr ubout my undertee dartanding of suboo Nallo vortsmaatin reaiinaion and resur 90
eocure my unde stuncing of aboots fnaur
process.
Model
Photodyprcol
Impurt Imagg
Feoture
Fraut Image
How well your understonding ing. Fyeu ander the model reasotingt:
How well your understanding (eg. you understand She reockt reasorengg:
Meod
C. das
Your confitionce the modelss presistion the above Model:
sbouiee drng siere
uy of reprascal and coaiiag Het ciboul aoreaer Notur
Park owre unwe darningg ou diee reprosed den rooneaa O candeer soo undiaiiiig vos tuse vuntaee recone yroess
Please rarking the models based on the colerancaee ore and only ore feature) and clar byleg. the model reasoning cleary.
Cen dos Avage Norc
Figure 28: Example of the survey questions. The examples are randomly selected, and the prototype with the highest similarity score for each example selected.
35

---

User study result of model ranking by coherence and clarity
worst
average
best
100
75
50
28
ProtoViT
ProtoPNet
Deformable ProtoPNet
Models
Figure 29: User Study result on model rankings based on its clarity and coherence
Table 9; User Study results model clarity. We perform one-sided t-test on the total score and understanding score of ProtoViT and ProtoPNet.Tho result shows that Proto has a statistically significant improvement in clarity of the reasoning process to ProtoPNet.
Model
Mean understanding score Mean confidence score
Mean score
Deformable ProtoPNet [13]
1.86
3.15
ProtoPNet [10]
2.68
Pio
3.85
3.47
13.42
T-Test
Mean + 1.96 std
T-stats
P-value
ProtoVil total score
ProtoPNet total score
4.75 0.894
10.37
2.16 x x10-20
ProtoViT Understanding score
ProtoPNet Understanding score
1.17 +: 0.209
10.89
5.75 x x100 x10-22
L Details on Car dataset
This section provides details on the implementation of ProtoViT on a second dataset. The Standford Car dataset [24] contains 81 44/8041 images for training and testing from 196 different car models. We performed similar offline data-augmentation as described in Sec. 4.1. After augmentation, the training set roughly 1,600 images per class. We assigned the algorithm to choose 10 class-specific prototypes for each of the 196 classes. Each of the prototypes is composed of 4 sub-prototypes. We kept the training schedule and hyper-parameters same as on the bird dataset. The specific training schedule and hyper-parae parameerrssserss settings are documented in Appendix Tables 5 and 4 respectively, Fig. 30 and Fig. 31 show examples the reasoning process for the DeiT-Small backbone. Fig. 32 and Fig. 33 show examples of the reasoning process for the CaiT-xxs-22 backbone. Fig. 34 and Fig. 35 show examples the reasoning process for the DeiT-Tiny backbone. Examples of global analysis and local analysis for Proto VIT with different backbones are shown in Fig. 37, Fig. 36, and Fig. 38 respectively.
M Broader Impact
Interpretability is essential ingredient trustworthy Al systems. Our work successfully integrates one of the most popular and powerful model families into prototype-based networks, which are one of the leading techniques for interpretable neural networks in computer vision. Our technique can
36

---

Reasoning for this car being a Mercedes Benz S-Class
Reasoning for this bird being a Mercedes Benz E-Class
Sedan 2012:
Sedan 2012:
Reweighed similarity
Total
Reweighed similority
Test Image
Prototypes
Test Image
Prototyses
weight
weigh
Looka
Lookelde
3.96 X 0.908 3.61
3,96 X 0.961 3.81
Lock
1.071
2.84 x 1.001 2.84
1.80 X 1.104 1.99
Morceliaa Benz
Mercedes Benz E-Closa Sodaa 2012:
B-Cluss Sodan 2012:
26.9
19,8
Figure 30: How Proto ViT with Deit-Small backbone classifies test image of a Mercedes Benz S-class Sedan 2012 to the correct class (left), and the second most likely class Mercedes Benz E-class sedan 2012 2 (right).
Reasoning
being a 2007:
Reasoning for bird being a 2012:
Total weight
Test
Prototypes
Cumubative similartty
Layer
Test
Prototypes
Cumulative simtuarty
Layer weiht
weight
weight
Lodka lhe
Looks
3,99
1.041
4,15
3.98 X 1.052
4.18
2.97 1.041 5.09
1.90 X 1.193 2.36
Looko lhe
Looks
2.65 X 1.189 3:13
1.58
X 1.196 1.87
1.164 2.99
1.05
1.238 1.38
: Total Points BAW ks suv 2012:
Total Points to GARN XG
SUV 2007:
25.7
19.5
Figure 31: How Proto VIT with Smill backbone classifies a test image of a BMW X5 SUV 2007 to the correct class (left), and the second most likely class BMW X6 SUV 2012 (right).
37

---

Reasoning for this car being an Audi R8 Coupe 2012:
Reasoning for this bird being an Audi S5 Coupe 2012:
Reweighed similarity
Tota weight
Reweighed sinilority
Test Image
Prototypes
Prototyses
weggh
Looku
Looks
3.80 X 0.985 3.78
2.25 1.078 2.43
X
2.09 x 0.987 2.85
1.91 X 1.110 2.12
Looks
Avdi RS
Audi 85
Coupe 2012:
27.2
Coupe 2012:
20.7
Figure 32: How Proto with CaiT-xxs24 backbone classifies test image of an Audi R8 Coupe 2012 to the correct class (left), and the second most likely class Audi S5 Coupe 2012 (right).
Reasoning for this car being a Lamborghini Gallardo 570-4 Superleggera 2012:
Reasoning for this bird being Ferrari 458 Italia Coupe
2012:
Reweighed
Last
Total
Rewsighed
Lost
Tosal weight
Test Image
Prototypes
Layer
wright
Test Image
Prototypes
Layer
siniiarity
3,62 X 1,051 5.80
2.15 1.093 2.35
X
Loda lee
Looks
3.35
1.078 3.61
1.69 X 1.061 1.79
3.52
1.056 5.51
1.69 X 1.047 1.77
Lodka le
2.82
1.029 2.90
Looksibe
1.57
1.109 1.74
Gallando LP S70-4
Superleggera 2012:
Lamborghhne
29.9
Feraal 2012:
16.8
Figure 33: How Proto VIT with CaiT-xxs24 backbone classifies test image of Lamborghini Gallardo LP 570-4 Superleggera 2012 to the correct class (left), and the second most likely class Ferrari 458 Italia Coupe 2012 (right).
38

---

Reasoning for this car being an Audi 85 Coupe 2012:
Reasoning for this bird being a Audi A5 Coupe 2012:
Reweighed similarity
Total weight
Reweighed sintiorit
Image
Prototypes
Prototyses
weggt
Looka
Looke
3.96 X 0.806 3.53
2.73 X 1.060 2.89
3.11 X 1.341 4.17
2.14 X 1.243 2.66
4
Points Audi B5
Ausi As Coups 2012:
23.8
Coupe 2012:
20.1
Figure 34: How Proto VIT with Deit- Tiny backbone classifies test image of an Audi S5 Coupe 2012 the correct class (left), and the second most likely class Audi A5 Coupe 2012 (right).
Reasoning for this car being Rolls-Royce Ghost Sedan Reasoning for this bird being Rolls-Royce Phantom
2012:
Sedan
Reweighed
Last Layer
Total wright
Rewsighed sinilarity
Lost
Tosal weight
Test Image
Prototypes
Test Image
Prototypes
Layer
3,68 X 1,260 4,64
3.40 X 1.018 3.46
Loda
Looks
3.46 0.899 3.11
3.11 0.907 2.82
2.75
1.189 5.27
2.33 X 1.114 2.60
Lodka lbe
2.60
1.359
3.51
Looks
2.16 1.284 2.79
Chost 2012:
Rolle-Royce
27.3
Rolis-Roye Phanton Sedan 2012:
21.7
Figure 35: How Proto ViT with Deit-Tiny backbone classifies test image of Rolls-Royce Ghost Sedan 2012 to the correct class (left), and the second most likely class Rolls- Royye Phantom Sedan 2012 (right).
39

---

Nearest Protolypes images boundepber
Prototype je bounding boo)
Nearest Training patches to the prototype (n beassing tex)
Neareet petches the prototype In besnding
Original
bet Image
Dnee
Suyda Canda
Serm
Figure 36: Examples of local analysis (left) and global analysis (right) of ProtoViT with CaiT-xxs24 backbone on the Stanford Cars dataset.
Nearest Prototypes Images
Prototype je bessedtngg box)
Nearest Training patches to the prototype (I hounding bea)
Nearest patches the protatype (In boundingann bsai
tesendegghe) box)
Original
tast Image
2007
Figure 37: Examples of local analysis (left) and global analysis (right) of Proto ViT with Deit-Small backbone the Stanford Cars dataset.
40

---

Nearest Prototypes to test images Ii boardingbel
Prototyps ja bounding boo)
Nearest Training patches to the prototype In tounding too)
Nearest Test patches the prototype In tounding
801
Figure 38: Examples of local analysis (left) and global analysis (right) of Proto VIT with Deit-tiny backbone on the Stanford Cars dataset.
be used for important computer vision applications discover new knowledge and create better human-Al interfaces for difficult, high-impact applications.
N Computational Cost
By introducing greedy matching and coherence loss, we do not observe a significant increase the computational cost. On the other hand, the computation of the adjacency mask may rely the power of CPUs, as involves more of matrix broadcasting and iterations. On 13th Gen Intel Core(TM) i9- 19-13900KF CPU, it takes 0.2 seconds to compute each iteration with batch size 128, and 1 with 2000 prototypes. Overall, it takes roughly hours to train the model with DeiT-Small backbone on the bird dataset, which is similar amount of training time ProtoPool [15] and TesNet [46] using similar amount of backbone parameters.
Training software and platform
We implemented our Proto using Pytorch. The experiments were run on NVIDIA Quadro RTX 6000 (24 GB), NVIDIA Ge Force RTX 4090 (24 GB) or NVIDIA RTX A6000 (48 GB).
41

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The experiments and results to support the claims in the introduction and abstract are all in the main paper and the appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the main paper Sec. 5 Limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
42

---

Justification:
There is no theoretical result and proof involved.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The training schedule and hyper-paramette are listed in Appendix Sec. B and Sec. A. The data information and augmentation are described in the Main and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
43

---

Answer: [Yes]
Justification:
The data are public, and code is provided in the github link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. wwtppcccpppppeeeccccccccccccceee / swisssoliccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We have listed the settings in the appenix for paramtere settings and training schedules. The data splits are decribed under each case study section in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
All of our results are listed with error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
44

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources of experiments are documented in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htthiisstttt///////eee/e http:/::///eepppeeeeeeeeee htt.rrpp...pppblccccccceeeeieeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impact discussed in the appendix and shortly discussed in conclusion section in the main paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
45

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:[NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They properly cited and credited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
46

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: the new model introduced in the paper is well documented. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
The paper includes user study and example questions included.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The details of the user studies, including the consent process are described in Appendix Sec. As the risks to the participants are minimal, we were exempted by our institution's IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
47


Reviews:
[{"review_id": "BQIyn5yJay", "reviewer": "Reviewer_dybc", "summary": "The paper presents ProtoViT, a method for interpretable image classification. ProtoViT incorporates ViT backbone with deformed prototypes that explains its predictions. ProtoViT consists of three components: \n\n(1) a feature encoding layer with a pre-trained ViT backbone, which computes a latent representation of an image; \n\n(2) a greedy matching layer which compares the latent representation to learned prototypes; and \n\n(3) an evidence layer which aggregates prototype similarity scores into a classification using a fully connected layer. \n\nQuantitatively, ProtoViT achieves better performance than previous prototype-based methods. Qualitatively, it identifies meaningful prototypes to explain the prediction.", "questions": "Please see Weaknesses Section.", "limitations": "Authors did provide limitations but a far-fetched one that does not really talk about the limitations of the current method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and illustrates the technical details. The qualitative visualization are helpful in understanding the method. Attached appendix provides a lot of meaningful details.\n\n\n2. The overall scheme of interpretable image classification is nicely presented with small patches!", "weaknesses": "1. It is understandable that one goal of interpretable image classification is to tell why something works. However, a bigger goal is to understand why something didn't really work. Is it possible to highlight the examples where the method could not predict the correct class? It would be then interesting to understand the reasons for failure.\n\n2. The method relies on a strong assumption of a solid backbone model that can already achieve a good performance on the given task. This restricts the applicability of the method to limited scenarios where we do not need interpretable image classification at the first place. In such a scenarios, the nearest neighbors obtained using latent feature representation and pixel correspondences may be sufficient.\n\n3. Limitations of the method is not clear from the paper. I could guess at certain places in the method section where things could go wrong. It would be better if the authors could illustrate those points with suitable qualitative analysis.\n\n4. User study is not properly designed and is not statistically significant.\n\nOverall: The paper is interesting. In a first reading, everything looks good. But then you start asking yourself questions about the different scenarios where the method will not work (and there are definitely such scenarios as can be judged from quantitative evaluation and user study) and why it will not work. The paper falls short in explaining them with details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We are glad that you find our paper interesting, well-written and nicely presented. We address your points below:\n> ... Is it possible to highlight the examples where the method could not predict the correct class? It would be then interesting to understand the reasons for failure. \n\nYes! Per your suggestion, we present examples of this in the shared pdf response. Interestingly, we found that some of the time where the algorithm appears to have gotten it wrong, it was actually because the dataset was mislabeled and the algorithm was actually correct (see the first example in Figure 1 of the pdf rebuttal). Appendix Fig. 8 also includes an example when the model misclassified Black Tern and Pigeon Guillemot. And this misclassification is consistent across different model algorithms because of the red feet that both birds have. We will include these additional examples in the revised paper.\n\n> The method relies on a strong assumption of a solid backbone model that can already achieve a good performance on the given task. This restricts the applicability of the method to limited scenarios where we do not need interpretable image classification at the first place. In such a scenarios, the nearest neighbors obtained using latent feature representation and pixel correspondences may be sufficient.\n\nThis is an important question that needs to be clarified. Nearest neighbors is not actually sufficient. First, calculating nearest neighbors for a full dataset is extremely expensive, which means training is extremely difficult since you’d have to calculate nearest neighbors at each iteration (or at least at many iterations); this makes it impractical for any complicated task.\n\nSecond, nearest neighbors are much more sensitive and difficult to troubleshoot. With prototypes, a person can look at all the prototypes and audit them, whereas you can’t do that with nearest neighbors for all the points. As we mentioned in our response to your first question, we found points that are mislabeled - if those were learned prototypes, we would have pruned them or relabeled them. We wouldn’t be able to check all those examples with nearest neighbors.\n\nThird,  we argue that the method is not limited to scenarios where we don’t need\n interpretability. A good example is  IAIA-BL, which uses prototypes for analyzing breast lesions in mammograms [1]. We **definitely** need interpretability here, as the stakes are high. Additionally, even when a black box model performs well, it doesn’t necessarily mean that it learns well. The example in our ablation section Appendix. Fig. 8, shows a version of our model that has a higher performance than the original; however, it is clear that the model confused the beak in the test image with the feet in the prototypes. \n\nFor applications that are high stakes, having an interpretable network allows us to troubleshoot the data and the model so that we can improve accuracy **above and beyond** the black box. The black box is just a good starting point for training our interpretable networks. In fact, we show that our network already has better performance than the black boxes, including the ViT-based and CNN-based models (see Table 2 of the main paper).\n\n> Limitations of the method is not clear from the paper...\n\nIn addition to what is discussed in the paper, the following limitations could apply to our method:\n\n1. Even though our method is comparable with other prototype models in terms of training speed, we should note that all prototype models take additional time compared to their black box backbone.\n\n2. Location misalignment is also a common problem in CNN-based models. As we discuss in our global response, this issue is still present in our model, but to a much lesser extent.\nWe will add these to the revised paper. \n\n> User study is not properly designed and is not statistically significant.\n\nWe apologize for the confusion. The user study results are actually significant at the alpha=0.05 level (all our p-values for the t-tests are below 0.05), as shown in Table 8 of the appendix. \n\nAs for the design of the user study, we followed a similar structure of the agreement task to that of the HIVE [2] paper, which is known for the quality of its human-studies experiments. However, we would be happy to consider suggestions to improve our user study.\n\n> Overall: The paper is interesting. In a first reading, everything looks good. But then you start asking yourself questions about the different scenarios where the method will not work ... The paper falls short in explaining them with details.\n\nWe thank the reviewer for their kind words regarding our paper. We hope that with additional clarification to the manuscript based on your and other reviewers’ comments, we have provided more detail about our method and its benefits and potentially downsides. \n\nWe believe that our method makes it easier to audit and understand models and that the prototype-reasoning gives us information on how to solve the problem, whether it is by augmenting the data to learn a specific concept. Importantly, it also tells us when we should not trust the prediction, as demonstrated above. Through comprehensive experiments and ablations, we have shown our method to perform reliably well at this task and is a strict improvement over other prototype networks as its localized parts are totally faithful to the reasoning process (since we do not require upsampling for our visualizations, something that other prototype based methods require).\nWe thank you for your comments and hope we have addressed them all. Please let us know if there is anything you would like us to further clarify.\n\n[1] Barnett, Alina Jade, et al. \"A case-based interpretable deep learning model for classification of mass lesions in digital mammography.\" Nature Machine Intelligence.\n\n[2] Kim, Sunnie SY, et al. \"HIVE: Evaluating the human interpretability of visual explanations.\" ECCV 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3eIPE47A71", "reviewer": "Reviewer_w1UG", "summary": "This paper presents a novel strategy to learn interpretable visual prototypes for visual transformers, with a good property of offering spatially deformed prototypes. The method also introduce an slot mechanism which can learn an adaptive number of prototypical parts. The proposed are wisely designed for visual transformer architectures.", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. It is nice to draw inspiration from the focal similarity when computing the  patch features.\n2. Different from Deformable ProtoPNet, the proposed method present a new way to accommodate geometric variations of objects.  \n3. The proposed method is validated on two benchmarks and with extensive ablation studies. \n4. In general, the paper is well written.", "weaknesses": "1. The first concerns is about the use of deformable prototypes that containing K sub-prototypes, which means the proposed method will have more learnable prototype vectors (K times) than ProtoPNet, TesNet, ProtoPFormer, and so on. These previous approaches only use 1*1 prototypes. Does the performance improvement of the proposed method come from the largely increased number of sub-prototypes?\n2. The idea of greedy matching algorithm is similar to the greedy prototype projection, proposed in [1, 2]. The authors are suggested to state their difference of the related works. Also, some important work using prototypes for interpretable image classification should be reviewed and discussed, such as [3, 4, 5]. \n3. Regarding the adaptive slots mechanism, it is good for the motivation of to learn an additional indicator to measure the importance of sub-prototypes. From my understanding, the learnable vector v is like a gate, which should be saved as model parameters after training. One potential limitation is such mechanism introduces an extra gate parameter v, compared with previous methods.\n4. It not much clear about the adjacency masking. Since the prototypes are not initialized to have the position information, how do choose the patch/feature tokens around the prototypes within r? \n5. The authors mention the issue of performance degradation after prototype projection. Does the proposed method still suffer from such issue? What extent will the performance drop?\n6. The method has too much loss coefficients, which are selected without detailed tuning procedure. \n\n\nReferences:\n\n[1] Knowledge Distillation to Ensemble Global and Interpretable Prototype-Based Mammogram Classification Models\n\n[2] Pixel-grounded prototypical part networks\n\n[3] PIP-Net: Patch-Based Intuitive Prototypes for Interpretable Image Classification\n\n[4] Learning support and trivial prototypes for interpretable image classification\n\n[5] Concept-level debugging of part-prototype network", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and comments. We are happy that you find our method novel and well-designed for transformers. We address your comments below:\n> The first concerns is about the use of deformable prototypes that containing K sub-prototypes, which means the proposed method will have more learnable prototype vectors (K times) than ProtoPNet, TesNet, ProtoPFormer, and so on. These previous approaches only use 1*1 prototypes. Does the performance improvement of the proposed method come from the largely increased number of sub-prototypes?\n\nThanks for the important question. We discuss this point in section F of the appendix. While our prototypes do consist of more parts than prior work, they represent a similar proportion of each input, since the latent space of our backbones have a finer spatial resolution. We are not sacrificing locality, or changing the general reasoning process. We only add flexibility to the prototypes by decomposing them into multiple, flexible parts. In fact, when using 1*1 prototypes, the model performance is **85.28 +/- 0.11** which is comparable to our final model. So, the improved performance is not necessarily because of the increased number of sub-prototypes. \n\nHowever, **the interpretability of the earlier method is very limited because the 1*1 patches are too small to convey much information**, which is exactly where the motivation of our work comes from. We are open to moving this section to the main body for the camera ready if the reviewer thinks that is warranted.\n\n>  Regarding the adaptive slots mechanism, it is good for the motivation of to learn an additional indicator to measure the importance of sub-prototypes. From my understanding, the learnable vector v is like a gate, which should be saved as model parameters after training. One potential limitation is such mechanism introduces an extra gate parameter v, compared with previous methods.\n\nThe reviewer is correct in that there is an additional parameter associated with each part of each prototype (the slot indicator), but we argue that this is not a limitation - it’s a **benefit**. This indicator allows us to learn prototypes with different numbers of parts, increasing the **flexibility** of the network and helping us reduce the overall number of prototypical parts when possible. Like adding any parameter to any network, there are benefits and drawbacks, but we confirm empirically that the benefits outweigh the drawbacks for performance. And because of this, we can learn prototypes that capture more specific features such as the red eyes in Fig. 4. Without these additional parameters, instead of learning the exact red eye feature, it will only learn the head of the bird as a whole. This is an important discussion which we will include in our revisions.\n\n> It not much clear about the adjacency masking. Since the prototypes are not initialized to have the position information, how do choose the patch/feature tokens around the prototypes within r?\n\nWe apologize for the confusion. Adjacency masking is tied in with the greedy matching procedure: we match the first part of a prototype to anything in a given image, mask out every latent patch more than radius r away from the selected patch, and greedily select the best match for the next part from among the cells that were not masked out. We hope this clarification helps.\n\n>  The authors mention the issue of performance degradation after prototype projection. Does the proposed method still suffer from such issue? What extent will the performance drop?\n\nYes, like other prototype based methods, we do see a small drop in performance after projection relative to before. The drop in performance is usually around 0.8 to 1% accuracy, but we report only accuracy values taken following the projection step. That means the strong performance we report already accounts for this slight drop in performance. \n\nAs described in Theorem 2.1 from ProtoPNet [1], under a perfect training setting, the drop in performance should be negligible. The big drop in performance typically seen is mainly caused by the fact that the prototypes are not trained semantically close enough to the latent patches. Since the prototypes trained in our method are semantically much closer to the latent patches, our drop in performance is minimal, as discussed above.\nWe will make this clear in our revised paper.\n\n> The method has too much loss coefficients, which are selected without detailed tuning procedure.\n\nLike any hyperparameter, these loss coefficients can be tuned by a variety of techniques. For simplicity, we used the values suggested in prior work for terms that already existed, and used a small grid search to select the other values. No dedicated tuning procedure is necessary for these coefficients. Additionally, we note that the number of coefficients in our loss (five) is comparable to other similar work, such as TesNet [2]. We will clarify this in the revisions.\n\n[1] Chen, Chaofan, et al. \"This looks like that: deep learning for interpretable image recognition.\" Advances in neural information processing systems 32 (2019).\n\n[2] Wang, Jiaqi, et al. \"Interpretable image recognition by constructing transparent embedding space.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NFXRuebUjH", "reviewer": "Reviewer_GK9W", "summary": "The authors introduce ProtoViT, a model that leverages the Visual Transformer (ViT) architecture and integrates prototypical parts for case-based reasoning. This method is self-explainable, adhering to the rule \"this looks like that.\" A novel aspect of ProtoViT is the use of prototypical parts of varying sizes, utilizing a ViT backbone. The authors claim that these prototypical parts are coherent and inherently interpretable. They evaluate ProtoViT on the CUB and Stanford Cars datasets, using accuracy as the metric for comparison. The training process for the model involves five loss components in addition to cross-entropy.", "questions": "How are you measuring all the claims, especially coherence and inherently interpretable models?\nCould you provide a prototype purity metric for PIP-Net? \nIs it possible to analyze whether ViT can be a backbone for prototypical parts or not, test with spatial misalignment benchmark?", "limitations": "There is no quantification of interpretability, no user-study, nor no reference to XAI benchmarks such as FunnyBirds and spatial misalignment benchmarks.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written, with images effectively illustrating the intended concepts. The introduction of the greedy matching algorithm is particularly engaging and holds significant importance for the community. The introduction section is well-crafted, clearly outlining the contributions. Additionally, the computational experiments are thorough and demonstrate comprehensive accuracy.", "weaknesses": "The primary concern lies in ensuring that the ViT backbone can maintain prototypical parts that are both local and inherently interpretable. Since ViT uses attention mechanisms that mix information from all patches, there is a risk of confusion for the end user. To address this, I suggest conducting a spatial misalignment benchmark [1] to analyze its influence.\n\nAdditionally, there are no metrics related to explainability demonstrating whether the model improves interpretability, such as with FunnyBirds [2] or through a user study [3].\n\n[1] Sacha, Mikołaj, et al. \"Interpretability benchmark for evaluating spatial misalignment of prototypical parts explanations.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 19. 2024.\n[2] Hesse, Robin, Simone Schaub-Meyer, and Stefan Roth. \"FunnyBirds: A synthetic vision dataset for a part-based analysis of explainable AI methods.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n[3] Kim, Sunnie SY, et al. \"HIVE: Evaluating the human interpretability of visual explanations.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and comments. We are happy that you find our matching algorithm engaging and a significant contribution and our experiments thorough and comprehensive. We address your comments below:\n\n> The primary concern lies in ensuring that the ViT backbone can maintain prototypical parts that are both local and inherently interpretable. Since ViT uses attention mechanisms that mix information from all patches, there is a risk of confusion for the end user. To address this, I suggest conducting a spatial misalignment benchmark [1] to analyze its influence.\n\nThank you for the great suggestion. As suggested, we have run experiments on the benchmark [1], and found our model outperforms models such as ProtoPool [2] and ProtoTree [3] that are CNN based architectures. We present the complete results in our shared response. Our model achieves very close performance in PLC to ProtoPNet [4] and strictly better performance in PAC and PRC. Our model is also robust to adversarial attacks with roughly only 2 - 3% drop in accuracy. These results suggest that our model with ViT can create prototypical parts that are both local and inherently interpretable equally or better than many of the influential existing models with CNN backbones. We will add these results to our revised paper. \n\nFurther, in section G of the appendix, we present several examples of global analysis. These figures show the highest activations of a variety of prototypes on training and test images, and consistently show that each prototype activates highly on a single semantic concept. This consistency suggests that our visualizations are **a good representation of the prototypes**, and not just spuriously selecting nice patches.\n\n> Additionally, there are no metrics related to explainability demonstrating whether the model improves interpretability, such as with FunnyBirds [2] or through a user study [3].\n\nWe apologize for the confusion, we did indeed run a user study based on the agreement task from HIVE, which is your reference [3]. It’s in Appendix I, labeled in the table of contents in the appendix as a user study. The study shows that our model improves interpretability, and shows statistically significantly improvements in user understanding and confidence on the model reasoning process. We will make sure the user study is clearly discussed in the main body of the revised paper.\n\n> How are you measuring all the claims, especially coherence and inherently interpretable models? Could you provide a prototype purity metric for PIP-Net? Is it possible to analyze whether ViT can be a backbone for prototypical parts or not, test with spatial misalignment benchmark?\n\nWe agree that these are important clarifications. For coherence, we have a definition in the training algorithm, Section 3.4, Equation 4. “Inherently interpretable models” are constrained to make their reasoning processes easier to understand, see work [5]. Our paper’s prototype-based reasoning is a constraint on the network to make its reasoning process easier to understand. The result from the newly run misalignment benchmark described above and in our global response shows that ViT can work as good as CNN backbones in our proposed method. \n\nFinally, while we agree that a purity metric would be useful, however, given the limited time to prepare our rebuttal, we decided to focus on other experiments, such as the misalignment experiments on the new benchmark [1] suggested by the reviewer (which we deemed more important). In our understanding, the purity metric simply quantifies whether prototypes consistently activate on the same concept, which we show to be the case through many examples of global analysis. We expect the purity metric to be strong based on our global and local analysis results and we are working on computing the purity metric to add to our revised paper.\n\nThanks again for the thorough review and the great suggestions. We believe we have addressed the main concerns raised by the reviewer. We are happy to provide additional clarifications if needed.\n\n[1] Sacha, Mikołaj, et al. \"Interpretability benchmark for evaluating spatial misalignment of prototypical parts explanations.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 19. 2024.\n\n[2] Rymarczyk, Dawid, et al. \"Interpretable image classification with differentiable prototypes assignment.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[3] Nauta, Meike, Ron Van Bree, and Christin Seifert. \"Neural prototype trees for interpretable fine-grained image recognition.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[4] Chen, Chaofan, et al. \"This looks like that: deep learning for interpretable image recognition.\" Advances in neural information processing systems 32 (2019).\n\n[5] Rudin, Cynthia, et al. \"Interpretable machine learning: Fundamental principles and 10 grand challenges.\" Statistic Surveys 16 (2022): 1-85.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bbvGLxf7TI", "reviewer": "Reviewer_Mc2e", "summary": "The authors present a novel method for interpretable image classification by incorporating a vision transformer (ViT) into the prototypical neural network framework which provides case-based reasoning to neural network based image classifiers. They claim that most existing prototypical methods are convolutional neural network (CNN)-based, and those methods are limited by spatially rigid prototypes, thus failing to handle geometric variations of objects. Existing methods that try to handle this geometric variation either rely on a continuous latent space, which is not compatible with ViTs, or are other prototype-based ViT that fail to provide inherently interpretable explanations. Due to these existing problems, the author present ProtoViT with the following contributions:\nIncorporates a ViT backbone that can adaptively learn interpretable prototypes that can handle geometric variation of different sizes.\nThey achieve the above with a greedy matching algorithm utilizing an adjacency mask and an adaptive slots mechanism.\nThey give empirical evaluation showing SOTA accuracy and a qualitative analysis showing the faithfulness and coherence of the learned prototype representations.", "questions": "Examples in the paper only show good outcomes. What do incorrect predictions look like, and does the explanation (prototypes) show a reason for why the model was incorrect?\n\nDo the authors have additional comments about potential information leakage between image patches in the attention step occurring in the activation of the prototypes? I ask this in context of a latent patch containing more information regarding other patches around it contributing to the learned prototype. When we project on this prototype, we look at its spatial position and project to the real image, but it may not tell the whole story.\n\nWhen determining the next sub-prototype, why do you only consider the last sub-prototype when creating the adaptive mask given radius r? Why not all the currently selected sub-prototypes? Wouldn’t you get more coverage and cohesive information if considering them all?\n\nWhy can’t prototypes be shared across classes? While I still think the prototypes learned are good, I think a limitation that wasn’t stated is the lack of across class prototype sharing.\n\nI feel like the coherence loss could limit your overall prototype representation. If all sub-prototypes have to be similar, then wouldn’t that hurt a prototype representation that has distinct parts. I’m thinking of a potential example of three differently colored stripes being next to each other. If the sub-prototypes are each of different colors, wouldn’t they be very different; thus discarded due to this loss? This is even shown in E.3. I know that is where the adaptive mask could combat this problem, but I would like more discussion on this. \n\nI’m curious how sensitive your method is to masking / perturbation. For example, if you mask out some or all the sub-prototypes for the top activated area in the test image, can the model still find the prototype elsewhere if it still exists in the image? A particular case I’m interested in is if a prototype is about the blue on a bird’s feathers, so you make that part that the model initially detected. Does the model still pick up the blue elsewhere?\n\nCould you expand more on why you “believe that simply adding prototype layers to an architecture without well defined “cases” does not make the new architecture more interpretable.” In particular, what do you mean by ‘well defined cases’?", "limitations": "The method lacks across class prototype sharing.\n\nThe coherence loss lack discussion on its limitation of making sub-prototypes being similar thus hindering diverse representation which may be important in a prototype.\n\nThe paper lacks comparison to other interpretable ViT methods", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Soundness:\n\nThe methods are clean and sound with ample ablation experiments, and it appears their approach can perform better than others (marginally) and have interpretable and coherent prototypes. \n\nPresentation:\n\nThe paper was easy to follow with clear claims, ideas and methods. While some of the figures could be a bit cleaner (such as the boundaries and borders in Figure 4), and some notation seemed a bit odd, they communicated their ideas/methods well.\n\nContribution:\n\nThe paper gives a clean method for utilizing ViTs in the prototypical framework of deeplearning for interpretability and even incorporates existing methods for making prototypes more flexible via utilizing the approach from Deformable ProtoPNet. They incorporate a novel coherence loss that encourages sub prototypes to be similar to each other. In addition, this paper utilizes a greedy matching algorithm with an adaptive mask to learn geometrically local sub-prototypes. This method also allows for an adaptive number of sub-prototypes through the slot pruning mechanism.", "weaknesses": "Soundness:\n\nThe lack of qualitative comparison with other ViT methods. I know the authors state that these other vision transformer methods do not project the learned prototypical features to the closest latent patches, but they still provide explanations. Could more be expanded on this and/or a figure showing this lack of reasoning/inherent interpretability?\n- This is my biggest concern\n\nPresentation:\n\nIn the section 3.4 for “Optimization of last layers” did you mean “... l-th class prototypes…” with a plural on the prototypes? This was unclear to me\n\nContribution:\n\nHowever due to already preexisting ViT prototype methods (with a lack of comparison to them), the contribution this ViT makes compared to others is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and comments. We are happy that you find our method sound and our experiments ample. We address your comments below:\n\n> The lack of qualitative comparison with other ViT methods..\n\nWe attempted to compare our visualizations to those from ProtoPFormer [1], but are unable to show any reasoning from ProtoPFormer, since they do not have explicit prototypes and their code does not include functionality to perform analysis on their prototypes. From their code, we were only able to extract prototype-wise activation patterns on a given test image and the masked out version of the test image from the global branch. We could not find any way to visualize the prototypes yielding these activation maps, which is likely because the prototypes are not projected. \nWe also attempted to visualize ViT-Net [2], but found that they have not provided any visualization code. We reached out to the authors requesting such code, but have not yet received any response. We are happy to discuss this point further during the discussion period if desired. We would like to emphasize that the other prototype-based ViTs are not able to address geometric variations. Those methods incorporate non-deformable CNN layers after the ViT encoder. Unfortunately, those methods do not project prototypes to the closest latent patches and thus have no visualizations. Moreover, even if they had a visualization of their prototype, it would be similar to ProtoPNet which we directly compared in Fig. 1. In short, we agree that such visual comparisons would be useful, but as discussed above, this is unfortunately not possible. We will however add parts of the discussion above to our revised paper. We hope this addresses your concerns. \n\n> In the section 3.4 for “Optimization..\n\nWe agree that the current phrasing is confusing – we will change it to  “... associated with class b for each prototype from class l”.\n\n> wrong prediction examples\n\nThank you for the great suggestion. We have added some examples of incorrect reasoning to the shared response. Interestingly, we found that some of the time where the algorithm appears to have gotten it wrong, it was actually because the dataset was mislabeled and the algorithm was actually correct. We present examples of this in the shared response and will add them to our revised paper.\n\n> ..  why only consider the last sub-prototype..\n\nConsidering the neighbors of only the most recently selected part helps encourage prototypical parts to have a consistent relationship – i.e., the third part must always be adjacent to the second. If a prototype represents a bird’s ankle, it should always be true that the leg part is adjacent to the foot part, which is adjacent to the toe part. \n\nMoreover, we have found that further restricting where sub-prototypes activate improved the semantic consistency of prototypes. The extreme case of this – removing adjacency masking entirely – is shown in Appendix Fig 8, where a prototype without any masking confuses a bird’s beak with another’s feet. As such, we aimed to keep adjacency masks tight by selecting only neighboring cells to the most recently activated sub-prototype.\n\n> Information Leakage: \n\nThis is another great question. Due to character limit, please refer to the global response and the response to reviewer **GC9W**. We have used empirical evaluation to show that the theoretical possibility of information leakage isn’t happening to our model.\n\n> shared across classes?\n\nPrototypes could indeed be shared across classes if desired – ProtoViT is compatible with methods like ProtoPool [3] and ProtoPshare [4]. For the purposes of this work, we used a simple linear layer between prototype activations and class predictions for ease of training. We wanted to highlight the impact of the novel elements of this work without combining too many features from the literature, in order to keep our contribution clear.  \n\nThanks again for the question, we will clarify this in our revised paper.\n\n> the coherence loss could limit overall prototype representation.\n\nCoherence loss only encourages the sub-prototypes to be semantically similar so that each prototype will represent only one semantic concept. This makes prototypes simpler to understand, since they represent a single, intuitive concept. \nIt is important to note that this coherence is enforced **in the latent space**. Every part of a sub-prototype does not need to be visually identical; they simply have to be semantically similar. If the three stripes described are really important for identifying the species, we might expect them to be encoded close to each other in the latent space, and thus be allowed to be grouped together by coherence loss. Finally, if so desired by the user, the coherence loss can be de-emphasized through tuning its corresponding hyperparameter. The inclusion of the coherence loss allows the users to control this aspect of the prototypes as in Appendix E.3.\n\n\n> perturbation? \n\nThis is a great question. We have included examples in the shared pdf and discuss this in our global response.\n\n> ‘well defined cases’?\n\nThis is a very key point that we are happy to further clarify. By well-defined cases, we mean that every prototype should be explicitly tied to one or more images so that we can visualize them. In both ProtoViT and the original ProtoPNet, this is achieved by projecting each prototype to be exactly equal to part of the latent representation of some training image. We can then refer back to the training image for a visual representation of the prototype. Without such a mechanism to enable visualizations, prototypes are just arbitrary learned uninterpretable tensors in the latent space of the network, which are not that different from a convolutional filter in any black box model. \n\nWe again thank the reviewer for their thoughtful comments. We hope we have addressed all your concerns in our response and are happy to provide additional clarifications if needed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their comments. In addition to addressing all of your concerns individually, we believe the following clarifications and additional experiments may be of interest to all of you. These additional experiments further reinforce the strength of our method and will be added to our revised paper.

**(1) Location misalignment benchmark:**

To ensure that the ViT backbone can maintain prototypical parts that are both local and inherently interpretable, we performed analysis on the benchmark [1] as suggested by reviewer **GK9W**. The quantitative result can be found in the shared pdf. Overall, we found that by integrating ViT backbone with our algorithm, it achieves equal or better performance than the influential CNN based prototype models such as ProtoPNet [2], ProtoPool [3], and Prototree [4] in terms of percentage change in location (PLC), percentage change in activation (PAC), and percentage change in ranking (PRC). For these metrics, lower is better. To ensure a fair comparison, we strictly follow the procedure of performing gradient based adversarial attacks as described in the paper. However, we would like to point out that since our prototypes only contain 4 (out of 196) or less patches, the adversarial attack performed is stronger to our model than other CNN based models, as a larger proportion of the images are perturbed for our model. Results can be found in Table 1 of the shared pdf. 

**(2) Perturbation Analysis:**

Additionally, we have provided several instances of the perturbation analysis suggested by reviewer **Mc2e**, in which we mask out the region selected by each prototype, as shown in Figure 2 of the shared pdf. In each row, we mask out all matched locations for a prototype (middle-left column) using a white mask on the black wings and a black mask on the red parts, and check where that prototype activates after masking (shown in the leftmost column). We then confirm that the activated region for other prototypes remains reasonable when the mask from another prototype is applied (right two columns). We observed that after removing the preferred region by each prototype, it activates on another reasonable alternative (e.g., a red belly prototype might activate on a red back as a second choice). 

Moreover, we performed the analysis on the test image with a modified background, and we observe that altering the background does not substantially impact the activation location of our ViT-based prototype, with all parts still solidly focused on the bird’s breast (again please refer to Figure 2 of the pdf). This is unlike the observation made in the original Location Misalignment Benchmark paper, where a CNN-based prototype model suffers from a dramatic shift in the activation location when there is a background change. This shows that our ProtoViT has a better location alignment than the CNN-based prototype models.

**(3) Misclassification examples:**

We provide the reasoning process of how our model misclassified a test image of a summer tanager and a slaty backed gull in Figure 1 of the shared pdf. We found that the misclassification of the given summer tanager example may be because of data mislabeling in the original dataset. A summer tanager does not have a black colored wing. That test image should indeed belong to the scarlet tanager class as the model predicted. Similar mislabeling cases also happen for Red Headed Woodpecker with image ID Red_Headed_Woordpecker_0018_183455 and Red_Headed_Woordpecker_0006_183383, as we found out when randomly selecting examples to present for the paper. 

These examples showcase the ability of our method to help us understand the reasoning process of the model, not just when it is right, but also when it is wrong.

**(4) Other prototype-based ViT models:**

Finally, we would like to emphasize that some models suggested by the reviewers for comparison (e.g., ProtoPFormer [5], ViT-Net [6] and etc.) do not perform any form of prototype projection, and as such do not have any explicit visualization associated with their prototypes. Without such a mechanism to enable visualizations, prototypes are just arbitrary learned tensors in the latent space of the network, which are not that different from a convolutional filter in any black box model. In contrast, our network does perform projection, allowing each prototype to be tied directly to one well defined training image. This allows us to present clear, faithful visualizations of model reasoning. 

[1] Sacha, Mikołaj, et al. "Interpretability benchmark for evaluating spatial misalignment of prototypical parts explanations." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 19. 2024.

[2] Chen, Chaofan, et al. "This looks like that: deep learning for interpretable image recognition." Advances in neural information processing systems 32 (2019).

[3] Rymarczyk, Dawid, et al. "Interpretable image classification with differentiable prototypes assignment." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[4] Nauta, Meike, Ron Van Bree, and Christin Seifert. "Neural prototype trees for interpretable fine-grained image recognition." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[5] Xue, Mengqi, et al. "Protopformer: Concentrating on prototypical parts in vision transformers for interpretable image recognition." arXiv preprint arXiv:2208.10431 (2022).

[6] Kim, Sangwon, Jaeyeal Nam, and Byoung Chul Ko. "Vit-net: Interpretable vision transformers with neural tree decoder." International conference on machine learning. PMLR, 2022.

Meta-review: The paper proposes to integrate deformable prototypes with vision Transformers for interpretable image classification. The initial review ratings were mixed. Yet after the rebuttal and discussion, all reviewers unanimously recommend this paper. They recognize the technical contributions and applaud the presentation of the paper. The authors are encouraged to incorporate review comments and rebuttal materials into the final version.
Decision: Accept (poster)