This Looks Like It Rather Than That: ProtoKNN For Similarity-Based Classifiers
Authors: Yuki Ukai
Tsubasa Hirakawa
Takayoshi Yamashita
Hironobu Fujiyoshi
Venue: ICLR 2023 poster (2023)
Keywords: XAI
Inherently Interpretable Model
This Looks Like That Framework
Fine-grained Image Classification
Deep Learning

Abstract: Among research on the interpretability of deep learning models, the 'this looks like that' framework with ProtoPNet has attracted significant attention. By combining the strong power of deep learning models with the interpretability of case-based inference, ProtoPNet can achieve high accuracy while keeping its reasoning process interpretable. Many methods based on ProtoPNet have emerged to take advantage of this benefit, but despite their practical usefulness, they run into difficulty when utilizing similarity-based classifiers, e.g., in domains where unknown class samples exist. This is because ProtoPNet and its variants adopt the training process specific to linear classifiers, which allows the prototypes to represent useful image features for class recognition. Due to this difficulty, the effectiveness of similarity-based classifiers (e.g., k-nearest neighbor (KNN)) on the 'this looks like that' framework have not been sufficiently examined. To alleviate this problem, we propose ProtoKNN, an extension of ProtoPNet that adopts KNN classifiers. Extensive experiments on multiple open datasets demonstrate that the proposed method can achieve competitive results with a state-of-the-art method.

Published as a conference paper at ICLR 2023
THIS LOOKS LIKE IT RATHER THAN THAT: PROTOKNN FOR SIMILARITY- BASED CLASSIFIERS
Yuki 1,, Tsubasa Hirakawa", Takayoshi Yamashita", and Hironobu Fujiyoshi GLORY Ltd., 2Chubu University
ABSTRACT
Among research on the interpretability of deep learning models, the 'this looks like that' framework with ProtoPNet has attracted significant attention. By com- bining the strong power of deep learning models with the interpretability of case- based inference, ProtoPNet can achieve high accuracy while keeping its reasoning process interpretable. Many methods based on ProtoPNet have emerged to take advantage of this benefit, but despite their practical usefulness, they run into dif- ficulty when utilizing similarity-basee classifiers. This is because ProtoPNet and its variants adopt the training process specific to linear classifiers, which allows the prototypes to represent useful image features for class recognition. Due to this difficulty, the effectiveness of similarity-based classifiers (e.g., k-nearest neighbor (KNN)) on the 'this looks like that' framework have not been sufficiently exam- ined. To alleviate this problem, we propose ProtoKNN, extension of ProtoP- Net that adopts KNN classifiers. Extensive experiments on multiple open datasets demonstrate that the proposed method can achieve competitive results with state- of-the-art method.
INTRODUCTION
Deep learning has achieved very high accuracy in a variety of computer vision tasks. However, since the reasoning process of deep learning models is black-boxed and cannot be interpreted by human operators, it is very difficult to validate their inference, and this impedes their utilization in high-risk domains. To alleviate this problem, several methods for constructing inherently interpretable models have been proposed. However, inherently interpretable models generally suffer from degraded accu- racy compared to black-box models. 'Gray-box' models have thus been proposed (Alvarez-Men) -Mels, 2018: Chen, 2019; Koh, 2020) to take advantage of the power of deep learning models while keep- ing the reasoning process interpretable. Among the gray-box model approaches, the 'this looks like that' framework with ProtoPNet (Chen, 2019) has attracted significant attention because it can guar- antee a transparent reasoning process without any additional supervision. ProtoPNet first calculates the similarity of the input samples to the prototypes corresponding to image patch in the training set and then classifies samples with inherently interpretable models on the basis of this similarity. This process enables ProtoPNet to explain its reasoning process by providing patches in the train- ing set that the model considers similar to the input sample. Thus, interpretability with case-based reasoning is achieved. Thanks to this advantage transparency, many methods based on ProtoPNet have been proposed (Wang, 2021; Nauta, 2021; Rymarczyk, 2021; Donnelly, 2022; Keswani, 2022; Rymarczyk, 2022).
When training ProtoPNet, the weights of the linear classifier connecting each of the prototypes and class logits are fixed, and the feature vectors corresponding to an image patch are linked to the pro- totypes if the prototypes make a positive contribution to the class logits to which the image belongs. This enables the prototypes to represent the image patches most useful for class recognition. How- ever, due to this special training process, it is difficult for ProtoPNet to utilize any classifiers other than the linear classifier. As an alternative, Nauta (2021) proposed ProtoTree, which use a deci- sion tree for the last classifier. However, this method is limited to the decision tree, which makes it difficult to utilize in similarity- based classifiers with the 'this looks like that' framework. Similarity- based classifiers perform inference the basis of similarities (or distances) between samples. As we will demonstrate in the experimental section and in the Appendix (Sec. D.2), interpreting the distance enables to obtain more fine-grained explanation in counterfactual manner, which is

---

Published as a conference paper at ICLR 2023
important for understanding and interpreting the reasoning process of the model. Therefore, in this work, we extend ProtoPNet so that we can utilize similarity- based classifiers (specifically, k-nearest neighbor (KNN) classifiers) in the 'this looks like that' framework.
When extending ProtoPNet to the Similarity-based classifier, it is no longer possible to pre-define the relationship between the prototypes and the class labels, which is necessary for calculating the cluster loss. As we will discuss in Sec. 2.2.2, it is also difficult to estimate this relationship from only one sample. Therefore, the main difficulty is how to estimate the relationship between the prototypes and the class labels. Our concept for estimating this relationship is to compare each sample in a mini- batch and sum up the most distinctive prototypes. This enables us estimate which prototypes are relatively more related to which samples and thus which classes. Then, our novel cluster loss can be defined based on this estimation. In summary, our contributions are three-fold:
We propose ProtoKNN, an extension of ProtoPNet that can utilize KNN classifiers. This is the first work to examine the effectiveness of similarity-based classifiers in the 'this looks like that' framework.
We developed novel loss function for ProtoKNN that replaces the cluster loss in Pro- toPNet. This enables to train our model without predefining the relationship between classes and prototypes.
The proposed method achieved competitive results with state-of th-t ProtoPNet variant multiple open datasets.
2
METHOD
In the following, we first describe the notation 1 used in this paper. Then, we present the training strategy of the proposed method and explain how to classify the samples. Finally, we demonstrate how to interpret the reasoning processes of our method. For context, also briefly revisit the origins of ProtoPNet and elaborate on why it cannot directly utilize similarity-base classifiers in the Appendix (Sec. A).
2.1 PRELIMINARY
The input image and its class labels are denoted as x and respectively. Unless otherwise specified, we use subscripts a, b... to denote the data indices. Thus, an input image and its class label are denoted as and Ya. respectively. The index sets of the images in minibatch and their cardinality are denoted as B and [B], respectively. We use F to denote the feature extractor and Z to denote the feature map output by F. i..., Z. F(z.). is used denote the feature vectors contained in pixel of the feature map Z. We also call these feature vectors 'image patch features' in this paper. After the transformation, the similarity between the prototypes {p:} and the image patch features contained in the feature map Z, are calculated. The maximum similarity value is defined as the similarity of the input image Ia to the prototype pi, as Sa, max Sim(z, p.). Here, we denote the similarity of the input image Ta as 8, and its component corresponding to p, as Sa,i- Sim is the function that calculates the similarity (cosine similarity in this paper) between the feature vectors and the prototypes. In the following, we refer to the similarity of the input image za. and the index sets of the prototypes are denoted P. The indicator function is denoted as 1 (condition), which returns 1 if the condition is true and 0 otherwise.
as the 'prototype profile'
2.2 TRAINING PROCESS
Originally, ProtoPNet utilized three loss functions: classification loss, cluster loss, and separation loss. In the proposed method, do use the separation loss because we expect the prototypes to be common among the samples with different class labels. Instead, we use the auxiliary loss function proposed in the context of deep metric learning to help the backbone model acquire better feature extraction ability. In summary, we train our models with three loss functions: classification loss Trusss the novel cluster loss Tedsts and auxiliary loss Tuuax- Figure/ 1 shows the loss scheme of the proposed method. The details of each loss are described in the following.
We basically follow https: /ttp:/ggtttt///b comm http///oooaaatiiiiooooooo _ot.ttion........
2

---

Published as a conference paper at ICLR 2023
GAP
Lincar
Lincar
GAP
Lauxx
Model backbore
tark
Image Xa
$
Image
Prototype P:
Lutge
Conv.
arguan Pr 2CK,
Feature map 7
Figure I: The loss scheme of the proposed method. The model transforms the input sample Ia into the feature map Z. and then encode it into the prototype profile 8a. We measure the distance between images on the space spanned by 8д- To optimize the distance, we impose Ltask on 8a- We also impose Leist to link the prototypes and the feature vectors contained in Z In addition, to help the model acquire the capability of feature extraction, we directly impose Laux on the output of the model backbone.
2.2.1 CLASSIFICATION LOSS Ltask
In contrast ProtoPNet and its variants, the proposed method performs inference the basis of similarity between samples. Therefore, in the space spanned by the similarity between the prototypes and the input images, it needs to be 'discriminable" rather than 'separable'. We therefore use the loss function (Wu, 2017) typically utilized in the context of deep metric learning instead of the ordinarily used cross-entropy loss for Lensk. Thus, we calculate Lrask as
Ltask
1
Σ (Idap + m B]+ + [m dan + B]+), aEB
N
where m, 3 are margin and learnable parameters, and p. and are the data indices of the anchor, the hard-positive, and the hard-negative samples, respectively. The ReLU function is denoted as [4+ and N is the number of terms that take non-zero value in the summation. dap and dan the Euclidean distance defined as |s, spll2 and | s, 12. respectively.
2.2.2 ESTIMATION OF PROTOTYPE ATTRIBUTION
As we will discuss the Appendix (Sec. A), the predefined relationship between the prototypes and the class labels is no longer available. Thus, to calculate the cluster loss, we need to calculate which image patch features should attached to each of the prototypes and to what degree. For this purpose, first estimate the affiliation of the prototypes to each class.
Specifically, we first estimate which prototypes are contained in which samples. Utilizing the proto- type profiles would be natural choice here, but, thresholding methods are not appropriate because the prototype profiles change their statistical properties during the training. Therefore, we estimate the affiliation by comparing the prototype profiles, i.e., the similarity of the samples to the proto- types, in the minibatch. the image feature represented by prototype p; is contained in sample x, and not in sample 2h, we can expect that the similarity (s.) of sample x. prototype p; greater than the similarity (8t) of sample dy to prototype p.. Therefore, by extracting the prototypes with large difference between s, and s for all of the indices in the minibatch a set of the prototypes contained in I., To achieve this, stochastic sampling with the Humbel-Mar trick is conducted by
IB), we can construct
E(ga: pa)
Σ r bEB/(a}
S0.i 8b,4
(2)
3

---

Published as a conference paper at ICLR 2023
Sec 2.2.2 Estimation of prototype attribution
Sec 2.2.3 Calculation Letse
Samping (Eq. e Elementwise subtraction
Modify the attribution (Eq. 10)
Model Ax
X
Class average (Eq, 43
Modified antribution Th
Averaged artribution B(y.p))
Prototype atillation POYPP
Gumbel max
Sum
Deblas (Eq. 5)
Sampling results B().p))
Debiased attribution Ely.p))
Calculate the cluster loss (Ec) 11)
Figure 2: Overall process of calculating the cluster loss. First, the prototype profiles are compared estimate which prototypes are likely to be included in each sample (this process is denoted as "Sampling'). Second, we average the 'sampling results' within class and then debias the results so that the class average of all results the same. Third, the debiased results are normalized so that the total amount of the prototypes contained in class is 1. We refer to the normalized results as prototype affiliation'. Finally, we modify the prototype affiliation and link the prototypes to image patch features on the basis of the modified attribution. Please refer to the main text for the details of each process.
where t is the temperature parameter (fixed to 0.05 in this paper). T is the Gumbel-Mat operation (Jang. 2017) and returns for the index of the sampled prototype and otherwise, i.c..
exp(s, + 77)
r(s:) =
arg max Σ exp (8k +Tk)
(3)
where Yi and "/ are the random variables that follow the standard Gumbel distribution. We obtain the sampling results P(y.p)) with respect to each class label by averaging the sampling results E( pt) among the samples that have the same class labels, as
Ely,p.) ELEB E(xa, Po 1 v)
ΣEE 186 3)
The sampling results Ely, P.) can still be biased toward some prototypes independent of the class labels, which is undesirable. Thus, de-bias the sampling results following the equation below so that they are not biased toward any specific prototype.
Ely p.)
B(y.p:) = EySEY E(yo P.)
(5)
where is the set of class labels contained in the mini-batch. This 'debiasing' enforces the sum of the sampling results with respect to the class labels are the same. In other words, the debiasing process reflects the assumption that in the absence of information about the class, the same amount of prototypes are observed with probability. Finally, we calculate the affiliation of the prototypes the class label y by normalizing the sampling results Ely. p:) so that the total amount of the prototypes contained in class would be the same (i.e., 1), as
Ély.pd p.) P(y.p.)
(6)
Eeer Flv P.)
2.2.3 CLUSTER LOss Lelst
Once we estimate the prototype affiliation (Sec. 2.2.2), can calculate and minimize the cluster loss to link the image patch features to the prototypes on the basis of that estimation. Here, due to the various views of objects in the images, the samples do not always contain the prototypes whose affiliation to the class high. Therefore, we need to modify the affiliation for each sample. Note that can relatively estimate how much the sample v. contains the image feature represented by
4

---

Published as a conference paper at ICLR 2023
the prototype p; the basis of the distance between them (mingeZ, ||2 -p.1)). Therefore, propose to modify the attribution considering the distance. Specifically, we formulate this problem linear assignment problem:
T = arg min
Σ
Co, s.t.
ΣTT = 1 = v),
= =
(7)
T REBRiE
iEP
aER
where Ny Es (m = y) is the number of samples contained in a mini-batch whose class label is y and Ca.5 is the cost function defined
Ca.i min =eZ
|| p:112-
The solution Te can be considered the modified attribution of the prototypes each sample, whose average among the samples match the attribution to the class. The components of T would be small when the sample za does not contain the prototype ps, ie., when the distance between them is large, and high the other way around. Thus, we use the solution T to calculate the cluster loss,
Ledst E Tap
NEY aEB,EE
Here, the set of all class labels contained in the minibatch.
To minimize Eq. 9, need to solve Eq. 7. However, quickly obtaining exact solution to Eq. difficult, and is not necessary for to be unique. Therefore, we approximately solve Eq. 7 with the Sinkhorn -noopp algorithm (Cuturi, 2013) and re-define T as
1 1TV log T
T T arg min TVV
Σ dEB,
T" Ca,Ca,i +
X
(10)
S.L.
T
1Bya y),
ΣTH Th agi
N₂ P(v.P)))
where À is the hyper-parameter determining the weight of the entropy regularization term (fixed to 0.05 in this paper).
We also modified the proposed cluster loss to maximize the gap between the minimum and the aver- age distance between the prototypes and the image patch features following ProtoPoo((Rymarzyy 2022). The aim of this modification is address the problem that the learned prototypes tend to focus on the background (Rymarczyk, 2022). We re-define our cluster loss (Eq. 9) as
Liist Σ Σ
Ou where Ca,i = min ||
P:ll
Σ =EZ.
1
p:ll
(1I)
=eZ.
DEY aEB
where Zn| is the number of pixels contained in the feature map Z,.
2.2.4 AUXILIARY LOSS Lauz
As stated earlier, we utilize the loss function proposed in the context of deep metric learning as auxiliary loss Lous to help the model backbone acquire better feature extraction ability. Here, we also specify that do not impose Laux the prototype profile but on the feature vectors obtained by applying Global Average Pooling (GAP) and a linear layer to the feature map output by the model backbone. We refer to this feature vector and its L2- no-oomzal one as g and g in this section. As default, we used proxy anchor loss (Kim, 2020) for Louz, which is formulated as:
Laux =
Σ QEQ+
log + Σ --0(s(0))00) geX,
+
Σ GEQ
log
1+ Σ OEX,
1Q+1
1QI
(12)
where 8, Q. and Q+ margin, scaling factor, the set of all proxies, and the set of positive proxies of data in the mini batch, respectively. Also, for each proxy q, the batch feature vectors is divided into two sets: X²+ set of positive feature vectors of q and X,- the others. We also defined s(ğ, q) 9 ||o|a above.

---

Published as a conference paper at ICLR 2023
is classified into "Marsh Wren" because it looks like
rather than
In that
most aclivaled area
looks unlike
looks unlike
looks like
0.421
looks unlike
looks unlike
looks like
0.366
looks like
looks like
looks uniike
0.302
Difference: 4.61
Figure 3: An example of the interpretation of the proposed method. Blue bounding boxes surround the areas where the prototypes are the most activated. The numerical values in the center of the figure indicate how much the distance between the anchor and the negative sample was increased by the corresponding prototype relative to the distance between the anchor and the positive sample. Please refer the final paragraph of Sec. 2.4 for more details.
2.2.5 OVERALL LOSS FUNCTION AND PROTOTYPE PROJECTION
We train our models with the combination of the loss functions described above. Specifically, the overall loss function of the proposed method can be written as
Lootal = Lrask + Aelat Loist + Auua Loouz
(13)
where Aetst and Aaur are the hyper-parameters that respectively determine the weight of each loss. We empirically set Adst = 0.1 and Aauz = 1.0 in this paper. After the training, each prototype is projected onto the nearest image patch features in the training following
z
p;
max
(14)
x Ip.||
This guarantees that each prototype represents the corresponding image patches, and thus the trans- parency of the prototypes is achieved. We can then achieve case-based interpretability by classifying samples on the basis of their similarity to the prototypes.
2.3 LABEL. PREDICTION
To predict the class label with method, given the test sample Zrest, first retrieve the nearest top-k samples from the training set. Here, k is the hyper-parameter of the KNN classifier and we the Euclidean distance for the distance metric, i.e., the difference between samples &. and 2s is calculated by ||s, sb..3. Then, the prediction Vpred is given by the majority decision the retrieved top-k samples R(xeest, k), i.e., Upred arg max,, Σx, E (( 1 y). In the experimental section, we mainly report the results with 1, 3, and 5.
2.4 INTERPRETATION THE REASONING PROCESS
We limit the following explanation of the proposed method's inference interpretation to the case of 1 for simplicity. The proposed method can achieve a counterfactual explanation for why the input sample is classified into the predicted class rather than another class. In the following, refer to the input image as the anchor sample and to the sample nearest to the anchor sample in the training set as a positive sample. We also refer to the class of interest other than the predicted class as a negative class and the nearest sample with the negative class label to the anchor sample in the training set negative sample.
As shown in Fig. 3, the proposed method explains its inference by using the positive and negative samples. In contrast the naive KNN classifier, which only provides the positive sample for the

---

Published as a conference paper at ICLR 2023
Accuracy comparison full images of CUB200-2011 with various model backbones. The results of ProtoPNet are directly borrowed from Donnelly (2022). 'V', 'R', and 'D' denote
Table I:
VGG, ResNet, and DenseNet, respectively. Here, the top-1 accuracy [%] is reported.
Model
V16
V19
R34
R50
R152
D121
D161
ProtoPNet(Chen, 2019)
70.3
72.6
72.4
81.1
74.3
74.0
75.4
Def. ProtoPNer(Donnelly, 2022)
75.7
76.0
76.8
86.4
79.6
79.0
81.2
ProtoKNN (k=1) + Margin
75.5
76.7
76.9
87.0
80.3
78.5
80.8
ProtoKNN (k=3) + Margin
76.6
77.2
77.3
87.1
80.6
79.3
81.2
ProtoKNN (k=5) + Margin
77.2
77.6
77.6
87.2
80.6
79.8
81.4
ProtoKNN (k=10) 10) Margin
77.5
77.6
77.4
87.0
80.5
79.9
81.3
reason of its inference, the proposed method can further explain why the distance between the anchor and the negative samples is larger than the distance between the anchor and the positive samples. Later, we will refer to the distance between the anchor and the positive (negative) samples as dap (dan). Here, we used the subscripts a, P, and to denote the data indices of the anchor, the positive, and the negative samples, respectively. In the proposed method, the distance between two samples is determined by the difference between the similarity of each sample the prototypes. Therefore, by extracting the prototypes that have a large contribution to dan and a small contribution to dap- can explain which image feature is responsible for the difference between dap and dan- In other words, the prototype with the largest contribution to da da l| 8n 11 ||8, 8pl12 can be considered as the reason for why the anchor sample is classified into the predicted class.
Consider the interpretation example shown in Fig. 3. First, the anchor sample (top left) and the negative class (Vermilion Flycatcher) are given. Then, the positive (top center) and the negative (top right) samples
retrieved from the training dataset. As result, find that da is 4.61 greater than dap (denoted as 'Difference") in this case. Next, as described above, we extract the prototypes that make the largest contribution to d - de as shown in the middle table in Fig. 3. The extracted results are the reason for why the anchor sample is classified into 'Marsh Wren' rather than 'Vermilion Flycatcher'. Specifically, these results show that the main reason is that the negative sample has red head, while the anchor and the positive samples do not. We can also see that this difference makes da 0.421 greater than dle-- Similarly, by repeating the interpretation described above for the second and third results, we can quantitatively interpret why da is greater than dap- Further examples for the interpretation of the proposed method can be found in the Appendix.
3 EXPERIMENTAL RESULTS
We conducted experiments on three public datasets: CUB200-2011 (Wah, 2011), Stanford Dogs (Khosla, 2011), and Stanford Cars (Krause, 2013). Due space limitations, we only describe the experimental results here. Other details (e.g., implementation details) are provided in the Appendix.
3.1 COMPARISON WITH THE OTHER METHOD
Tables and 2 show the top-1 accuracy of the proposed method with various model backbones on CUB200-2011 and Stanford Dogs, respectively. Here, we directly borrowed the results of ProtoPNet from Donnelly (2022). In these experiments, we used the full images following Donnelly (2022). As shown, our method achieved competitive results with Deformable ProtoPNet (Donnelly. 2022).
Tables 3 and 4 show the comparison results with the state-of-the-an method among variants of ProtoPNet. Here, we cropped the images using bounding boxes following the previous studies (Chen, 2019; Wang, 2021; Rymarczyk, 2022). We also set the number of prototypes to that of ProtoPool (Rymarczyk, 2022). As we can see, our method achieved a higher accuracy than the other method when using single model, and it even achieved competitive results with the ensemble results of the other method in Table 4. Specifically, our method achieved a higher accuracy than ProtoPool when used the same model backbone and the same number of prototypes. These findings demonstrate the effectiveness of our method, which eliminates the need for the sub-optimal optimization process with fixed-weight elassification layer required by the conventional methods.

---

Published as a conference paper at ICLR 2023
Accuracy comparison full images of Stanford Dogs with various model backbones. 'V', 'R', and 'D' denote VGG, ResNet, and DenseNet, respectively. Here, the top-1 accuracy [%] is reported.
Table 2:
'Table 3: Accuracy comparison Stan- ford Cars with state-of of-tr--art method. Here, 'R' denotes ResNet and top-1 ac- curacy [%6] is reported.
Method
V19
R152
D161
Method
R50
ProtoPNet(Chen, 2019)
73.6
76.2
77.3
ProtoTree(Nauta, 2021)
86.6
Def. ProtoPNec(onnnell, 2022)
77.9
86.5
83.7
2022)
88.9
ProtoKNN (k=1)+ + Margin
76.9
85.9
84.7
ProtoKNN (k=l) Margin
90.2
ProtoKNN (k=3) Margin
78.6
86.9
85.7
ProtoKNN (k=3) Margin
90.8
ProtoKNN (k=5) + Margin
79.3
87.3
85.7
ProtoKNN (k=5) + Margin
90.9
ProtoKNN (k=10) Margin
79.5
87.5
85.7
ProtoKNN (k=10) Margin
90.7
Table 4: Accuracy comparison CUB200-2011 with state-of-the-art method. *** and **** denote the ensemble results with 3 and 5 models, respectively.
Table 5: Ablation study on Lauc using full
images of CUB200-2011, Top-1 accuracy [%] is reported here.
Method
Top-1 Acc.
k
Laux
Res34
Res50
(Coee, 2019)
79.2% (*84.8%)
k=
None
74.7%
86.0%
ProtoTree(Nauta, 2021)
82.2% (**87.2%)
k=3
None
75.3%
86.5%
TesNet(Wang, 2021)
82.8% (**86.2%
k=5
None
75.9%
86.6%
ProtoPool(Rymarezzyk 2022)
85.5% (**87.6%)
k=1
Margin
76.9%
87.0%
Def. ProtoPNec(Donneely, 2022)
86.4% (**87.8%)
k=3
Margin
77.3%
87.1%
ProtoKNN (k=1)= + Margin
87.1%
k=5
Margin
77.6%
87.2%
ProtoKNN (k=3) Margin
87.4%
k=1
Proxy-Anchor
77.0%
87.0%
ProtoKNN (k=5) + Margin
87.5%
k=3
Proxy-Anchor
77.5%
87.3%
ProtoKNN (k=10) Margin
87.2%
k=5
Proxy-Anchor
77.3%
87.2%
3.2 ABLATION STUDY
To examine the effectiveness of each component of the proposed method, conducted ablation study on CUB200-2011 I with the full images. In the following, we present the ablation study for Lanz- We include the ablation study for cluster loss Letst in the Appendix.
The results of the ablation study are shown in Table 5. As we can see, the accuracy improved regardless of whether margin loss proxy anchor loss adopted for Tuauz- Thus,
confirm that Lawe helps the backbone model to acquire better feature extraction ability. We can also see that the proposed method utilized the learned good feature representation whether the ranking-based or classification--aat based loss was adopted for Liuur Moreover, the improvement was marginal when ResNet50 was adopted for the model backbone compared when ResNet34 was adopted. This is because we used ResNet50 pretrained on i i-Naturalist 2017, so it has already acquired good feature extraction ability. Therefore, we can conclude that Lauz more effective when the pretrained model has not already acquired good feature extraction ability.
4 RELATED WORKS
Research on the interpretability of deep learning models can be divided into two approaches: (1) the post-hoc approach, which analyzes trained black-box models, and (2) the ante-hoe approach, which constructs inherently interpretable models. The post-hoc approach (Zhou, 2016; Selvaraju, 2017; Lundberg, 2017: Bau, 2017: Jiang, 2021; Hernandez, 2022) is beneficial when analyzing al- ready deployed models because re-training is required. However, in some cases, the explanations provided by post-hoc method will have nothing to do with the model inference (Rudin, 2019). To address this problem in fidelity, some works have utilized the Shapley value (Shapley, 1951) and proposed methods with a theoretical guarantee (Lundberg, 2017: Hamilton, 2022). However, cal- culating the Shapley value is hardard and it is difficult apply these methods to high-dimensional
8

---

Published as a conference paper at ICLR 2023
data such as raw images. Moreover, most methods explain the model inference by providing the image region that contributes the most to the model inference, and the properties of the explanation thus very different from those provided by ProtoPNet and its variants. This makes it extremely difficult to evaluate the interpretability of these approaches in a unified manner, which is why we have omitted any comparison with the post-hoe approaches from the paper. In the following, describe the variants of ProtoPNet (Chen, 2019), which is the basis of our method.
After Chen (2019) proposed the "this looks like that' framework with ProtoPNet, many methods based on ProtoPNet were proposed, and they have improved in both accuracy and efficiency by reducing the number of the prototypes (Wang, 2021; Rymarczyk, 2021: Donnelly, 2022; Keswani, 2022: Rymarczyk, 2022). In particular, ProtoPool (Rymarczyk, 2022) has succeeded in directly learning the prototypes that are common among inter-class samples and achieved high aceuracy. However, as we will explain in the Appendix (Sec. A), these methods all have difficulty in utilizing similarity- classifiers. ProtoPool, even though it is quite successful, requires 'slots' specific to certain classes and still has difficulty utilizing classifiers other than linear ones. Alternatively, ProtoTree (Nauta, 2021) utilizes a decision tree for its classifier and has achieved high accuracy while reducing the number of prototypes, However, its training process is specific to the decision tree and it thus also has difficulty utilizing the similarity based classifier. To address this problem, our work extends ProtoPNet that our knowledge, it is the first work that combines the similarity-basee classifier with the 'this looks like that' framework.
utilize the similarity-based classifier, and to the best of
Our method is also related to research in the field of deep metric learning. Many methods have been proposed to achieve a good feature representation space in this context (Wu, 2017: Wang, 2019; Kim, 2020). most of which utilize the cosine similarity to calculate the similarity between samples (Kim, 2020). However, is unreasonable to L2-normalize the similarity of each sample to the prototypes. which makes it difficult to directly adopt these loss function as Ltask in our method. Instead, we adopt these losses as Toauz to take advantage of the deep metric learning and help the model backbone acquire better feature representation.
5 DISCUSSION
Limitation. Our method utilizes the KNN classifier for classification, so we need to maintain the similarity of each sample in the training set to the prototypes during inference. We also need to calculate the distance between the input sample and each of the samples in the training dataset. This is not desirable from the viewpoint of either memory efficiency or computational efficiency. The computational cost of our method is O(ND), where N refers to the number of samples and D to the number of channel dimensions. Although this cost is negligible when adopting the datasets used in this paper, it might be problematic when scaling up to larger datasets. However, we believe these problem can be solved by properly selecting the samples from the training set to be used during the inference, which we leave to future work.
Our method also requires a sufficient number of training iterations to ensure the prototypes are close enough to certain image patch features. Since most of the deep metric learning methods tend to overfit the training dataset and degrade its accuracy the longer it is trained(Kim, 2021), we need some trick such as regularization(King 2021) or distillationPPrkk, 2019) from a well-generalizzble teacher in order to achieve high accuracy in the image retrieval settings with our method. Verifying the effectiveness of such tricks is outside the scope of this paper and we leave to future work. Please note that the number of training iterations of our method is not higher than that of the other variants of ProtoPNet. Moreover, the training time of our method (Resnet 50 on the CUB200-2011 dataset) is nearly four hours, which is almost the same ProtoPool in experimental settings (one RTX3090 GPU).
Conclusion. In this paper, we proposed method in which KNN classifier is combined with the "this looks like that' framework. Extensive experiments on multiple open datasets showed that the proposed method achieved competitive results with state-of otate-of-the-art the-art ProtoPNet variant, thus demonstrating the effectiveness of utilizing similarity- based classifiers in the 'this looks like that' framework.

---

Published as a conference paper at ICLR 2023
REFERENCES
D. Alvarez-Melis and T. S. Jaakkola. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 7786-7795, 2018.
D. Bau, B. Zhou, A. Khosla, A. Oliva, A. Torralba Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 6541-6544 2017.
C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin and K. Su. This looks like that: Deep learning for in- terpretable image recognition. In Advances in Neural Information Processing Systems (NeurIPS), pages 8928-8939, 2019.
M. Cuturi. Sinkhorn distances: lightspeed computation of optimal transportation distances. In Ad- vances in Neural Information Processing Systems (NeurIPS), pages 2292-2300 2013
J. Deng, J. Guo, N. Xue and S. Zafciriou. Areface: Additive angular margin loss for deep face recog- nition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4685-4694, 2019.
Jon Donnelly, Alina Jade Barnett, Chaofan Chen Deformable protopnet: an interpretable image classifier using deformable prototypes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1225-10227,,000
M. Eisenberger, A. Toker, L. Leal-Taixé, F. Bernard and D. Cremers. A Unified Framework for Implicit Sinkhorn Differentiation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 509-518, 2022
M. Hamilton, S. Lundberg, S. Fu, L. Zhang, W. T. Freeman. Axiomatic explanations for visual search, retrieval, and similarity learning. In Proceedings of the International Conference on Learn- ing Representations (ICLR), 2022.
E. Hernandez and S. Schwettmann, D. Bau, T. Bagashvili, A. Torralba and J. Andreas. Natural Language Deseriptions of Deep Visual Features. In Proceedings of the International Conference on Learning Representations (ICLR), pages 5875-5888, 2022.
E. Jang, S. Gu, B. Poole Categorical Reparameterizzation with Gumbel-Softmax In Proceedings of International Conference on Learning Representations (ICLR), 2017.
Jiang, C. Zhang, Q. Hou, M. Cheng, and Wei. Layercam: Exploring hierarchical class activation maps for localization. In IEEE Transactions on Image Processing 30, pages 5875-5888, 2021.
D. Kazhdan, B. Dimanov, M. Jamnik, P. Liò and A. Weller. Now you see me (cme): Concept-based model extraction. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) Workshops, vol. 2699, 2020.
M. Keswani, S. Ramakrishnan, N. Reddy, V. N. Balasubramaniaa. Proto2proto: can you recognize the car, the way do? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10233-10243, 2022.
A. Khosla, N. Jayadevaprakash, B. Yao, and L. Fei-Fei. Novel Dataset for Fine-Grained Image Categorization. In First Workshop Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, 2011.
S. Kim, D. Kim, M. Cho, S. Kwak Proxy anchor loss for deep metric learning Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3238-3247, 2020.
Y. Kim and W. Park. Multi-level distance regularization for deep metric learning. In the thirty-fifth AAAI conference on Artificial Inteligence, pages 1827-1835, 2021
K. Kobs, M. Steininger, A. Dulny, A. Hotho. Do different deep metric learning losses lead similar learned features? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10644- 10644-10654, 2021.
10

---

Published as a conference paper at ICLR 2023
P. W. Koh, T. Nguyen. Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, P. Liang. Concept bottleneck models. In Proceedings of the 37th International Conference on Machine Learning, pages 5338- 5348, 2020.
Krause, M. Stark, Deng, and L. Fei-Fei. 3d object representations for lin-grained categoriza- tion. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
S. M. Lundberg and S. Lee A Unified Approach Interpreting Model Predictions. In Advances in Neural Information Processing Systems (NeurIPS), pages 4765-4774, 2017.
H. Luo, Y. Gu, X. Liao, Lai and W Jiang. Bag of tricks and strong baseline for deep person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR) Workshops, pages 4321-4329, 2019
M. Masana, Idoia Ruiz, J. Serrat, J. Van De Weijer and A.M. Lopez Metric learning for novelty and anomaly detection. In British Machine Vision Conference (BMVC), 2018.
P. Mettes, E. Pol, C. Snoek. Hyperspherical Prototype Networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 1487-1497, 2019.
M. Nauta, R. van Bree and C. Seifert. Neural prototype trees for interpretable fine-grained image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 14933-14943 2021.
W. Park, D. Kim, Y. Lu, M. Cho Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3967-3976, 2019.
K. Roth, T. Milbich, Sinha, P. Gupta, B. Ommer and P. Cohen. Revisiting training strategies and generalization performance in deep metric learning. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 8242-8252, 2020.
C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. In Nature Machine Intelligence, vol. 1, no. 5, pages 206-215, 2019.
D. Rymarczyk, E.. Struski, J. Tabor, B. Zieliński. Protopshare: Prototypical parts sharing for simi- larity discovery in interpretable image classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1420-1430, 2021.
D. Rymarczyk, L. Struski, M. Górszczak, K. Lewandowska, J. Tabor, B. Zieliński. Protopool: inter- pretable image classification with differentiable prototypes assignment. Proceedings of Euro- pean Conference on Computer Vision (ECCV), 2022.
R. R. Selvaraju, M. Cogswell, A. Das, Vedantam, Parikh and D. Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 618-626, 2017
L. S. Shapley. Notes on the n-person game i:: The value of n-person game, 1951.
C. Wah, S. Branson, P. Welinder, Perona, S. Belongie. The caltech-ucsd birds-200-2011 dataset.
In Computation and Neural Systems Technical Report, 2011.
Wang, H. Liu, X. Wang, L. Jing. Interpretable image recognition by constructing transparent embedding space. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 895-904, 2021.
X. Wang, X. Han, W. Huang. D. Dong, M. R. Scott. for deep metric learning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5022-5030, 2019
loss with general pair weighting
Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, S. Diego and P. Perona. Caltech- uesd birds 200. In California Institute of Technology, CNS-TR-201, 2010.

---

Published as a conference paper at ICLR 2023
C. Y. Wu, R. Manmatha, A. J. Smola and P. Krahenbuhl. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2840-2843 2017.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva and A. Torralba. Learning deep features for discrim- inative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2921-2929, 2016
A REVISITING PROTOPNET
In this section, we briefly revisit the origins of ProtoPNet and elaborate on why it cannot directly utilize similarity-based classifiers.
ProtoPNet first transforms an input image Ha into a feature map Z, with its feature extractor F. After the transformation, the similarity between the prototypes (i I.... and the image patch features contained in the feature map Zu are calculated. The maximum similarity value defined as the similarity of the input image za to the prototypes.
Sa,i max Sim(z,p.).
(15)
=EZ.
Here, we denote the similarity the input image Ta as 8a and its component corresponding to P: as Sa,i- Sim is the function that calculates the similarity between the feature vectors and the prototypes. Many functional forms have been proposed for Sim (Chen, 2019; Nauta, 2021; Wang, 2021; Donnelly, 2022), and in this study, we use cosine similarity. As stated in the main paper, the inference is conducted on the basis of this similarity.
The training process of ProtoPNet is divided into two steps: (1) training the feature extractor and the prototypes and (2) fine-tuning the classification layer. In the following, we mainly describe the training of the feature extractor and the prototypes. In this step, the classification layer weights are fixed by the predefined relationship between the class labels and prototypes, and the training is conducted to minimize the loss function composed by classification loss, cluster loss, and separation loss. Classification loss (typically, cross-entropy loss) is utilized to train the model so that it can solve the classification task. Cluster loss used to link the image patch features and the prototypes within the same class, and the separation loss is vice versa. In ProtoPNet, the cluster loss and the separation loss are defined as
1 min min D(z,pi), Leep Leter
Σ min min D(z, (:).
(16)
iE =EZ
sep zEZ,
Here, we denote the function that calculates the (Euclidean) distances between the image patch features and the prototype p as ((z, p). We also denote the index sets of the prototypes belonging to the class label ye as Pt. and vice versa as p As explained above, ProtoPNet needs to define which prototype contributes which class logits, and this definition enables each of the prototypes to represent useful image features for recognizing the class labels. This definition also utilized in calculating the cluster loss. However, in the training for the similarity- classifiers, this predefined relationship can no longer be used. Thus, ProtoPNet cannot directly utilize a similarity- based classifier for its final classifier. To address this problem, our proposed training process features a novel loss function that replaces the cluster loss in ProtoPNet (as described in Sec. 2.2.3).
After the training, each prototype is projected onto the nearest image patch features in the training set. This makes the prototypes transparent and enables us to interpret their meanings. Thus, the case- based interpretability achieved by classifying samples based on the similarity of the input samples to the prototypes. Note that the interpretability of the classification layer is the only requirement for achieving case-based interpretability.
B DIFFERENTIABILITY OF PROPOSED CLUSTER LOSS
When calculating the cluster loss, we use the argmax function when sampling the prototypes (Eq- 2) to strictly set the affiliation of the prototypes that should not be included to zero. Therefore, the
12

---

Published as a conference paper at ICLR 2023
affiliation of the prototypes each class P(y, p.) is not differentiable with respect to the model pa- rameter. In addition, since it consumes lot of memory to obtain the gradient through the Sinkhorn- Knopp algorithm with the automatic differentiation, we treat T2, as the constant and use the gradient through Ca, to train the model parameters. These problems for obtaining the gradients of Tæ, with respect to the model parameters can be solved by replacing the argmax function with softmax and providing the explicit derivative of the Sinkhorn-Knopp algorithm (Fisenberger, 2022). Validating these solution is outside the scope of this paper and is left for future work.
C IMPLEMENTATION DETAILS
We conducted our experiments based on the implementation of Roth (2020) and used VGGI6, VGG19, Resnet34, Resnet50, R5snet152, Densenet121 21, and Densenet161 for our model backbone in this paper. Following previous works (Nauta, 2021; Donnelly, 2022; Rymarczyk, 2022), we uti- lized the models pretrained on i-Naturerist 2017 when conducting experiments on CUB200-2011 and using ResNet50 for the model backbone. We also used the models pretrained on ImageNet for the other experimental settings. The size of the input images was transformed into 224x 224, so the resolution of the feature map output from the model backbone was 7x7. Following previous studies (Rymarczyk, 2022; Nauta, 2021), we reduced the number of channels in the feature map to 128 for the experiments on Stanford Cars and to 256 for the other datasets by applying a Xx] convolutional layer. After the channel reduction, we calculated the similarity between the prototypes and the im- age patch features contained in the feature map, and the maximum values of the similarity among the feature map were output the similarities of the input samples to the prototypes. These sim- ilarities were then used to calculate Tatsk during the training and to retrieve KNN samples during the inference. We also applied Global Average Pooling (GAP) and a linear layer to the feature map output by the model backbone and obtained the feature vectors, where the number of feature vector channels was set to the same as that of the prototypes. Note that these feature vectors were not used during the inference, though we did use them during the training to calculate Lauz. When cropped images were used, we followed previous work (Rymarczyk, 2022) and set the number of prototypes to 195 for Stanford Cars and to 202 for CUB200-2011. We also set the number of prototypes to 512 on all datasets when we used the full images.
During the training, we used the Adam optimizer and set the learning rate to le-5 for the model backbone and to le-3 for the other layers. For data augmentation, we used RandomPerspective, ColorJitter, RandomHorizontall
RandomAffine, and RandomCrop following ProtoTree (Nauta, 2021). When testing with cropped images, we simply resized them to 224x224, and when using full images, added RandomResizedCan training. We also resized the short side of the images to 256 and then cropped the center to resize them to x24x 224 when testing with the full images. Further, to reduce the complexity during train- ing, we first initialized the prototypes so that they were uniformly placed on the hyper-sphere and fixed them (Mettes, 2019), which means they were updated only when we projected them onto cer- tain image patch features in the training set after initialization. The number of epochs was set to 140, 140, and 60 for the experiments on Stanford Cars, CUB200-2011, and Stanford Dogs, respectively. We constructed the minibatch so that it contained 56 classes and two images per class. When we used 1enseNet161 1 for our model backbone, due to limitations on the GPU memory, the minibatch contained 42 classes and two images per class. Note that we report the average experimental results with three different seeds in this paper.
and set the minimum scale parameter to 0.6 during the
D FURTHER EXPERIMENTAL RESULTS
In this section, we explain the experimental results we omitted from the main paper due to space limitations.
D. ABLATION STUDY ON THE CLUSTER LOSS
In this subsection, we first describe the experimental settings of the ablation study the proposed cluster loss and then present the results. The difference of each setting here can be described as the difference in how much the image patch features are linked each of the prototypes, i.e., the
13

---

Published as a conference paper at ICLR 2023
difference in T in Eq. 9 prototypes are linked to the prototypes in all of the settings. Table 6 summarizes the settings in the ablation study that are relevant to the difference in Ta, as described below.
Eq. 11. This is because the image patch features nearest to the
First, 'Just take the most similar' is adopted for the baseline, which simply links the im- age patches to the nearest prototypes. This setting is equivalent to setting Te to 1(; arg arg,,, (min.ez, || Pj())). In '+ Sampling', the image patch features are linked the pro- totypes by using the estimated attribution of the prototypes to each class. Here, the attributions are estimated on the basis of the similarity difference of each sample in the minibatch to the prototypes. Specifically, we use the average of the sampling (Eq. 4) results within the samples that have the same class labels. Thus, in the setting of '+ Sampling'. Tw. is defined as Eya. ps) Ej Elya, by using E(y.p) p.) defined in Eq. 4. Here, we divide by Σ; E(ya:Ps) so that the cluster loss weight added to each sample is the same. In '+ De-bias', we use the de-biased attribution (Eq. 5) instead of the average of the sampling results, i.e., we define T as E(Wa,P:)/E, E(y.,Pi). where EVy, is defined in Eq. 5. Next, in '+ Linear assignment', we modify the attribution of the prototypes to each sample to account for the absence of the prototypes due to differences in the object views.
Note that we modified the attribution by solving the linear assignment problem, so in the setting of '+ Linear assignment', we define 22, as in Eq. 10. Finally, in + Maximize the gap (full)', we redefine the cluster loss from Eq. 9 to Eq. 11. Note that the definition of T is the same as in ++ a,
Linear Assignment'.
The results of the ablation study are shown in Table 7. We adopted ResNet50 for the model backbone and used the full images of CUB200-201 in this experiment. As discussed in Sec. C, we fixed the prototypes on the hypersphere and thus the accuracy does not vary among the experimental settings. However, without debiasnin, i.e., in the 'Just take the most similar' and '+ Sampling' settings, image patch features are linked to only few of the prototypes by the cluster loss, and as a result, they are not linked to the other prototypes and are not distributed over the hypersphere, which means the prototypes after the projection are also not distributed on the hypersphere. This can be confirmed by the fact that the variance of the prototypes on the hypersphere these case is much smaller compared to the other settings. Here, we defined the variance of the prototypes on the hypersphere as
Σ P: PEP
(17)
Since we initialized the prototypes so that their variance on the hypersphere would be maximized, the small value of the variance observed here suggests the destruction of the original learned structures by projecting the prototypes onto the image patch features.
We can see the accuracy is degraded when we adopted '+ Sampling' and '+ De-bias', which means the inconsistency of attachment between the image patch features and the prototypes in these set- tings. However, the accuracy is recovered while maintaining the variance high when adopt '+ Linear assignment', which confirms that the image patch features were properly linked to the pro- totypes by modifying their attributions to account for the differences in object views by solving the linear assignment problem. Finally, in '+ Maximize the gap (full)', we can see that the vari- ance further increased from that in '+ Linear assignment'. The reason for this increase may be that maximizing the gap also acted as separation loss and thereby led to the dispersal of image patch features over the hypersphere. Our findings here demonstrate the effectiveness of each component of the proposed cluster loss.
Table 6: Summary of experimental settings of ablation study on cluster loss.
T
Just take the most similar
1(1 = arg min, (min zZ.
1 Pyll)
Sampling
Elya: pi)/ Σg Elya,Pi)
+ Debias
E(ya: (tya.p)// px)/E; E(ya,P))
+ Linear assignment
arg min a.: Σa, T Ca, i + ITy To
T
s.t. E, 1 = y), Σ. ". = P(y.p.) E, 1 = v)
14

---

Published as a conference paper at ICLR 2023
Table 7:
Results of ablation study cluster loss with CUB200-2011, We used ResNet50 for
model backbone and Margin loss for Lauz.
Top-1
1 Accuracy
Variance of the prototypes
k=1
k=3
k=5
Just take the most similar
87.0%
87.1%
87.3%
0.6624
Sampling
86.2%
86.6%
86.7%
0.6646
Debias
86.7%
87.0%
87.0%
0.7267
+ Linear assignment
86.9%
87.1%
87.2%
0.7211
Maximize the gap (full)
87.0%
87.1%
87.2%
0.7489
D.2 FURTIER EXAMPLES OF THE INTERPRETATION OF THE PROPOSED METHOD
Examples of the interpretation of our method are shown in Figs. and 5. Here, in both figures, we used randomly selected samples and negative classes.
15

---

Published as a conference paper at ICLR 2023
is classified into "Least Tern" because looks like
rather than
that
looks like
looks like
looks unlike
0.379
looks like
looks like
looks unlike
0.214
looks like
looks like
looks unlike
0.199
Difference: 5.96
classified into 'Black and White Warbler" because looks like
rather than
that
looks unlike
looks unlike
looks like
0.152
looks like
looks like
looks unlike
0.141
looks unlike
looks unlike
looks like
0.134
Difference: 4.29
classified into "GMC Canyon Extended Cab 2012" because it looks like
rather than
In that
looks like
looks like
looks unlike
0.260
looks like
looks like
looks unlike
0.204
looks unlike
looks unlike
looks like
0.182
Difference: 3.60
Figure 4: Examples of the interpretation of the proposed method. Red bounding boxes surround the areas where the prototypes are activated the most.
16

---

Published as a conference paper at ICLR 2023
classified into "Hyundai Santa SUV 2012" because looks like
rather than
that
looks like
looks like
looks unlike
0.175
looks like
looks like
looks unlike
O. 158
looks unlike
looks unlike
looks like
0.14 143
Difference: 2.67
classified into "Kuvasz" because it looks like
rather than
that
looks like
looks like
looks unlike
0.410
looks unlike
lcoks unlike
looks like
0.239
looks unlike
looks unlike
looks like
0.231
Difference: 4.96
is classified into "Eskimo Dog" because looks like
rather than
In that
looks like
looks like
looks unlike
O 0.406
looks like
looks like
looks unlike
0.392
looks unlike
looks unlike
looks like
0. 0.252
Difference: 4.60
Figure 5: Examples of the interpretation of the proposed method. Red bounding boxes surround the areas where the prototypes are activated the most.
17


Reviews:
[{"review_id": "MZz-ymfhaOd", "reviewer": "Reviewer_RzJT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the effectiveness of KNN as a similarity-based classifier as opposed to the linear classifier of the original ProtoPNet. The paper also proposes a loss function to be used instead of the clustering term in the loss function of the original ProtoPNet. Another difference is that the paper does not use the separation loss. This allows the prototypes to be common among different classes which is previously addressed in ProtoPool (Rymarczyk 2022). They also use some additional losses from previous work empirically they showed that when added to the mix, they may help in some settings.  The performance of the proposed method is examined on two datasets and the results are competitive with the previous variants of ProtoPNet.", "summary_of_the_review": "I have major concerns and I hope the authors can provide clarifications. ", "strength_and_weaknesses": "+The paper adapted the general architecture of ProtoPNet (chen 2019) to work with a KNN rule. Previously we have seen ProtoTree which uses a decision tree instead of a linear layer. The current paper uses KNN.\n\n-From algorithmic perspective, what is the advantage of using ProtoKNN over other methods that are not based on KNN? The authors listed extending the ProtoPnet to use KNN as a contribution. But, what are the advantages of using KNN with ProtoPNet compared to other variants? In the introduction, authors briefly mentioned \"situations where unknown classes or anomalies exist. But the rest of the paper does not provide any further explanation or any experiment to support the effectiveness of the proposed method in those situations.\n\n-The method is not properly described. See my specific questions below. \n\n-A lot of pairwise distances between images patch features are required. An analysis of the training and inference time is required. The performance of the proposed method is on par with Deformable ProtoPNet (Donnelly, 2022), so the training time becomes important. The authors provided the results of ensemble of Deformable ProtoPNet and all other comparison methods but not for their own method. Is it because of the time complexity of training the proposed method that prohibited doing so?\n\n-The authors examined K=1, 3 and 5 in the KNN. In most experiments K=5 has performed better than others. This suggests that larger values of K should also be examined. \n\n-What is the summation in the denominator of eq. 7?\n\n-What is the summation in the denominator of eq. 8?\nNeither are explained other than vaguely stating they are for “debiasing” and “normalizing”.\n\n-Eq. 9 is not properly described. What each summation is for?\n\n-The auxiliary losses are not described in the methodology. Their definitions are provided in the ablation analysis in Sec 3.2. I suggest moving their description to where they are introduced.\n\n-Comparison methods have been used along with different backbones. ProtoPNet and Deformable ProtoPNet are used with VGG19, Res152 and DenseNet161 backbones, but ProtoTree and ProtoPool are used with Res50 backbone. Therefore it is not possible to compare the results. Why? Also why the results of the proposed method for Res152 is inferior to R50?  \n\n-The results of the ablation analysis in Table 7 show that the sampling, debiasing and linear assignment steps have no significant effect on the accuracy. The provided justification is based on variance of prototypes. Why increasing variance from 0.66 to 0.74 is important? No qualitative results are provided to shed light on this. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GqmIvVYOoBM", "reviewer": "Reviewer_94zB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an extension to ProtoPNet that utilizes KNN classifiers. To do this the paper replaces ProtoPNet loss with 3 losses:\n\na) A classification loss $L_{task}$ eq 3 which replaces the regular cross-entropy loss.\n\nb) Cluster loss $L_{clst}$ summarized in fig 2; which first calculates the affiliation of the prototypes to each class than links the prototypes and image patch features based on the affiliation.\n\nc) An auxiliary loss  $L_{aux}$ proposed in the context of deep metric learning by Wu,2017.\n\nThe total loss is the sum of all three losses with hyperparameters.\n\nFor label prediction top K samples from the training set based on euclidean distance and the majority vote is used.\n", "summary_of_the_review": "Overall empirical results show that ProtoKNN produces better results from baselines and the explanations produced by ProtoKNN are very intuitive. However, my main concern is the cost of ProtoKNN and some clarity issues in the writing.", "strength_and_weaknesses": "Strength:\n--\n\n- Empirical evidence shows that ProtoKNN outperforms other baselines on multiple datasets and different architectures.\n- Counterfactual explanations produced by ProtoKNN show both positive and negative examples.\n\n------------------\n\nWeakness:\n--\n- Training ProtoKNN is non-triaval.\n- Method is quite expensive in inference time, especially for large datasets (ignoring any memory issues).\n- No discussion of the computational cost  (training time etc..) was included.\n---------------------\nClarifications:\n-\n   - It is not clear why using $L_{task}$ is better than using cross-entropy.\n   - I am a bit confused about how the class label is chosen, the euclidean distance between samples is calculated between the raw samples or the prototypes?\n  - I really appreciate figure 2 which tries to summarize the cluster loss, however, it is not very clear what do the color bars stand for.\n  - Since  $L_{aux}$ is used in the main loss function maybe include the actual loss equation in the main text.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "4Tiolsuol5e", "reviewer": "Reviewer_g4E3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims at training deep learning models with interpretability. The paper first studies the existing ProtoPNet models, which calculate the similarity of the input to the prototypes and classifies samples on the basis of the similarity. The paper points out the issue of the vanilla ProtoPNet model -- ProtoPNet is not able to utilize any classifiers other than the linear classifier. To address this issue, the paper extends ProtoPNet and proposes ProtoKNN. The proposed ProtoKNN method does not require predefined relationship between classes and prototypes during training.The paper examines the effectiveness of similarity-based classifiers in the “this looks like that” framework by using ProtoKNN. The experiments on several open datasets show ProtoKNN achieves better or similar results compared to previous approaches.\n", "summary_of_the_review": "In summary, I think this paper has clear strengths and weaknesses. I would like to assess it as a paper around the acceptance threshold. ", "strength_and_weaknesses": "Strengths:\n- The paper studies an important problem and addresses an issue of an existing popular approach. I generally appreciate the efforts of this paper.\n- The motivation of this paper is clear and the paper is well-positioned; the related works have been discussed well.\n- The method makes sense to me. The proposed cluster loss is novel and interesting.\n- The experimental results are strong.\n- The paper has discussed the limitations (e.g., memory efficiency, differentiability) fairly well. \n\n\nWeaknesses:\n- The paper can be presented more clearly. (1) The high-level idea of the method should be discussed in the introduction, while the current introduction only talks about motivation of the research; (2) section 2.3.2 is lengthy and not easy to follow; (3) figure 1 and figure 2 are hard to follow and there are nearly no captions for the figures; (4) the paper does not talk about the auxiliary loss, which can really confuse readers; (5) figure 3 is not easy to understand.\n- Although in terms of experimental results, the paper is similar or stronger than previous approaches, I think the experiment section can be made more informative and insightful. For example, there is no analysis on the proposed cluster loss; also, I expect to see experiments where unknown class samples exist, such that similarity-based classifiers can be fully utilized.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-n6ea7TNain", "reviewer": "Reviewer_o3Fc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a ProtoKNN model, which uses k-nearest neighbor (kNN) on prototype similarities to classify an input image. In particular, the proposed model uses a ProtoPNet backbone (with cosine similarity), and classifies an input image by comparing its prototype similarities with those of the training images. After finding the top k training images whose prototype activations are similar to the prototype activations of the input image, the model uses the majority class label of those k images as the predicted class of the input image. To train their ProtoKNN model, the authors used a loss function consisting of: (1) a task loss, which encourages an input image to have similar prototype activations as another image of the same class, and to have different prototype activations from an image of a different class; (2) a modified cluster loss, which encourages each training input to have at least one patch close to a prototype that is affiliated with its own class; and (3) an auxiliary loss (e.g., margin loss, proxy anchor loss) from deep metric learning. The authors compared their ProtoKNN with ProtoPNet, Deformable ProtoPNet, and other variants of ProtoPNet models, and claimed that their model is able to achieve a higher accuracy even without ensembling. In addition, the authors also demonstrated the explanations generated by their model (Figure 3).", "summary_of_the_review": "Based on the strengths and weaknesses discussed above, I believe that the paper has some value, but needs improvement especially in terms of clarity.", "strength_and_weaknesses": "Strengths:\n\n- Interpretability: The proposed model is interpretable. Its predictions can be explained in terms of training images with similar prototypical activations. In particular, the proposed model can explain why an input image is similar to another image of the predicted class, but is dissimilar to an image not of the predicted class.\n- Accuracy: The proposed model improves accuracy over existing variants of ProtoPNet models.\n\nWeaknesses:\n\n- Technical soundness: In equation (3), how do you choose the hard-positive and the hard-negative samples for computing the task loss?\n- Clarity: The description of cluster loss is difficult to follow. In addition, Figure 3 is confusing -- I had a hard time trying to understand it.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster