ProtoAttend: Attention-Based Prototypical Learning
Authors: Sercan O. Arik
Tomas Pfister
Venue:  (2020)
Keywords: Interpretability
sample-based explanations
prototypes
confidence estimation

Abstract: We propose a novel inherently interpretable machine learning method that bases decisions on few relevant examples that we call prototypes. Our method, ProtoAttend, can be integrated into a wide range of neural network architectures including pre-trained models. It utilizes an attention mechanism that relates the encoded representations to samples in order to determine prototypes. The resulting model outperforms state of the art in three high impact problems without sacrificing accuracy of the original model: (1) it enables high-quality interpretability that outputs samples most relevant to the decision-making (i.e. a sample-based interpretability method); (2) it achieves state of the art confidence estimation by quantifying the mismatch across prototype labels; and (3) it obtains state of the art in distribution mismatch detection. All this can be achieved with minimal additional test time and a practically viable training time computational cost.

Under review as a conference paper at ICLR 2020
PROTOATTEND: ATTENTION-BASED PROTOTYPICAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a novel inherently interpretable machine learning method that bases decisions on few relevant examples that we call prototypes. Our method, ProtoAt- tend, can be integrated into a wide range of neural network architectures including pre-trained models. utilizes an attention mechanism that relates the encoded representations to samples in order to determine prototypes. The resulting model outperforms state of the art in three high impact problems without sacrificing accu- racy of the original model: (1) it enables high-quality interpretability that outputs samples most relevant to the decision- makinn (i.e. a sample- sample-based based interpretability method); (2) it achieves state of the art confidence estimation by quantifying the mismatch across prototype labels; and (3) it obtains state of the art in distribution mismatch detection. All this can be achieved with minimal additional test time and a practically viable training time computational cost.
INTRODUCTION
Deep neural networks have been pushing the frontiers of artificial intelligence (AI) by yielding excellent performance in numerous tasks, from understanding images (He et al., 2016) to text (Conneau et al. 2016). Yet, high performance is not always a sufficient factor as some real- world deployment scenarios might necessitate that an ideal AI system is 'interpretable' such that builds trust by explaining rationales behind decisions, allow detection common failure cases and biases, and refrains from making decisions without sufficient confidence. In their conventional form, deep neural networks are considered as black-box models they are controlled by complex nonlinear interactions between many parameters that difficult to understand. There are numerous approaches, (Kim et al. 2018 Erhan et al.) 2009 Zeiler & Fergus] 2013; Simonyan et al. (201)). that bring post-hoc explainability of decisions to already-trained models. Yet, these have the fundamental limitation that the models are not designed for interpretability. There are also approaches on the redesign of neural networks towards making them notable ones include sequential attention (Bahdanau et al. (2015), capsule networks (Sabour et al. 2017), and interpretable convolutional filters (Zhang et al. 2018).
as in this paper. Some
0.6 0.3
Weights tesss
Prototypes
Classifier
"bird"
10.8 Confidonce
Darabase
Figure 1: ProtoAttend bases the decision few prototypes from the database. This enables interpretability of the prediction (by visualizing the highest weight prototypes) and confidence estimation for the decision (by measuring agreement across prototype labels).
We focus on inherently-interpreann totypical learning. Prototypical learning decomposes decision making into known samples (see Fig. referred here as prototypes. We base method on the principle that prototypes should constitute a minimal subset of samples with high interpretable value that can serve as a distillation or condensed view of a dataset (Bien & Tibshirani, 2012). Given that the number of objects human interpret is limited (Miller 1956). outputting few prototypes can be an effective approach for humans to understand the Al model behavior. In addition such interpretability, prototypical learning:
deep neural network modeling with the foundations of pro-

---

Under review as a conference paper at ICLR 2020
(I) provides efficient confidence metric by measuring mismatches in prototype labels, allowing performance to be improved by refraining from making predictions in the absence of sufficient confidence, (2) helps detect deviations in the test distribution by measuring mismatches in prototype labels that represent the support of the training dataset, and (3) enables performance in the high label noise regime to be improved by controlling the number of selected prototypes. Given these motivations, prototypes should be controllable in number, and should be perceptually relevant to the input in explaining the decision making task. Prototype selection in its naive form is computationally expensive and perceptually challenging (Bien & Tibshirani 2012). We design Proto/Ateed to address this problem in an efficient way. Our contributions can be summarized as follows:
1. We present principles that can be guiding for the design of inherently- interrtttaeee models based on sample-based interpretabilityy.
2. We propose novel method, ProtoAttend, for selecting input-dependent prototypes based on an attention mechanism between the input and prototype candidates. ProtoAttend is model-agnostic and can even be integrated with pre-trained models.
3. Proto-Attend allows interpreting the contribution of each prototype via the attention outputs.
4. For 'condensed view', we demonstrate that sparsity in weights can be efficiently imposed via the choice of the attention normalization and additional regularization.
5. On image, text and tabular data, we demonstrate the four key benefits of ProtoAttend: interpretabil- ity, confidence control, diagnosis of distribution mismatch, and robustness against label noise. ProtoAttend yields superior quality for sample-based interpretability, better-calibrated confidence scoring, and more sensitive out-of-distribution detection compared to alternative approaches. PtonoA enables all these benefits via the same architecture and method, while maintaining comparable overall accuracy.
2 RELATED WORK
The principles of ProtoAttend are inspired by (Bien & Tibshirani 2012). They formulate prototype selection
Prototypical learning:
integer program and solve it using a greedy approach with linear program relaxation. It seems unclear whether such approaches can be efficiently adopted to deep learning. (Chen et al. (2088) and (Li et al. 2018) introduce prototype layer for interpretability by replacing the conventional inner product with a distance computation for perceptual similarity. In contrast, our method uses an attention mechanism quantify perceptual similarity and can choose input-dependent prototypes from a surge-scale candidate database. (Yeh et al. 2018) decomposes the prediction into linear combination of activations of training points for interpretability using representer values. The linear decomposition idea also exists in ProtoAttend, but the weights are learned via an attention mechanism and sparsity encouraged in the decomposition. In (Koh Liang 2017), the training points that are the most responsible for given prediction are identified using influence functions via oracle access to gradients and Hessian-vector products.
Metric learning aims to find an embedding representation of the data where similar data points are close and dissimilar data pointers are far from each other. ProtoAttend is motivated by efficient learning of such embedding space which can be used to decompose decisions. Metric learning for deep neural networks is typically based on modifications to the objective function, such as using triplet loss and N-pair loss (Sohn) 2016 Cui et al.) [2016] HHofeer & Ailon 2014). These yield perceptually meaningful embedding spaces yet typically require a large subset of nearest neighbors to avoid degradation in performance (Cui et al. 2016). (Kim al. (018)) proposes deep metric learning framework which employs an attention- based ensemble with a divergence loss so that each learner can attend to different parts of the object. Our method has metric learning capabilities like relating similar data points, but also performs well on the ultimate supervised learning task.
Metric learning:
Afention-based few-shot learning: Some of our inspirations are based on recent advances in attention- based few-shot learning. In (Vinyals et al. (016).. an attention mechanism is used to relate an example with candidate examples from a support set using a weighted nearest-neighhor classifier applied within an embedding space. In (Ren et al.) 2018), incremental few-shot learning is implemented using an attention attractor network on the encoded and support sets. In (Snell et al. 2017), a non-linear mapping is learned to determine the prototype of a class as the mean of its support set in the embedding space. During training, the support set is randomly sampled to mimic the inference task. Overall, the attention mechanism in our method follows related principles but fundamentally differs in that shot learning aims for generalization to unseen classes whereas the goal of method is robust and interpretable learning for seen classes.
2

---

Under review as a conference paper at ICLR 2020
Uncertainty and confidence estimation: ProtoAttend takes novel perspective on the perennial problem of quantifying how much deep neural networks' predictions can be trusted. Common approaches are based using the scores from the prediction model, such the probabilities from the softmax layer of a neural network, yet it has been shown that the raw confidence values are typically poorly calibrated (Guo al., (017). Ensemble of models (Lakshminarayaaan et al.) (017) is one of the simplest and most efficient approaches. but significantly increases complexity and decreased interpretability. In (Papernot & McDaniel (018), the intermediate representations of the network are used to define distance metric. and a confidence metric is proposed based on the conformity of the neighbors. (Jiang et al., 2018). proposes a confidence metric based on the agreement between the classifier and modified nearest-neighbor classifier on the test sample. In (DeVries & Taylor 2018), direct inference of confidence output considered with a modified loss. Another direction of uncertainty and confidence estimation is Bayesian neural networks that return a distribution over the outputs (Knddall & Gal 2017b) (Mullachery et al. 2018) (Kendall & Gal 2017a).
3 PROTOATTEND: ATTENTION-BASED PROTOTYPICAL LEARNING
Consider training set with samples, T = {xi, vi). Conventional supervised learning aims to learn model s (x;; S) that minimizes a predefined loss 1/ > L(y,, D; s(,; { at each iteration, where B is the batch size for training. Our goal is to impose that decision making should be based on only a small number of training examples, i.e. protolypes, such that their linear superposition in an embedding space can yield the overall decision and the superposition weights correspond to importuurooi their importance. Towards this goal, we propose defining a solution to prototypical learning with the following six principles:
vi = fx;; () encodes all relevant information of x; for the final decision. f() considers the global distribution of the samples, i.e. learns from all {x;, vs). Although all the information in training dataset is embodied in the weights the encode we construct the learning method in such a way that decision is dominated by the prototypes with high weights. ii From the encoded information, we can find decision function so that the mapping g n) is close to the ground truth ye, in consistent way with conventional supervised learning. iii. Given candidates xj to select the prototypes from, there exists weights Pij (where Pi,j 2 and Σj=1 Pt,j = 1), such that the decision g ((jj= is close to the ground truth y/i- iv. When the linear combination Σ/=1 Pt. is considered, prototypes with higher weights (e) :n). ;
Pt. (c), ;)) (where v(e) = f(x]()
Pi,j have higher contribution in the decision g e Pi v. The weights should be sparse only a controllable amount of weights pt, should be non- zero. Ideally, there exists an efficient mechanism for outputting Pt.j to control the sparsity without significantly affecting performance.
vi. The weights pt., depend on the relation between input and the candidate samples, Pt.j = r(x,, x; (c), r)), based on their perceptual relation for decision making. We do not introduce any heuristic relatedness metric such distances in the representation space, but we allow the model to learn the relation function that helps the overall performance.
Learning involves optimization of the parameters 0, r, 1 of the corresponding functions. If the proposed principles (such as reasoning from the linear combination of embeddings or assigning relevance to the weights) not imposed during training but only at inference, a high performance cannot be obtained due to the train-test mismatch, as the intermediate representations can be learned in an arbitrary way without any necessities satisfy them B] The subsequent section presents ProtoAttend and training procedure to implement it.
3.1 NETWORK ARCHITECTURE AND TRAINING
The principles above are conditioned on efficient learning of an encoding function to encode the relevant information for decision making, a relation function to determine the prototype weights, and
S represents the trainable parameters for s(: S) and is sometimes not show for notation convenience.
2 'Training of f() may also involve initializing with pre-trained models or transfer learning. For example, commonly-used distance metries in the representation spaces fail at determining perceptual relevance between samples when the model is trained in vanilla way (Sitawarin & Wagner 2019.
3

---

Under review as a conference paper at ICLR 2020
final decision making block to return the output. Conventional supervised learning comprises the encoding and decision blocks. On the other hand, it is challenging to design a learning method with a relation function with reasonable complexity. To this end, adapt the idea of attention (Corbettal & Shulman, 2002) Vaswani et al. 2017), where the model focuses on an adaptive small portion of input while making the decision. Different from conventional employment of attention in sequence or visual learning. we propose to use attention at sample level, such that the attention mechanism is used determine the prototype weights by relating the input and the candidate samples via alignment of shows the proposed architecture for training and inference. The three main blocks are described below:
their keys and queries. Fig. [
Training
Inference
Irputt Irput batch
Encoder
FC
cQuiriis
attention
Relanionnll
weights
Sparsity regulerization
Decision
Oosputt
Inpout
Encoder
FC
quaries
Relational
attinntion
values
weights
Decision
volues
values
1-0 Matching
Comfidence
losses
Prototype
lobel match
regulaniztion
Confidlncaa
Encodar
Encodar
Prototype
label match
Protecypes
Candiclate batch
Candidate
database
Thresholding
Figure 2: ProtoAttend method for training and testing. Shared encoder between input samples and the candidate samples generates input representations, that are mapped to key, query and value embeddings (with a single nonlinear layer). The alignment between keys and queries determines the weights of the prototypes, and the linear combination of the values determines the final decision. Conformity of the prototype labels is used as a confidence metric.
Encoder: A trainable encoder employed to transform B input samples (note that may be at inference) and D samples from the database of prototype candidates (note that D may be as large as the entire training dataset at inference) into keys, queries and values. The encoder is shared and jointly updated for the input samples and prototype candidate database, to learn a common representation space for the values. The encoder architecture can be based on any trainable discriminative feature mapping function, e.g. ResNet (He et al. 2016) for images, with the modification of generating three types of embeddings. For mapping of the last encoder layer to key, query and value embeddings, simply use single fully-connected layer with nonlinearity, separately for each [ For input samples, V e Bxduue and Q R Bx denote the values and queries, and for candidate database samples K(e) E Dxdae and V(e) E Dxxdout denote the keys and values. For keys and queries, separate representations as the entire system is not symmetric, there are lot of candidate samples and the model may prefer to learn the keys to arrange the representation space such that it is meaningful when their inner products with a single query are considered.
Relational attention: The relational attention yields the weight between the jth sample and 5th candidate, Pi.j via alignment of the corresponding key and query in dot-product attention form
K (c) / /
Pr.
where na( is a normalization function to satisfy pa, and Σイ= Pt.j 1 for which we consider softmax and sparsemax (Martins& & Astudillo, 2016 The choice of the normalization function an efficient mechanism to control the sparsity of the prototype weights, demonstrated in experiments. Note that the relational attention mechanism does not introduce any extra trainable parameters.
Decision making: The final decision block simply consists of linear mapping from convex combination of values that results in the output yi- Consider the convex combination of value embeddings, parameterized by a:
D ú (a(a) = -1 a E.-Prosyo
"There are other viable options for the mapping but we restrict it a single layer to minimize additional number of trainable parameters, which becomes negligible in most cases.
'e use A₁ to denote the tth row of A. " "Sparsemax encourages sparsity by mapping the Euclidean projection onto the probabilistic simplex.
4

---

Under review as a conference paper at ICLR 2020
For 0, L (ys, (0)) is the conventional supervised learning loss (ignoring the relational attention mechanism) that can only impose principles (i) and (ii), but not the principles (iii)-vi).. A high accuracy for D, (0) merely indicates that the value embedding space represents each input sample accurately. For a = 1, (y:, (1) encourages the principles (i), (iii)-(iv) but not the principles (ii) and (vi) high accuracy for yi (1) indicates that the linear combination of value embeddings accurately maps to the decision. For (vi), we propose that there should be a similar output mapping for the input and prototypes, for which we encourage high accuracy for both (0) and (1( with loss term that is mixture of L (ye, Ds (0)) and L (1s, di(()) guidance with an intermediate term, as (0.5), ), is required. Lastly, when a S 0.5, we obtain the condition that the input sample itself has the largest contribution in the linear combination. Intuitively, the sample itself should be more relevant for the output compared other samples, so the principles (iii) and (iv) can be encouraged. We propose and compare different training objective functions in Table We observe that the last four are all viable options as the training objective. with similar performance. We choose the last one for the rest of the experiments, in some cases, slightly better prototypes are observed qualitatively (see Sect. 5.2 for further discussion).
Table 1: Ablation study. Impact of various training losses on ProtoAttend with softmax attention for Fashion- -MNIST. 1 S i S N, is the training iteration index and N, is the total number of iterations.
Acc. %
Acc. %
[C]
E [C}
Training objective function
for (0)
for (1)
$=V
87y
L. (yt, Vi (0))
94.28
13.13
0.029
0.194
L (e. Me (1))
10.92
94.21
0.103
0.002
(ye, 2/ (0.5))
94.01
94.25
0.927
0.049
L (yt, Vs (0))
(yt, (I)
94.37
94.38
0.931
0.047
(1 i/N() ) (yi. (0))+
94.14
94.18
0.927
0.049
(i/N,) T. (yh, (.(1))
L (yi, (0)) (36, de (1))+
94.37
94.45
0.928
0.047
(yt, Us (0.5))
To control the sparsity of the weights (beyond the choice of the attention operation), we also propose sparsity regularization term with a coefficient Asparse in the form of entropy. Leparse (p)
- -1/B 1 Ej=1 Pt.j log(ps.j + c), where e is a small number for numerical stability. Leparse is minimized when p has only 1 non-zero value.
3.2 CONFIDENCE SCORING USING PROTOTYPES
Reliability diagram
100
7000
20
8000
5003
Accuaey
Canfilence
Figure 3: Impact of confidence on ProtoAttend accuracy. Reliability diagram for Fashion-MNIST, as in (Papernot & McDaniel 2018). Bars (left axis) indicate the mean accuracy of predictions binned by confidence; the red line (right axis) shows the number of samples across bins.
ProtoAttend provides linear decomposition (via value embeddings) of the decision into prototypes that have known labels. Ideally, labels of the prototypes should all be the same as the labels of the
For example, simply assigning non-zero weights to another predetermined class, prototypical learning method can obtain perfect accuracy, but the assignment of predetermined class would be arbitrary.
5

---

Under review as a conference paper at ICLR 2020
input. When prototypes with high weights belong to the same class, the model shall be more confident and a correct classification result is expected, whereas in the cases of disagreement between prototype labels, the model shall be less confident and the likelihood of wrong prediction is higher. With the motivation of separating correct vs. incorrect decisions via its value, we propose a confidence score based on the agreement between the prototypes:
C; = --EPiL PLJ 1(01 = -;))
j=1
where I() is the indicator function. Table shows the significant difference of the average confidence metric between correct vs. incorrect classification cases for the test dataset, as desired. In Fig. 3 the impact of confidence on accuracy is further analyzed with the reliability diagram as in (Papernot & McDaniel 2018). When test samples are binned according to their confidence, it is observed that the bins with higher confidence yield much higher accuracy. There are small number of samples in the bins with lower confidence, and those tend to be the incorrect classification cases. In Section 4.4 the efficacy of confidence score in separating correct incorrect classification is experimented in ceciidence-contrllee prediction setting, demonstrating how much the prediction accuracy can be improved by refraining from small number of samples with low confidence at test time.
To further encourage confidence during training, we also consider a regularization term Leon (p) = -1/B EI/ Ep- Pt.j I(y, ()) yı) with a coefficient Aconf. Leony is minimized when all proto-
types with Pi.j > 0 are from the same ground truth class with output
4 EXPERIMENTS
4.1 SETUP
We demonstrate the results of ProtoAttend for image, text and tabular data classification problems with different encoder architectures (see Supplementary Material for details). Outputs of the encoders are mapped queries, keys and values using fully-connected layer followed by ReLU. For values, layer normalization (Lei Ba el al. 2016) is employed for more stable training. A fully-connected layer used the decision making block, yielding logits for determining the estimated class. Softmax cross entropy loss is used as L(). Adam optimization algorithm is employed (Kingma & Ba 2014) with exponential learning rate decay (with parameters optimized on a validation set). For image encoding, unless specified, we use the standard ResNet model (He et al., (016)). For text encoding, we use the very deep convolutional neural network (VDCNN) (Conneau et (0166) model, inputting sequence of raw characters. For tabular data encoding, we use an LSTM model (Hochreiter & Schmidhuber 1997), which inputs the feature embeddings at every timestep. See Supplementary Material for implementation details, additional results and discussions.
4.2 SPARSE EXPLANATIONS OF DECISIONS
We foremost demonstrate that our inherently-interpretbbbl model design does not cause significant degradation in performance. Table 2 shows the accuracy and the median number of prototypes required to add up to particular portion of the decisior for different prototypical learning cases. In all cases, very small accuracy gap observed with the baseline encoder that is trained in conventional supervised learning way. The attention normalization function and sparsity regularization are efficient mechanisms to control the sparsity the number of prototypes required much lower with sparsemax attention compared to softmax attention and can be further reduced with sparsity regularization (see Supplementary Material for details). With a small decrease in performance, the number of prototypes can be reduced just handful 10 There is difference between datasets, as intuitively expected from the discrepancy the degree of similarity between the intra-class samples.
Note that the gradients of this regularization term with respect to Pr.j either 0 or and is often insufficient to train the model itself from scratch. But it is observed to provide further improvements in some cases. FF.g. if the prototype weights are 10.1, 0.15, 0.05, 0.25, 0.1, 0.05, 0.28, 0.02], then 2 prototypes are required for 50% of the decision, 6 for 90% and 7 for 95%.
1oWe observe that excessively high sparsity (to yield 1-2 prototypes in most cases) may sometimes decrease the quality of prototypes due overfitting discriminative features that are less perceptually meaningful.

---

Under review as a conference paper at ICLR 2020
Table 2: ProtoAttend achieves interpretability without significant degradation in performance. Accu- racy and median number of prototypes to add up to 50%, 90% and 95% of the decision, quantified with prototype weights.
No. of prototypes
Dataset
Method
Acc. %
50 %
90 %
95 %
Baseline enc.
99.70
MNIST
Softmax attn.
99.66
365
1324
1648
Sparsemax attn.
99.69
2
4
5
Baseline enc.
94.74
Softmax attn.
94.42
712
2320
2702
Fashion-MNIST
Sparsemax attn.
94.42
4
10
11
Sparsemax attn. + sparsity reg.
94.47
1
2
2
Baseline enc.
91.97
Softmax attn.
91.69
317
1453
1898
CIFAR-10
Sparsemax attn.
91.44
5
14
16
Sparsemax attn. sparsity reg.
91.26
2
3
4
Baseline enc.
98.25
Softmax attn.
98.20
63
190
225
DBPedia
Sparsemax attn.
97.74
2
4
Baseline enc.
85.68
Sottmax attn.
85.64
2263
9610
12419
Income
Sparsemax attn.
85.58
20
57
67
Sparsemax attn. + sparsity reg.
85.41
3
6
7
Input
Prototypes
Input
Prototypes
Input
Prototypes
Input Prototypes
0:37
0.37
0,55
0.45
0:53
6
6
6
0.47
0.24
0.78
0.12
2
2
0.52
1.00
0.31
0.31
3
3
3
(a) MNIST & Fashion MNIST
(b) Fruits
Figure 4: Example inputs and ProtoAttend prototypes for (a) MNIST (with sparsemax). Fashion- MNIST dataset (with sparsemax and sparsity regularization) and (b) Fruits (with sparsemax and sparsity regularization). For MNIST & Fashion-MNIST, prototypes typically consist of discriminative features such as the straight line shape for the digit 1, and the long heels and strips for the sandal. For Fruits, prototypes often correspond to the same fruit captured from a very similar angle.
Prototypes Input
abcehas esadan anapa (born june in son salvador salvodoc) palvodocon football player vho currestly plays soys pango
footballer played
vel (nov atl) howell played with cisy-eouth feen 1953 to 1957. 0.42 prerieranip tasea and is 1957 wos city's best and fassest award. 2osell made his vel debut with the at ailda football club
nember club' is 1854 1956 ntta
In 1958 atter being sigred from tareanla In 1953.
valfgang pact
fanuary
province uneeptalia)
player eaptainiig the
0.22
included in selout 1985 fifa world li cup. despite this the defender never played ratoh for weat gereasy and had resire early because of the aftecte an injury piched up in the lste 1060s had to his gass.
0.16 beatley jumes Hoon auvembess 19821 unplish footbullu who plays tocunto lous fsso blackburn roveEs righs-Sack.
iiverpon1
gon
Figure 5: Example inputs and ProtoAtteed prototypes for DBPedia (with sparsemax). While classify- ing the inputs as athlete, prototypes have very similar sentence structure, words and concepts.
7

---

Under review as a conference paper at ICLR 2020
Inputs
Prototypes
0.34
0.22
0.06
0.07
AgR 2N
deleer 01311
Colige 4300
Fetest 20115
Zhcage J41274
Semeconlles
Sacical, JE
DArTONL Flaces 3000 CRETIAS
Sacioal, prwri Serce carried
cucatsenecep RES finlly
Vulle
201AT:/03510 C3LLE
pace
PAL4--
08catioeat50
coptiel
capiica 2014
290176 2033
2039
repliin bol g91h 1
FASlue soudae Dittab-diataa
001211 0sioed-Pcates
CACSIN CC222C Carooda
0.25
0.13
0.09
0.09
Peleclesttp
Statee
CROJDAL
CACCIAL 2048
Esited
Figure 6: Example inputs and ProtoAttend prototypes for Adult Census Income (with sparsemax and sparsity regularization), For the first example, all prototypes have similar age, two share similar education level and one has the same occupation. For the second example, three prototypes have the same occupation, all work more than 40 hours/week, and three have postgraduate education.
Figs. and 6 exemplify prototypes for image, text and tabular data. In general, perceptually-simine samples are chosen as the prototypes with the largest weights. We also compare the relevant samples found by ProtoAttend with the methods of representer point selection (Yeh et al. (018)) and influence functions (Koh Liang 2017) (see Supplementary Material for details) on Animals with Attributes dataset. As shown in Fig. 7 our method finds qualitatively more relevant samples. This case also
exemplifies the potential of our method for integration into pre-trained models by addition of simple layers for key, query and value generation.
Input
ProtoAttend
Representer point selection
Influence functions
Figure 7: Samples found by ProtoAttend vs. representer point selection Yeh et al. 2018 and
influence function (Koh & Liang 2017) for the two examples from (Yeh et al. 2018 on Animals with Attributes dataset. See Supplementary Material for more examples.
4.3 ROBUSTNESS TO LABEL NOISE
Table 3: Label noise ratio vs. accuracy for bascline encoder, dropout method (Arpit et al. 2017) (optimizing the keep probability) and ProtoAttend with sparsemax attention and sparsity regularization for CIFAR-10.
Test accuracy
Noise level
Baseline
Dropout
ProtoAttend
0.8
57.02
56.76
60.50
0.6
71.27
72.15
74.67
0.4
77.47
78.99
80.04
As prototypical learning with sparsemax attention aims to extract decision decision-making information from small subset of training samples, it can be used improve performance when the training dataset
8

---

Under review as a conference paper at ICLR 2020
contains noisy labels (see Table 3 The optimal value of Asparse increases with higher noisy label ratios, underlining the increasing importance of sparse learning.
4.4
CONFIDENCE-CONTROLLED PREDICTION
99.7
96
95
99.5
Accuracy
Accuracy
94
99.3
93
ProteAttend
ProtoAttend (1 network)
99.1
contidence
92
ensemblc (4 networks)
dkNN credioliit
trust score (1 network)
97
99
100
65
70
80 85
90
95
100
Ratio of samples %
Ratio of samples %
Figure 8: Confidence-controllee prediction. (a) Accuracy vs. ratio of samples for MNIST. We compare dkNN (Papernot & McDaniel, 2018) and prototypical learning (with softmax attention and /=01/=0..1 using the same network architecture from (Papernot & McDaniel 2018) without augmentation. (b) Accuracy vs. ratio of samples for CIFAR-TO. We compare prototypical learning (with softmax attention and (00.1) with trust score (Jiang al., 2018) and deep ensemble
Lakshminarayanan et al. 2017) methods for the same baseline encoder network architecture.
By varying the threshold for the confidence metric, a trade-off can be obtained for what ratio of the test samples that the model makes a prediction for vs. the overall accuracy obtains on the samples above that threshold 12 Figs. 8a) and ((b) demonstrate this trade-off and compare it to alternative methods. The sharper slope of the plots show that our method is superior to dkNN (Papernot & McDaniel 2018) and trust score (Jiang et al. 2018), the methods based quantifying the mismatch with nearest-neighbor samples, in terms of finding related samples. Although the baseline accuracy is higher with 4 ensemble networks obtained via deep ensemble (Lakshminarayanaal our method utilizes a single network and the additional accuracy gains by refraining from uncertain predictions is similar our approach shown by the similar slopes of the curves.
al., (017))
Overall, the baseline accuracy can be significantly improved by making less predictions. Compared to the state of the art models, our canonical method with simple and small models shows similar accuracy by making slightly fewer predictions c.g. for MNIST, (Wan et al. 2013) achieves 0.21% % error rate, that obtained by our method refraining from only 0.45% of predictions using ResNet-32 and for DBpedia, (Sachan & Petuum 2018) achieves 0.91 % error, that is obtained by our method refraining from 3% of predictions using 9-layer VDCNN. In general, the smaller the number of prototypes. the smaller the trade-off space. Thus, softmax attention (which normally results in more prototypes) is better suited for confidence-controllee prediction compared sparsemax (see Supplementary Material for more comparisons).
4.5 OUT-OF--ISTRIRUTION SAMPLES
Well-calibrated confidence scores at inference can be used to detect deviations from the training dataset. As the test distribution deviates from the training distribution, prototype weights tend to mismatch more and yield lower confidence scores. Fig. 9 (a) shows the ratio of samples above
certain confidence level as the test dataset deviates. Rotations deviate the distribution of test images from the training images, and cause significant degradation in confidence scores, well as the overall
For fair comparison, we re-optimize the learning rate parameters Note that this trade-off is often more meaningful to consider rather than the metrics based on the actual value of confidence score itself, as methods may differ in how they define the confidence metric, and thus yield very different ranges and distributions for
separate validation set.

---

Under review as a conference paper at ICLR 2020
accuracy. On the other hand, using test image from different dataset, degrade them further. Next, Fig. (b) shows quantification of out oftt----iittiutiionn detection with prototypical learning, using the method from (Hendryeks & Gimpel 2016). ProtoAttend yields an AUC of 0.838, being on par with the-state of the art approaches (Hendrycks et al.).
Ratto of samppie %
True Positive Rate
2.3
34-
Nas fon MINS
90 rot. Fachion TWcNton-NISS WNIST
180 rot. Fashion SNNES
WNIST
ProtoAtttedd with cont. reg.
CIARD (R-dmm)
Softmax baseline
04 68
Confidence
Talse Positive Rate
(a)
(b)
Figure 9: Out-of-distribution detection. (a) Ratio of samples above the confidence level for pro- totypical learning with softmax attention, trained with Fashion-MNIST, and tested on the shown datasets. E.g. if we assess the ratio of samples above confidence 0.9, is far more likely that those samples come from the same distribution with the training dataset. (b) ROC curve for in-distribution vs. out-of-distribution detection, using CIFAR-10 10 as in-distribution and SVHN as nut-of-distribation computed using the method from (Hendrycks & Gimpel| (016) and compared to the proposed base- line in (Hendrycks & Gimpel| 2016). Softmax attention and confidence regularization (Aconf = 0.1) are used.
COMPUTATIONAL COST
ProtoAttend requires only a very small increase in the number of learning parameters (merely two extra small matrices for the fully-connected layers to obtain queries and keys). However, it does require a longer training time and has higher memory requirements to process the candidate database. At inference, keys and values for the candidate database can be computed only once and integrated into the model. Thus, the overhead merely becomes the computation of attention outputs (e.g. for CIFAR-10 10 model, the attention overhead inference is less than 0.6 MFLOPs, orders of magnitude lower than the computational complexity of a ResNet model). During training on the other hand, both forward and backward propagation steps for the encoder need to be computed for all candidate samples and the total time is higher (e.g. 4.45 times slower to train until convergence for CIFAR-10 compared to the conventional supervised learning). The size of the candidate database is limited by the memory of the processor, so in practice we sample different candidate databases randomly from the training dataset at each iteration. For faster training, data and model parallelism approaches are straightforward to implement e.g., different processors can focus on different samples, or they can focus on different parts of the convolution inner product operations. Further computationalll--fiico approaches may involve less frequent updates for candidate queries and values.
CONCLUSIONS
We propose an attention-based prototypical learning method, ProtoAttend, and demonstrate its usefulness for wide range of problems on image, text and tabular data. By adding relational attention mechanism to an encoder, prototypical learning enables novel capabilities. With sparsemax attention, it can base the learning on few relevant samples that can be returned at inference for interpretability, and can also improve robustness to label noise. With softmax attention, it enables confidence-contred
prediction that outperform state of the art results with simple architectures by simply making slightly fewer predictions, as well as enables detecting deviations from the training data. All these capabilities achieved without sacrificing overall accuracy of the base model.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Devansh Arpit, Stanislaw Jastrzkbski, Nicolas Ballas, David Krueger, Emmanuel Bengio, et al. A Closer Look Memorization in Deep Networks. vrXiv:170.0559%/ 2017.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
Jacob Bien and Robert Tibshirani. Prototype selection for interpretable classification. 1rX1v:1202.5933, 2012.
Chaofan Chen, Oscar Li, Alina Barnett, Jonathan Su, and Cynthia Rudin. This looks like that: deep learning for interpretable image recognition. arXiv: 10574, 2018.
Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann LeCun. Very deep convolutional networks for natural language processing. arXiv: /60/66666667777 2016.
Maurizio Corbetta and Gordon L. Shulman. Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews Neuroscience, 3:201-215, 2002.
Yin Cui, Feng Zhou, Yuanqing Lin, and Serge J. Belongie. Fine-gran Fine-grained categorization and dataset bootstrapping using deep metric learning with humans in the loop. CVPR, 2016.
Terrance DeVries and Graham W. Taylor. Learning Confidence for Out-of-Distributio Detection in Neural Networks. (xXiv:182.00486. 2018.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. In Technical report, 2009.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. arXiv: 07599, 2017.
H A Haenssle, C Fink, R Schneiderbauer, F Toberer, T Buhl, al. Man against machine: diag- nostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison 58 dermatologists. Annals of Oncology. 29(8):1836-1844 2018.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-diisrributin examples in neural networks. arXiv:1610.022136 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. arXiv:1812.04600
Sepp Hochreiter and Jrgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. arXiv:1412.6622,, 2014.
ISIC. ISIC Archive, 2016. URL http::tttt/.w... isic archive.com/
Anthony Jerant, Jennifer T. Johnson, Catherine Demastes Sheridan, and Timothy Caffrey, Early detection and treatment of skin cancer. Am Fam Physician, 2000.
Heinrich Jiang, Been Kim, and Maya R. Gupta. To trust or not to trust classifier. In NIPS, 2018.
Alex Kendall and Yarin Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? In arXiv:1703.0497, March 2017a.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? In NIPS, 2017b.
B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, and R. Sayres. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In ICMI., 2018
11

---

Under review as a conference paper at ICLR 2020
Wonsik Kim, Bhavya Goyal, Kunal Chawla, Jungmin Lee, and Keunjoo Kwon. Attention- based ensemble for deep metric learning. In ECCV, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Pang Wei Koh and Percy Liang. Understanding Black-box Predictions via Influence Functions. In ICML, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NIPS. 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. arXiv:1607.06450, 2016.
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In AAAI, 2018.
André F. T. Martins and Ramón Fernández Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In MLR, 2016
G.A. Miller. The magical number seven, plus or minus 2: Some limits on our capacity for processing information. Psychological review, 63:81-97, 04 1956.
Vikram Mullachery, Aniruddh Khera, and Amir Husain. Bayesian neural networks. arXiv:1801.07710. 2018.
Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, inter- pretable and robust deep learning. Wwxivv1803.044755 2018.
Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard S. Zemel. Incremental few-shot learning with attention attractor networks. arXiv:/810.07218. 2018
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. Dynamic routing between capsules. In NIPS, 2017.
Devendra Singh Sachan and Petuum. Revisiting Istm networks for semi-supervised text classification via mixed objective function. KDD, 2018
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv: 2013.
Chawin Sitawarin and David A. Wagner. On the robustness of deep k-nearest neighbors. arXiv:1903.08333, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NIPS, 2017.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NIPS. 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, al. Attention is all you need. wwXiv:1706.037660 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, 2016.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun. and Rob Fergus. Regularization of neural networks using dropconnect. In JCML, 2013.
Chih-Kuan Yeh, Joon Sik Kim, Ian En-Hsu Yen, and Pradeep Ravikumar. Representer point selection for explaining deep neural networks. 1rXiv:/811.09720 2018
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. 131 1.2000, 2013.
Quanshi Zhang, Ying Nian Wu, and Song Chun Zhu, Interpretable convolutional neural networks. In CVPR, 2018.
12

---

Under review as a conference paper at ICLR 2020
A PSEUDO CODE FOR TRAINING
Algorithm Pseudo-code ProtoAttend training
1: Inputs:
Training dataset T. encoder model h(x: 0), classifier model h(v; Φ), normalization function n, input batch size B, candidate batch size D, attention dimension datts a values be used for loss: (0, 0.5, 1), task-specific loss function L, ADAM learning rate and exponential decay rate parameters 31 and B3
2: Initialize Trainable encoder parameters and classifier layer parameters Ø
3: while until convergence do
4: Sample mini-batch from the training dataset for the inputs: (x:; y/) ;=1 T
(c),D
5: Sample a mini-batch from the training dataset for the prototypes: (xj ,1/j /j=1 ~ T
6: for = ..... do
7:
Obtain queries and values for the input:
Q₁ Vi - 1(x;0)
8:
for jorj= L..., D do
9:
Obtain keys and values for the prototypes:
K (c) ví <- h(x()))))
10:
for i 1,.... B do
11:
for j 1=,...... D do
Estimate the relational attention coefficients:
Piu - n K (c) 02'Q2" / dast
Obtain the predictions for i 1, 1,... B do
14:
Vila 0) (vii))
- + (0.5r)+ +.5 E,= Pi Pf.
(.(a 0.5)
1) + Phoy P
15:
Estimate the total loss function
Lhatch < 1/B E.H, Ly: d (0)) (yi, (T)( (yi, (0.5))
Update the encoder model and the classifier layer
Φ ò ADAM V L T, B1, 32)
0 0 ADAM g .1.22
B RELATION TO INFLUENCE FUNCTIONS
Here we clarify the relationship our work to influence functions from a theoretical perspective. Influence functions quantify how model's predictions would change if we did not have particular training point. For the purpose of sample-based explainability, Koh & Liang 2017) proposes that the
relation between an input sample and the candidate sample
13
[c] can be obtained by quantifying
All training samples used candidate samples in Koh & Liang 2017
13

---

Under review as a conference paper at ICLR 2020
the influence of upweighting (xj 0) 3i (0)) on the loss query point (xi, W:):
I - V (()))) (H
V ((.1))
(4)
(0.0)
where (0,8) the Hessian and positive definite by assumption. Let's consider the singular value decomposition (H (0,0) can written as:
E. upz and also define the function k(x, v) ((o,o) Then, Eq.
(0) 1) (- ..) -... I5j (T) k(xx U3
(5)
We can observe that I is in the form of an inner product between two functions applied on (x4, y4) and (xx ). These two functions are composed of shared (and potentially complex) function, followed by a linear mapping with non-shared parameters. This expression is indeed similar form with the argument of the normalization function for attention in Eq. [ where the queries and keys are obtained by shared encoder except the last layer. The only notable difference is that ProtoAttend
encoder functions merely input X₁ and x (((), not the ground truth labels. Instead of relying on ground truth labels or complex Hessian estimations, ProtoAttend infers the encoded representations for the queries and keys directly in feedforward way, by learning from the entire training dataset. Note that PrenoAttend does not use a separate encoder for values, and obtains high performance by sharing the vast majority of the parameters while obtaining the keys, queries and values.
In Koh & Liang 2017), Influence Functions are also related to nearest neighbor search- based relevant point determination approaches. for sample-based explainability. When Euclidean space is considered for distances, with the assumption that all points have the same norm, the inner product between the representations correspond to their similarity. This scenario is the special case of ProtoAttend when we use the same representation for keys, queries and values, and when we train with only a 0 loss term although we would use Pt, for similarity determination. As studied in (Koh & Liang 2017), nearest neighbor-based methods are far less accurate in capturing the effect of model training, compared to Influence Functions. Our empirical results in Figs. [ and show superior performance of ProtoAttend compared to Influence Functions in finding perceptually more similar samples.
Overall, unlike Influence Functions, ProtoAttend modifies the model training for the desired goals, that fundamentally yields more degrees of freedom to optimize while achieving superior prototype learning quality effectively.
C TRAINING DETAILS
Different candidate databases are sampled randomly from the training dataset each iteration. Training database size is chosen to fit the model to the memory of a single GPU. D at inference is chosen sufficiently large to obtain high accuracy. Table shows the database size D for the datasets used in the experiments. The size of the prototype candidate database should be sufficiently large such that the model can attend to reasonable prototypes with high coefficients (separately for each input). With appropriate sparsity mechanisms, normally only end up with few prototypes with large coefficients. Indeed, most of the coefficients would be zero with sparsemax activation and sparsity regularization.
Table 4: Datasets and database size D.
Database size
Dataset
Encoder
Training
Inference
MNIST
ResNet
1024
32768
Fashion-MNIST
ResNet
1024
32768
CIFAR-I0
ResNet
1024
32768
Fruits
ResNet
256
4096
ISIC Melanoma
ResNet
256
4096
DBPedia
VDCNN
512
4096
Census Income
LSTM
4096
15360
14

---

Under review as a conference paper at ICLR 2020
C.I IMAGE DATA
C.I.1 MNIST DATASET
We apply random cropping after padding each side by 2 pixels and per image standardization. The base encoder uses standard 32 layer ResNet architecture. The number of filters is initially 16 and doubled every 5 blocks. In each block, two 3 x 3 convolutional layers are used to transform the input, and the transformed output is added to the imput after a 1 1 convolution. 4x downsampling is applied by choosing the stride as 2 after 5thh and 10th blocks. Each convolution followed by batch normalization and ReLU nonlinearity. After the last convolution, 7 7 average pooling is applied. The output is followed by fully-connected layer of 256 units and ReLU nonlinearity, followed by layer normalization (Lei Ba al. 2016). Keys and queries are mapped from the output using fully-connected layer followed by ReLU nonlinearity. where the attention size is dau=16. Values are mapped from the output using a fully-connected layer of douc=64 units and ReLU nonlinearity. followed by layer normalization. For the baseline encoder, the initial learning rate is chosen as 0.002 and exponential decay is applied with rate of 0.9 applied every 6k iterations. The model trained for 84k iterations. For prototypical learning model with softmax attention, the initial learning rate is chosen as 0.002 and exponential decay is applied with a rate of 0.8 applied every 8k iterations. The model is trained for 228k iterations. For prototypical learning model with sparsemax attention, the initial learning rate is chosen as 0.001 and exponential decay is applied with a rate of 0.93 applied every 6k iterations. The model is trained for 228k iterations. All models use batch size of 128 and gradient clipping above 20.
C.1.2 FASHION -NIST DATASET
We apply random cropping after padding each side by 2 pixels, random horizontal flipping, and per image standardization. The base encoder uses a standard 32 layer ResNet architecture, similar to our MNIST experiments. For the baseline encoder, the initial learning rate is chosen as 0.0015 and exponential decay is applied with rate of 0.9 applied every 10k iterations. The model is trained for 332k iterations. For prototypical learning with softmax attention, the initial learning rate is chosen as 0.0007 and exponential decay is applied with a rate of 0.92 applied every 8k iterations. The model is trained for 450k iterations. For prototypical learning with sparsemax attention, the initial learning rate is chosen as 0.001 and exponential decay applied with a rate of 0.9 applied every 8k iterations. The model is trained for 392k iterations. For prototypical learning with sparsemax attention and sparsity regularization (with Asporse = 0.0003), the initial learning rate is chosen as 0.001 and exponential decay is applied with a rate of 0.94 applied every 8k iterations. Aron 0.1 is chosen when confidence regularization is applied. The model is trained for 440k iterations. All models use batch size of 128 and gradient clipping above 20.
C.1.3 CIFAR-10 10 DATASET
We apply random cropping after padding each side by 3 pixels, random horizontal flipping, random vertical flipping and per image standardization. The base encoder uses a standard 50 layer ResNet architecture. The number of filters is initially 16 and doubled every 8 blocks. In each block, two 3 x 3 convolutional layers are used to transform the input, and the transformed output is added to the input after 1 x convolution. 4x downsampling is applied by choosing the stride as 2 after 8th and 16th blocks. Each convolution is followed by batch normalization and the ReLU nonlinearity. After the last convolution, 8 x 8 average pooling applied. The output is followed by a fully-connected layer of 256 units and the ReLU nonlinearity. followed by layer normalization (Lei Ba et al. 2016). Keys and queries are mapped from the output using a fully-connected layer followed by the ReLU nonlinearity, where the attention size is 11. Values are mapped from the output using a fully-connected layer of dow==128 units and the ReLU nonlinearity. followed by layer normalization. For the baseline encoder, the initial learning rate is chosen as 0.002 and exponential decay is applied with rate of 0.95 applied every 10k iterations. The model is trained for 940k iterations. For prototypical learning with softmax attention, the initial learning rate is chosen 0.0035 and exponential decay is applied with a rate of 0.95 applied every 10k iterations. The model is trained for 625k iterations. For prototypical learning with sparsemax attention, the initial learning rate is chosen as 0.0015 and exponential decay is applied with a rate of 0.95 applied every 10k iterations. The model is trained for 905k iterations. For prototypical learning with sparsemax attention and sparsity regularization
15

---

Under review as a conference paper at ICLR 2020
(with Asparse 0.00008), the initial learning rate is chosen as 0.0015 and exponential decay is applied with a rate of 0.95 applied every 12k iterations. Aconf = 0.11 is chosen when confidence regularization is applied. The model is trained for 450k iterations. All models use batch size of 128 and gradient clipping above 20.
CIFAR-10 experiments with noisy labels. For CIFAR- 10 experiments with noisy labels for the base encoder we only optimize the learning parameters. Noisy labels sampled uniformly from the set of labels excluding the correct one. The baseline model with noisy label ratio of 0.8 uses an initial learning rate of 0.001, decayed with rate of 0.92 every 6k iterations, and is trained for 1 5k iterations. For the dropout approach, dropout with a rate of 0.1 is applied, and the model uses an initial learning rate of 0.002, decayed with a rate of 0.85 every 8k iterations, and is trained for 24k iterations. The baseline model with noisy label ratio of 0.6 uses an initial learning rate of 0.002, decayed with a rate of 0.92 every 6k iterations, and is trained for 12k iterations. For the dropout approach, dropout with rate of 0.3 is applied, and the model uses an initial learning rate of 0.002, decayed with a rate of 0.92 every 8k iterations, and is trained for 18k iterations. The baseline model with noisy label ratio of 0.4 uses an initial learning rate of 0.002, decayed with a rate of 0.92 every 6k iterations, and is trained for 15k iterations. For the dropout approach, dropout with rate of 0.5 is applied, and the model uses initial learning rate of 0.002, decayed with rate of 0.92 every 6k iterations, and is trained for 18k iterations. For experiments for the prototypical learning model with sparsemax attention,
optimize the learning parameters and Asparse. For the model with noisy label ratio of 0.8, Asparse = 0.0015, initial learning rate is chosen as 0.0006 and exponential decay is applied with a rate of 0.95 applied every 8k iterations. The model is trained for 108k iterations. For the model with noisy label ratio of 0.6, Asparse = 0.0005, initial learning rate is chosen as 0.001 and exponential decay is applied with rate of 0.9 applied every 8k iterations. The model is trained for 92k iterations. For the model with noisy label ratio of 0.4, Asparse = 0.0003, initial learning rate is chosen as 0.001 and exponential decay applied with a rate of 0.9 applied every 6k iterations. The model is trained for 122k iterations.
C.1.4 FRUITS DATASET
We apply random cropping after padding each side by 5 pixels, random horizontal flipping, ran- dom vertical flipping and per image standardization. In the encoder, first, a downsampling with a convolutional layer is applied with stride of 2, and using 16 filters, followed by downsampling with max- max-pooling with a stride of 2. After obtaining the 25 x 25 inputs, a standard 32 layer ResNet architecture (similar to MNIST) is used, followed by fully-connected layer of 128 units and the ReLU nonlinearity, followed by layer normalization (Lei Ba et al.) 2016). Keys and queries are mapped from the output using a fully-connected layer followed by the ReLU nonlinearity, where the attention size is dau==16. Values are mapped from the output using a fully-comnected layer of d6w units and the ReLU nonlinearity, followed by layer normalization. W eight decay with a factor of 0.0001 is applied for the convolutional filters. The model uses a batch size of 128 and gradient clipping above 20.
C.1.5 ISIC MELANOMA DATASET
The ISIC Melanoma dataset is formed from the ISIC Archive (ISIC] 2016) that contains over 13k dermoscopie images collected from leading clinical centers internationally and acquired from variety of devices within each center. The dataset consists of skin images with labels denoting whether they contain melanoma are benign. We construct the training and validation dataset using 15122 images (13511 benign and 1611 melanoma cases), and the evaluation dataset using 3203 images (2867 benign and 336 melanoma). While training, benign cases are undersampled in each batch to have 0.6 ratio including candidate database sets at training and inference. All images are resized to 128 x 128 pixels. We apply random cropping after padding each side by pixels, random horizontal flipping, random vertical flipping and per image standardization. In the encoder, first, a downsampling with a convolutional layer is applied with a stride of 2, and using 16 filters, followed by a downsampling with max-pooling with stride of 2. After obtaining the 32 x 32 inputs, the base encoder uses a standard 50 layer ResNet architecture (similar to CIFAR1O), followed by fully-connected layer of 128 units and the ReLU nonlinearity, followed by layer normalization (Lei Ba et al.] 2016).. Keys and queries are mapped from the output using a fully-connecter layer tollowed by the ReLU nonlinearity, where the attention size is 11. Values are mapped from the
16

---

Under review as a conference paper at ICLR 2020
output using a fully- layer of dou==64 units and the ReLU nonlinearity, followed by layer normalization. For the baseline encoder, the initial learning rate is chosen as 0.002 and exponential decay is applied with rate of 0.9 applied every 3k iterations. The model is trained for 220k iterations. For prototypical learning with softmax attention, the initial learning rate is chosen as 0.0006 and exponential decay is applied with rate of 0.9 applied every 3k iterations. The model is trained for 147k iterations. For prototypical learning with sparsemax attention, the initial learning rate is chosen as 0.0006 and exponential decay is applied with rate of 0.9 applied every 4k iterations. The model is trained for 166k iterations. All models use batch size of 128 and gradient clipping above 20.
C.1.6 ANIMALS WITH ATTRIBUTES DATASET
We train ProtoAttend with sparsemax attention using the features from a pre-trained ResNet-50 as provided in (Yeh et al. 2018). To map the pre-trained features, we simply insert a single fully- connected layer with 256 units with ReLU nonlinearity and layer normalization, followed by the individual fully-connected layers of keys, queries and values (16, 16 and 64 units respectively with ReLU nonlinearity). Sparsity regularization is applied with Asparse = 0.000001. We train the model for 70k iterations. The initial learning rate is chosen as 0.0006 and exponential decay is applied with a rate of 0.8 applied every 10k iterations. A classification accuracy above 91 % obtained for the test set.
C.2 TEXT DATA
C.2.1 DBPEDIA DATASET
There are 14 output classes: Company, Educational Institution, Artist, Athlete, Office Holder, Mean Of Transportation, Building, Natural Place, Village, Animal, Plant, Album, Film, Written Work. As the input, 16-dimensional trainable embeddings are mapped from the dictionary of 69 raw characters (Coneeau et al. 2016). The maximum length is set to 448 and longer inputs are truncated while the shorter inputs are padded. The input embeddings are first transformed with a 1-D convolutional block consisting 64 filters with kernel width of 3 and stride of 2. Then, 8 convolution blocks as in (Conneau iet al.) 2016) are applied, with 64, 64, 128, 128, 256, 256, 512 and 512 filters respectively. All use the kernel width 3,, and after each two layers, max pooling is applied with kernel width of 3 and a stride of 2. All convolutions are followed by batch normalization and the ReLU nonlinearity. Convolutional filters use weight normalization with parameter 0.00001. The last convolution block is followed by k-max pooling with 1=8 (Conneau et al. 2016). Finally, we apply two fully-connected layers with 1024 hidden units. In contrast to (Conneau et al. 2016), we also use layer normalization (Lei Ba et al., 2016) after fully-connected layers we observe this leads to more stable training behavior. Keys and queries are mapped from the output using a fully-connected layer followed by the ReLU nonlinearity, where the attention size is d1u=16. Values are mapped from the output using a fully-connected layer of 66 units and the ReLU nonlinearity. followed by layer normalization. For the baseline encoder, initial learning rate is chosen 0.0008 and exponential decay is applied with a rate of 0.9 applied every 8k iterations. The model is trained for 212k iterations. For prototypical learning model with softmax attention, the initial learning rate is chosen 0.0008 and exponential decay is applied with a rate of 0.9 applied every 8k iterations. The model is trained for 146k iterations. For prototypical learning model with sparsemax attention, the initial learning rate is chosen as 0.0005 and exponential decay is applied with a rate of 0.82 applied every 8k iterations. The model is trained for 270k iterations. All models use a batch size of 128 and gradient clipping above 20. We do not apply any data augmentation.
C.3 TABULAR DATA
C.3.1 ADULT CENSUS INCOME
There are two output classes: whether or not the annual income is above $50k. Categorical categories such as the 'marital-status" are mapped to multi-hot representations. Continuous variables are used after a fixed normalization transformation. For 'age', the transformation first subtracts 50 and then divides by 30. For 'fnlwgt', the transformation first takes the log, and then subtracts 9, and then divides by 3. For iouucationnnmm , the transformation first subtracts 6 and then divides by 6. For "hours-per-week: the transformation first subtracts 50 and then divides by 50. For "capital-gain"
17

---

Under review as a conference paper at ICLR 2020
and 'capital-loss' the normalization takes the log, and then subtracts 5, and then divides by 5. The concatenated features are then mapped to a 64 dimensional vector using a fully-connected layer, followed by the ReLU nonlinearity. The base encoder uses LSTM architecture, with 4 timesteps. At each timestep, 64-dimensional inputs are applied after a dropout with rate 0.5. The output of the last timestep is used after applying dropout with rate 0.5. Keys and queries are mapped from this output using a fully-connected layer followed by the ReLU nonlinearity, where the attention size d=u==16. Values are mapped from the output using a fully-connected layer of 16 units and the ReLU nonlinearity, followed by layer normalization. For the baseline encoder, the initial learning rate is chosen as 0.002 and exponential decay applied with a rate of 0.9 applied every 2k iterations. The model is trained for 4.5k iterations. For the models with attention in prototypical learning framework, the initial learning rate is chosen as 0.0005 and exponential decay is applied with a rate of 0.92 applied every 2k iterations. The softmax attention model is trained for 13.5k iterations and the sparsemax attention model is trained for 11.5k iterations. For the model with sparsity regularization, the initial learning rate is 0.003 and exponential decay is applied with a rate of 0.7 applied every 2k iterations, and the model is trained for 7k iterations. All models use a batch size of 128 and gradient clipping above 20. We do not apply any data augmentation.
D ADDITIONAL PROTOTYPE EXAMPLES
Fig. 10 exemplify prototypes for CIFAR-10. For most cases, we observe the similarity of discrimina- tive features between inputs and prototypes. For example, the body figures of birds, the shape of tires, the face patterns of dogs, the body figures of frogs, the appearance of the background sky for planes, are among the features apparent in examples.
Fig. shows additional prototype examples for DBPedia dataset. Prototypes have very similar sentence structure, words and concepts, while categorizing the sentences into ontologies.
Fig. 12 shows example prototypes for ISIC Melanoma. In some cases, we observe the commonalities between input and prototypes that distinguish melanoma cases such as the non-circular geometry or ireeullarly-notched borders (Jerant et al.) (000)). Compared to other datasets, ISIC Melonama dataset yields lower interpretable prototype quality on average. We hypothesize this to be due to the perceptual difficulty of the problem as well as the insufficient encoder performance shown by the lower classification accuracy (despite the acceptable AUC).
Fig. 13 shows more comparison examples for prototypical learning framework with sparsemax attention vs. representer point selection (Yeh et al. 2018) on Animals with Attributes dataset. For some cases, including chimpanzee, zebra, dalmatian and tiger, ProtoAttend yields perceptually very similar samples. The similarity of the chimpanzee body form and the background, zebra patterns, dalmatian pattern on the grass, and tiger pattern and head pose, are prominent. Representer point selection fails capture such similarity features as effectively. On the other hand, for bat, otter and wolf, the results are somewhat less satisfying. The wing part of the bat, multiple count of the otters with the background, and the color and furry head of the wolf seem be captured, but with less apparent similarity than some other possible samples from the dataset. Representer point selection method also cannot be claimed to be successful in these cases. Lastly, for leopard, ProtoAttend only yields one non-zero prototype (which is indeed statistically rare given the model and sparsity choices). The pattern of the leopard image seems relevant, but it is also not fully satisfying to observe single prototype that is not perceptually more similar. All of the test examples in Fig. [3 are classified correctly with our framework and all of the shown prototypes are also from the correct classes
E COMPARISON OF CONFDENCCE PREDICTION FOR SOFTMAX vs. SPARSEMAX
Figs. 14 Hand] 15 show the accuracy ratio of samples for softmax vs. sparsemax attention without confidence regularization. The baseline accuracy (at 100% prediction ratio) is higher for softmax attention for some datasets, whereas higher for sparsemax for some others. On the other hand, higher number of prototypes yielded by softmax attention results in a wider range for confidence-controlledd prediction trade-off.
18

---

Under review as a conference paper at ICLR 2020
Input
Prototypes
0.53
0.37
0.07
0.67
0.22
0.10
Input
Prototypes
0.12
0.09
0.05
0.07
0.31
0.21
0.09
0.10
0.09
0.06
0.65
0.22
0.08
0.16
0.15
0.12
0.40
0.14
0.18
0.06
0.08
0.21
0.18
0.48
0.28
0.12
0.10
0.07
0.06
0.97
With sparsemax
(b) With sparsemax and sparsity regulariza- tion
Figure 10: Example inputs and corresponding prototypes for CIFAR-10.
19

---

Under review as a conference paper at ICLR 2020
Input
viectesis infuncata in moch is the acctiidee fandly. it var described by de tostget is 1959. it in found in radaguscar,
hirdwing vith small dloco) spot on the hindelog.
the forowing in
with indiatines diacal the underside is unicolorous
Prototypes
grarntn
0.32
consless prairie duner. the forewings sbout 13.3 will teetres (0.72 thi. the forest are dark browr to blank doraally with anewyy haft to plncish bat? hands. the hindwinga ase deep piskiab aed with black rackings. adulta cze ving tecn aid
north
colorade.
0.22
lassocampidas fasily. specias nutolyptos
Input
prybant ie Mind thrillar film diracced by sachle p- tarande and prodiond by earoch that. the fill releaned december 2010 arthangel
Prototypes
0.44
dacle olivia swrilland goeaph conten and noceshoad well for the sorwen by banry fatrell and lutar bal from farrall ungablieted chore etery whas ever happased to conets charloctel recatved seves scndaey woard conteatione.
in har final film.tte rorla
saneal io Eleethrstar uslayslan norts released in 1052. st ie directed by the nocable director taeal and produoed by daeld lathopuilis Lis the ravie sture jistinu siddharts ponuka catonn und blavuna. it zu buss koow for its evurd winning song aulamnev is the campus povernmen enginwuring college thrissur
0.23
auny vittu geathup grotsum, morie
Prototypes Input
achool providea coepcutenzaiee uducatios thus. 230 students unmully is zine shaoogt tvelve, baces high aciool Ie con of two publlc high echooia to pratete councy and is the sola high aatool aduinletared by the Tazen school district.
0.50
0:33 11000 regunoa renglishr high ochool) is chileon high setool located in regunoe osotaponl. province chile.
Figure 11: Example inputs and corresponding prototypes for DBPedia (with sparsemax).
20

---

Under review as a conference paper at ICLR 2020
Input
Prototypes
0.11
0.11
0.1
0.12
0.08
0.08
0.07
0.05
0.05
0.23
0.10
0.06
Figure 12: Example inputs and corresponding prototypes for ISIC Melanoma (with sparsemax attention).
21

---

Under review as a conference paper at ICLR 2020
Input
ProtoAttend
Representer point selection
Figure 13: Relevant samples found by ProtoAttend with sparsemax attention representer point
selection Yeh et al. 2018) for the examples from Supplementary Material (Ych al. 2018).
22

---

Under review as a conference paper at ICLR 2020
29.00
96.78
Accoracy 26
0.092
Accuracy
59.73
20.10
09.68
Spanserax allm.
95
Sparserriax ellns
BL099
Softmex
Softmax atth.
20.00
Baseline
00.6
90.7
92.8
59.5
100.
Retio serpples %
Ratio of samples %
(a) MNIST
(b) Fashion MNIST
Figure 14: Accuracy vs. ratio of samples for (a) MNIST and (b) Fashion MNIST, for confidence levels between 0 and 0.999.
100
Rocurecy %
Accuracy
Sparserax attn.
Sparsemax
Softmax sttn.
Softmax attn.
Baseline
Beseline
samples
samples
(a) DBpedia
(b) Adult Census Income
Figure Accuracy vs. ratio of samples for (a) DBpedia and (b) Adult Census Income, for confidence levels between 0 and 0.999,
23

---

Under review as a conference paper at ICLR 2020
As impactful case study, consider melanoma detection problem with ISIC dataset (ISIC 2016) in Supplementary Material. In medical diagnosis, is strongly desired to maintain a sufficienlly high prediction performance, potentially by verifying the decisions of an Al system by medical experts in the cases where the Al models are not confident. By refraining from some predictions, as shown in Fig. T6 demonstrate unprecedentedly high AUC values without using transfer learning highly-customized models (Haenssle et al. 2018).
038
0.92
0.93
Sparserax otbn.
Baselne enc.
Rat of 9%
Figure 16: Area-under-curve (AUC) vs. ratio of samples for ISIC Melanoma with softmax attention, for confidence values ranging between 0 and 0.99.
F HUMAN USER STUDY ON THE USEFULNESS OF PROTOTYPES
We perform user study by asking humans how much extra image helps in explaining the guessed class of the input, after showing what the trained network predicts for that input. We consider the Animals with Attributes dataset (exemplified in Fig. [3}. We randomly pick test samples and assess how much showing the top prototype makes a difference. The results in Table 5 shows that ProtoAttend picks
Table 5: Human ratings (mean score and 95% confidence interval) on how much an extra image helps guessing the class of the input.
Sampling method
Score (out of 5)
Top prototype by ProtoAttend Randomly sampled from the predicted class Randomly sampled from any class
4.33 + 0.09
3.97 0.12
1.33 + 0.09
G CONTROLLING SPARSITY VIA REGULARIZATION
Agpana
Median number of prototypes
1000
Asana +0,0001
95% decision
20.001
100
20000
300k
600k
Number of Iterations
Figure 17: Number of training iterations vs. median number prototypes to explain 95% of the decision (in logarithmic scale), for Fashion-MNISS MNIST with softmax attention.
24

---

Under review as a conference paper at ICLR 2020
Fig. 17 shows the impact of sparsity regularization coefficient training. By varying the value of Asparse; the number of prototypes can be efficiently controlled. For high values of sparsity regularization coefficient, the model gets stuck at point where it is forced make decision from a low number of prototypes before the encoder model is properly learned, hence typically yields considerably lower performance. We also observe sparsity mechanism via sparsemax attention yield better performance than softmax attention with high sparsity regularization.
H PROTOTYPE QUALITY
In general, the following scenarios may yield low prototype quality:
1. Lack of related samples in the candidate database.
2. Perceptual difference between humans and encoders in determining discriminative features.
3. High intra-class variability that makes training difficult.
4. Imperfect encoder that cannot yield fully accurate representations of the input.
5. Insufficiency of relational attention determine weights from queries and keys.
6. Inefficient decoupling between encoder & attention blocks and the final decision block.
There can be poblem--depndder fundamental limitations on (1)-(3), whereas (4)-(6) are raised by choices of models and losses and can be further improved. We leave the quantification of prototype quality using information--therretii metrics or discriminative neural networks to future work.
I UNDERSTANDING MISCLASSIFICATION CASES
One of the benefits of prototypical learning is insights into wrong decision cases. Fig. exemplifies prototypes with wrong labels, that give insights about why the model is confused about particular input (c.g. due to similarity the visual patterns). Such insights can be actionable to improve the model performance, such adding more training samples for the confusing classes modifying the loss functions.
Inputs
Prototypes
(cal)
(dog)
(irplane)
(ship)
(bird)
(aipplane)
Figure 18: Example prototypes with wrong labels for CIFAR-10.
25


Reviews:
[{"review_id": "r1evmU2pFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes an attention-based prototype learning algorithm, which introduces an attention operation to assign different weights to the prototypes. Comprehensive experiments demonstrate that the proposed method is efficient and effective in various tasks. \n\nI have the following comments:\n-\tThe authors did a really good job in empirical studies, verifying the superiority of ProtoAttend.\n\n-\tThe novelty of the main idea is limited and may provide a limited contribution to the research community.\n\n-\tThe authors clarified that ProtoAttend is an inherently interpretable algorithm. However, the interpretability is proved by naïve Prototype learning only. A rigorous theoretical proof should be provided to demonstrate its interpretability.\n\n-\tI would be appreciated if the authors provide the pseudo-code to show the training procedure of ProtoAttend.\n\nOverall, I think this work is not ready for publishing unless the theoretical property is well understood.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlFcFhb9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a sample-based self-explaining method for image classification. The basic idea is adopt the attention mechanism to learn the relation between the latent representation of the query sample and training samples, and identify the training samples with higher similarity as the prototype. The classification decision is based on the label consistency between the identified prototypes (with the relation score in attention mechanism as the weight of different prototypes in determining the  label agreement)\n\nThe proposed model is intrinsically interpretable since the prototypes with higher weights can play as the decision explanation. And the authors have conducted experiments to show that such self-explaining mechanism based on attention model can achieve comparable classification accuracy with original black-box models.\n\nThe presentation of the paper is clear and easy to follow. But I have several concerns regarding the choice of prototypes and the evaluation of the interpretation:\n\n1) According to Eq. (2), it seems that all training samples are used as the prototypes (but with different weights). Why not just use the top few prototypes? Would this such setting introduce a lot of noise, since many training samples are from different classes? \n\n2) Since one focus of the paper is to provide interoperation of the classification model, some more experiments are needed to evaluate how well the interoperation is. For example, some crowdsourcing experiments to check if the provided prototypes can help human users correctly guess the model prediction.\n\n3) I think the authors should also compare with the black-box model when we use the attention mechanism as a post-hoc interpretation. One straightforward baseline is that use the black-box model for classification, and pick the top \"prototypes\" with the highest similarity in the latent representation. Such comparison can help to validate if incorporating attention mechanism in the model design can provide better quality prototypes.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byx38twAYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The aim of this work is to make deep learning classifiers more interpretable by \"projecting\" each input sample into a small collection of prototype examples (with some weighting over those) and then basing the decision on a combination of the latent representations of the chosen prototypes. In this way, the chosen category can be justified as the input being similar to the selected prototypes. Additionally, this approach makes it possible to obtain a confidence score at test time.\n\nThe choice fo the encoders for the prototypes and the examples is asymmetric (the first using keys and the second queries). This is not justified. Is it empirically better than using the same encoding for both before feeding them to the relational attention?\n\nThis work aims to satisfy many desiderata (listed in section 3). The decisions made to accomplish these are reasonable although somewhat arbitrary. In fact, several ways to encode the desiderata in the loss function are listed in Table 1.\n\nQualitatively, in the presented comparison with influence networks and representer point selection, ProtoAttend seems to choose more representative examples.\n\nIt is not easy to find a direct, quantitative way to compare this type of work with the existing literature, but from a qualitative perspective, the set goal (which is an important one) is achieved.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an interpretable machine learning method, ProtoAttend, that bases decisions on few relevant "prototypes." The proposed method uses an attention mechanism (possibly sparse, via sparsemax) that relates the encoded representations to samples in order to determine prototypes. The resulting model enables similarity-based interpretability, confidence estimation by quantifying the mismatch across prototype labels, and can be used for distribution mismatch detection. 

While the proposed model is interesting, the reviewers raised several concerns regarding the choice of prototypes and the evaluation of human interoperation. The paper would benefit from more experiments besides the provided user studies to check if the provided prototypes can help human users correctly guess the model prediction. I encourage the authors to address these suggestions in a future resubmission.
Decision: Reject