ProtoAttend: Attention-Based Prototypical Learning

1 INTRODUCTION . Deep neural networks have been pushing the frontiers of artificial intelligence ( AI ) by yielding excellent performance in numerous tasks , from understanding images ( He et al. , 2016 ) to text ( Conneau et al. , 2016 ) . Yet , high performance is not always a sufficient factor - as some realworld deployment scenarios might necessitate that an ideal AI system is ‘ interpretable ’ , such that it builds trust by explaining rationales behind decisions , allow detection of common failure cases and biases , and refrains from making decisions without sufficient confidence . In their conventional form , deep neural networks are considered as black-box models – they are controlled by complex nonlinear interactions between many parameters that are difficult to understand . There are numerous approaches , ( Kim et al. , 2018 ; Erhan et al. , 2009 ; Zeiler & Fergus , 2013 ; Simonyan et al. , 2013 ) , that bring post-hoc explainability of decisions to already-trained models . Yet , these have the fundamental limitation that the models are not designed for interpretability . There are also approaches on the redesign of neural networks towards making them inherently-interpretable , as in this paper . Some notable ones include sequential attention ( Bahdanau et al. , 2015 ) , capsule networks ( Sabour et al. , 2017 ) , and interpretable convolutional filters ( Zhang et al. , 2018 ) . We focus on inherently-interpretable deep neural network modeling with the foundations of prototypical learning . Prototypical learning decomposes decision making into known samples ( see Fig . 1 ) , referred here as prototypes . We base our method on the principle that prototypes should constitute a minimal subset of samples with high interpretable value that can serve as a distillation or condensed view of a dataset ( Bien & Tibshirani , 2012 ) . Given that the number of objects a human can interpret is limited ( Miller , 1956 ) , outputting few prototypes can be an effective approach for humans to understand the AI model behavior . In addition to such interpretability , prototypical learning : ( 1 ) provides an efficient confidence metric by measuring mismatches in prototype labels , allowing performance to be improved by refraining from making predictions in the absence of sufficient confidence , ( 2 ) helps detect deviations in the test distribution by measuring mismatches in prototype labels that represent the support of the training dataset , and ( 3 ) enables performance in the high label noise regime to be improved by controlling the number of selected prototypes . Given these motivations , prototypes should be controllable in number , and should be perceptually relevant to the input in explaining the decision making task . Prototype selection in its naive form is computationally expensive and perceptually challenging ( Bien & Tibshirani , 2012 ) . We design ProtoAttend to address this problem in an efficient way . Our contributions can be summarized as follows : 1 . We present principles that can be guiding for the design of inherently-interpretable models based on sample-based interpretability . 2 . We propose a novel method , ProtoAttend , for selecting input-dependent prototypes based on an attention mechanism between the input and prototype candidates . ProtoAttend is model-agnostic and can even be integrated with pre-trained models . 3 . ProtoAttend allows interpreting the contribution of each prototype via the attention outputs . 4 . For a ‘ condensed view ’ , we demonstrate that sparsity in weights can be efficiently imposed via the choice of the attention normalization and additional regularization . 5 . On image , text and tabular data , we demonstrate the four key benefits of ProtoAttend : interpretabil- ity , confidence control , diagnosis of distribution mismatch , and robustness against label noise . ProtoAttend yields superior quality for sample-based interpretability , better-calibrated confidence scoring , and more sensitive out-of-distribution detection compared to alternative approaches . 6 . ProtoAttend enables all these benefits via the same architecture and method , while maintaining comparable overall accuracy . 2 RELATED WORK . Prototypical learning : The principles of ProtoAttend are inspired by ( Bien & Tibshirani , 2012 ) . They formulate prototype selection as an integer program and solve it using a greedy approach with linear program relaxation . It seems unclear whether such approaches can be efficiently adopted to deep learning . ( Chen et al. , 2018 ) and ( Li et al. , 2018 ) introduce a prototype layer for interpretability by replacing the conventional inner product with a distance computation for perceptual similarity . In contrast , our method uses an attention mechanism to quantify perceptual similarity and can choose input-dependent prototypes from a large-scale candidate database . ( Yeh et al. , 2018 ) decomposes the prediction into a linear combination of activations of training points for interpretability using representer values . The linear decomposition idea also exists in ProtoAttend , but the weights are learned via an attention mechanism and sparsity is encouraged in the decomposition . In ( Koh & Liang , 2017 ) , the training points that are the most responsible for a given prediction are identified using influence functions via oracle access to gradients and Hessian-vector products . Metric learning : Metric learning aims to find an embedding representation of the data where similar data points are close and dissimilar data pointers are far from each other . ProtoAttend is motivated by efficient learning of such an embedding space which can be used to decompose decisions . Metric learning for deep neural networks is typically based on modifications to the objective function , such as using triplet loss and N-pair loss ( Sohn , 2016 ; Cui et al. , 2016 ; Hoffer & Ailon , 2014 ) . These yield perceptually meaningful embedding spaces yet typically require a large subset of nearest neighbors to avoid degradation in performance ( Cui et al. , 2016 ) . ( Kim et al. , 2018 ) proposes a deep metric learning framework which employs an attention-based ensemble with a divergence loss so that each learner can attend to different parts of the object . Our method has metric learning capabilities like relating similar data points , but also performs well on the ultimate supervised learning task . Attention-based few-shot learning : Some of our inspirations are based on recent advances in attention-based few-shot learning . In ( Vinyals et al. , 2016 ) , an attention mechanism is used to relate an example with candidate examples from a support set using a weighted nearest-neighbor classifier applied within an embedding space . In ( Ren et al. , 2018 ) , incremental few-shot learning is implemented using an attention attractor network on the encoded and support sets . In ( Snell et al. , 2017 ) , a non-linear mapping is learned to determine the prototype of a class as the mean of its support set in the embedding space . During training , the support set is randomly sampled to mimic the inference task . Overall , the attention mechanism in our method follows related principles but fundamentally differs in that few-shot learning aims for generalization to unseen classes whereas the goal of our method is robust and interpretable learning for seen classes . Uncertainty and confidence estimation : ProtoAttend takes a novel perspective on the perennial problem of quantifying how much deep neural networks ’ predictions can be trusted . Common approaches are based on using the scores from the prediction model , such as the probabilities from the softmax layer of a neural network , yet it has been shown that the raw confidence values are typically poorly calibrated ( Guo et al. , 2017 ) . Ensemble of models ( Lakshminarayanan et al. , 2017 ) is one of the simplest and most efficient approaches , but significantly increases complexity and decreased interpretability . In ( Papernot & McDaniel , 2018 ) , the intermediate representations of the network are used to define a distance metric , and a confidence metric is proposed based on the conformity of the neighbors . ( Jiang et al. , 2018 ) , proposes a confidence metric based on the agreement between the classifier and a modified nearest-neighbor classifier on the test sample . In ( DeVries & Taylor , 2018 ) , direct inference of confidence output is considered with a modified loss . Another direction of uncertainty and confidence estimation is Bayesian neural networks that return a distribution over the outputs ( Kendall & Gal , 2017b ) ( Mullachery et al. , 2018 ) ( Kendall & Gal , 2017a ) . 3 PROTOATTEND : ATTENTION-BASED PROTOTYPICAL LEARNING . Consider a training set with samples , T = { xi , yi } . Conventional supervised learning aims to learn a model s ( xi ; S ) that minimizes a predefined loss 1/B · ∑B i=1 L ( yi , ŷi = s ( xi ; S ) ) 1 at each iteration , where B is the batch size for training . Our goal is to impose that decision making should be based on only a small number of training examples , i.e . prototypes , such that their linear superposition in an embedding space can yield the overall decision and the superposition weights correspond to their importance . Towards this goal , we propose defining a solution to prototypical learning with the following six principles : i. vi = f ( xi ; θ ) encodes all relevant information of xi for the final decision . f ( ) considers the global distribution of the samples , i.e . learns from all { xi , yi } . Although all the information in training dataset is embodied in the weights of the encoder2 , we construct the learning method in such a way that decision is dominated by the prototypes with high weights . ii . From the encoded information , we can find a decision function so that the mapping g ( vi ; η ) is close to the ground truth yi , in a consistent way with conventional supervised learning . iii . Given candidates x ( c ) j to select the prototypes from , there exists weights pi , j ( where pi , j ≥ 0 and ∑D j=1 pi , j = 1 ) , such that the decision g ( ∑D j=1 pi , jv ( c ) j ; η ) ( where v ( c ) j = f ( x ( c ) j ; θ ) ) is close to the ground truth yi . iv . When the linear combination ∑D j=1 pi , jv ( c ) j is considered , prototypes with higher weights pi , j have higher contribution in the decision g ( ∑D j=1 pi , jv ( c ) j ; η ) . v. The weights should be sparse – only a controllable amount of weights pi , j should be nonzero . Ideally , there exists an efficient mechanism for outputting pi , j to control the sparsity without significantly affecting performance . vi . The weights pi , j depend on the relation between input and the candidate samples , pi , j = r ( xi , x ( c ) j ; Γ ) , based on their perceptual relation for decision making . We do not introduce any heuristic relatedness metric such as distances in the representation space , but we allow the model to learn the relation function that helps the overall performance . Learning involves optimization of the parameters θ , Γ , η of the corresponding functions . If the proposed principles ( such as reasoning from the linear combination of embeddings or assigning relevance to the weights ) are not imposed during training but only at inference , a high performance can not be obtained due to the train-test mismatch , as the intermediate representations can be learned in an arbitrary way without any necessities to satisfy them.3 The subsequent section presents ProtoAttend and training procedure to implement it .

Summary: The aim of this work is to make deep learning classifiers more interpretable by "projecting" each input sample into a small collection of prototype examples (with some weighting over those) and then basing the decision on a combination of the latent representations of the chosen prototypes. In this way, the chosen category can be justified as the input being similar to the selected prototypes. Additionally, this approach makes it possible to obtain a confidence score at test time.