Prototypical Examples in Deep Learning: Metrics, Characteristics, and Utility

Machine learning ( ML ) research has investigated prototypes : examples that are representative of the behavior to be learned . We systematically evaluate five methods for identifying prototypes , both ones previously introduced as well as new ones we propose , finding all of them to provide meaningful but different interpretations . Through a human study , we confirm that all five metrics are well matched to human intuition . Examining cases where the metrics disagree offers an informative perspective on the properties of data and algorithms used in learning , with implications for data-corpus construction , efficiency , adversarial robustness , interpretability , and other ML aspects . In particular , we confirm that the “ train on hard ” curriculum approach can improve accuracy on many datasets and tasks , but that it is strictly worse when there are many mislabeled or ambiguous examples . 1 INTRODUCTION . When reasoning about ML tasks , it is natural to look for a set of training or test examples that is somehow prototypical—i.e. , that is representative of the desired learned behavior . Although such prototypical examples have been central to several research efforts , e.g. , in interpretability ( Bien & Tibshirani , 2011 ) and curriculum learning ( Bengio et al. , 2009 ) , no generally-agreed-upon definition seems to exist for prototypes , or their characteristics . For modern deep-learning models , whose behavior is often inscrutable , even the very existence and usefulness of prototypical examples has seemed uncertain until the recent work of Stock & Cisse ( 2017 ) . Inspired by that work we ( 1 ) identify a set of desirable properties for prototypicality definitions ; ( 2 ) systematically explore different metrics used in prior work , as well as new metrics we develop , for identifying prototypical examples in both training and test data ; ( 3 ) study the characteristics of those metrics ’ prototypes and their complement set—the outliers—using both quantitative measures and a qualitative human study ; and , ( 4 ) evaluate the usefulness of prototypes for machine-learning purposes such as reducing sample complexity or improving adversarial robustness and interpretability . Our prototypicality metrics are based on adversarial robustness , retraining stability , ensemble agreement , and differentially-private learning . As an independent result , we show that predictive stability under retraining strongly correlates with adversarial distance , and may be used as an approximation . Unequivocally , we find that distinct sets of prototypical and outlier examples exist for the datasets we consider : MNIST ( LeCun et al. , 2010 ) , Fashion-MNIST ( Xiao et al. , 2017 ) , CIFAR-10 ( Krizhevsky & Hinton , 2009 ) , and ImageNet ( Russakovsky et al. , 2015 ) . Between all of our metrics , as well as human evaluators , there is overall agreement on the examples that are prototypes and those that are outliers . Furthermore , the differences between metrics constitute informative exceptions , e.g. , identifying uncommon submodes in the data as well as spurious , ambiguous , or misleading examples . Usefully , there are advantages to training models using only prototypical examples : the models learn much faster , their accuracy loss is not great and occurs almost entirely on outlier test examples , and the models are both easier to interpret and more adversarially robust . Conversely , at the same sample complexity , significantly higher overall accuracy can be achieved by training models exclusively on outliers—once erroneous and misleading examples have been eliminated from the dataset . 2 DEFINING AND IDENTIFYING PROTOTYPES . In designing a metric to identify prototypes , many approaches may seem intuitive . We identify the following properties as desirable for any prototypicality metric : • Independent of the learning task : A prototypicality metric should be applicable to all types of machine-learning tasks , whether they are based upon unsupervised or supervised approaches , or whether they constitute classification tasks or generative tasks , etc . • Independent of the modeling approach : A metric should identify the overall same prototypical examples regardless of the machine-learning paradigm , model architecture , capacity , or hyperparameters that are used to learn the task . • Aligned with human intuition : At least for tasks on which humans do well , prototypical examples identified by a metric should strongly overlap with those identified by humans . • Covers all apparent data modes : A metric should provide a balanced view of all modes of prototypical examples ( e.g. , even when multiple disparate modes have a single output classification label ) . In the presence of imbalance in the frequency of data modes , the metric should provide coverage while reflecting frequency : a mode supported by only a handful of examples need not be as prototypical as other modes in the same class . • Provides a stable ranking : A metric should not only identify examples , but also rank them in terms of prototypicality , and this rank should be stable ( i.e. , have low variance ) even though the metric is likely computed by a randomized procedure . • Applies to both training and test data : A metric should allow both training examples and test examples to be ranked in terms of prototypicality . • Predicts test accuracy : Models that are trained only on prototypical training examples should still achieve good accuracy on the prototypical test examples—at least when the training and test datasets are balanced and the models use machine-learning methods known to perform well overall . Other cases need not be so predictive ( e.g. , if training is done only on outliers ) ; we explore this experimentally in Section 3 and Section 4 . Although metrics for prototypicality should generally satisfy the above properties , they need not do so perfectly . In fact , our experimental results show that the differences between metrics can be highly informative . In particular , as described in Section 3.3 , such differences can improve interpretability and provide insights into model behavior that are not achievable by a single , overall accuracy number , e.g. , by giving explanations of failures by example ( Caruana et al. , 1999 ) . 2.1 METRICS FOR PROTOTYPICALITY . A number of metrics for identifying prototypes might satisfy the desirable properties we identified above , possibly even the early methods based on concept similarity introduced by Zhang ( 1992 ) . In addition to the work presented below , a complete survey of related work is found in the Appendix . To start with , consider two strawmen prototypicality metrics based on either learning order or gradient magnitude . A model may be expected to learn prototypes early in training because they are presumably more common and somehow “ simpler ” than all the edge-case outliers . Conversely , near the end of training , one might expect the relative magnitude of the gradient ‖∇θ ` ( fθ ( x ) ) ‖ to be small for prototypical examples x , as each such example should have very little to teach the model . From this , two metrics can be defined , which we evaluated in experiments that averaged hundreds of model training runs to minimize random-initialization and stochastic-learning effects . Unfortunately , both of these metrics exhibited both very high variance as well as apparently-low signal , defining prototypes sets that appeared random upon inspection and that did not satisfy our stated desirable property of predicting test accuracy . Thus , we do not present results for these two metrics . Instead , we define and apply the following five metrics , each of which ranks examples by their relative ordinal number ( i.e. , position in the sorted order of the measured value ) : Adversarial Robustness ( adv ) : Prototypical examples should be more adversarially robust . As a measure of prototypicality , the distance to the decision boundary measured by an adversarialexample attack was recently proposed and utilized by Stock & Cisse ( 2017 ) . Specifically , for an example x , the measure finds the perturbation δ with minimal ‖δ‖ such that the original x and the adversarial example x+ δ are classified differently ( Biggio et al. , 2013 ; Szegedy et al. , 2013 ) . To compare prototypicality , the work of Stock & Cisse ( 2017 ) that inspired our current work used a simple and efficient ` ∞-based adversarial-example attack based on an iterative gradient descent introduced by Kurakin et al . ( 2016 ) . That attack procedure computes gradients to find directions that will increase the model ’ s loss on the input within an ` ∞-norm ball . They define prototypicality as the number of gradient descent iterations necessary to change the class of the perturbed input . Instead , the adv metric ranks by the ` 2 norm ( or faster , less accurate ` ∞ norm ) of the minimal-found adversarial perturbation ( Carlini & Wagner , 2017 ) . This is generally more accurate at measuring the distance to the decision boundary , but comes at a performance cost ( it is on average 10-100× slower ) . Holdout Retraining ( ret ) : The intuition beind our ret metric is that a model should treat a prototypical example the same regardless of whether , or when , it was used in the training process . Assume we are given a training dataset X , a disjoint holdout dataset X̄ , and an example x ∈ X for which to assess prototypicality . To begin , we train a model f ( · ) on the training data X to obtain model weights θ . We train this model just as how we would typically do—i.e. , with the same learning rate schedule , hyper-parameter settings , etc . Then , we fine-tune the weights of this first model fθ ( · ) on the held-out training data X̄ to obtain new weights θ̄ . To perform this fine-tuning , we use a smaller learning rate and train until the training loss stops decreasing . ( We have found it is important to obtain θ̄ by fine-tuning θ as opposed to training from scratch ; otherwise , the randomness of training leads to unstable rankings that yield specious results . ) Finally , given these two models , we compute the prototypicality of x as the difference ‖fθ ( x ) − fθ̄ ( x ) ‖ . The exact choice of metric ‖·‖ is not important ; the results in this paper use the symmetric KL-divergence . While this metric is similar to the one considered in Ren et al . ( 2018 ) , it differs in important ways : notably , our holdout retraining metric is conceptually simpler , more stable numerically , and more computationally efficient ( because it does not require a backward pass to estimate gradients in addition to the forward pass needed to compare model outputs ) . Because our metric is only meaningful for data used to train the initial model , in order to measure the prototypicality of arbitrary test points , we actually train on the test data and perform holdout retraining on the original training data . Ensemble Agreement ( agr ) : Prototypical examples should be easy for many types of models to learn . We train multiple models of varying capacity on different subsets of the training data ( see Appendix C ) . The agr metric ranks examples ’ prototypicality based on the agreement within this ensemble , as measured by the symmetric KL-divergence between the models ’ output . Concretely , we train many models fθi ( · ) and , for each example x , evaluate the model predictions , and then compute 1N2 ∑N i=1 ∑N j=1 JS-Divergence ( fθi ( x ) , fθj ( x ) ) to rank the example ’ s prototypicality . Model Confidence ( conf ) : We expect models to be confident on prototypical examples . Based on an ensemble of models like that used by the agr metric , the conf metric ranks examples by the mean confidence in the models ’ predictions , i.e. , ranking each example x by 1N ∑N i=1 max fθi ( x ) . Privacy-preserving Training ( priv ) : We can expect prototypical , well-represented modes of examples to be classified properly by models ( e.g. , deep neural networks ) even when trained with guarantees of differential privacy ( Abadi et al. , 2016 ; Papernot et al. , 2016 ) . However , such privacypreserving models should exhibit significantly reduced accuracy on any rare or exceptional examples , because differentially-private learning attenuates gradients and introduces noise to prevent the details about any specific training examples from being memorized . Outliers are disproportionally likely to be impacted by this attenuation and added noise , whereas the common signal found across many prototypes must have been preserved in models trained to reasonable accuracy . Our priv metric is based on training an ensemble of models with increasingly greater ε privacy ( i.e. , more attenuation and noise ) using ε-differentially-private stochastic gradient descent ( Abadi et al. , 2016 ) . Our metric then ranks the prototypicality of an example based on the maximum ε privacy at which the example is correctly classified in a reliable manner ( which we take as being also classified correctly in 90 % of less-private models ) . This ranking embodies the intuition that the more tolerant an example is to noise and attenuation during learning , the more prototypical it is .

Summary: The paper proposes methods for identifying prototypes. Unfortunately, a formal definition of a prototype is lacking, and the authors instead present a set of heuristics for sorting data points that purport to measure 'prototypicality', although different heuristics have different (and possibly conflicting) notions of what this means. The experiments are not very convincing, and often present results that are either inconclusive or negative, i.e. seem to demonstrate that prototypes are not very useful.