Reducing Overconfident Errors outside the Known Distribution

1 INTRODUCTION . In machine learning and computer vision , the i.i.d . assumption , that training and test sets are sampled from the same distribution ( henceforth “ familiar ” distribution ) , is so prevalent as to be left unwritten . In experiments , it is easy to satisfy the i.i.d . condition by randomly sampling training and test data from a single pool , such as photos of employees ’ faces . But in real-life applications , test samples are often sampled differently ( e.g. , faces of internet users ) and may not be well-represented , if at all , by the training samples . Prior work ( Lakshminarayanan et al. , 2017 ) has shown networks to be unreliable when tested on semantically unrelated input ( e.g . feeding CIFAR into MNIST-trained networks ) , but users would not expect useful predictions on these data . However , we find this issue extends to semantically related input as well , such as gender classifiers applied to faces of older or younger people than those seen during training , which is a more common occurrence in practice and more problematic from a user ’ s perspective . We demonstrate that , counter to the intuition that unfamiliarity should lead to lack of confidence , current algorithms ( deep networks ) are more likely to make highly confident wrong predictions when faced with such “ novel ” samples , both for real-world image datasets ( Figure 1 ; see caption ) and for toy datasets ( Figure 2 ; see subcaption 2 ( a ) and Appendix A ) . The reason is simple : the classification function , such as a deep network , is undefined or loosely regulated for areas of the feature space that are unobserved in training , so the learner may extrapolate wildly without penalty . Confident errors on novel samples can be catastrophic . Whether one would ride in a self-driving car with a 99.9 % accurate vision system , probably depends on how well-behaved the car is on the 0.1 % mistakes . When a trained model labeled a person as a gorilla ( Zhang , 2015 ) , the public trust in that system was reduced . When a driving vision system confidently mistook a tractor trailer ( Yadron & Tynan , 2016 ) , a person died . Scholars that study the impact of AI on society consider differently distributed samples to be a major risk ( Varshney & Alemzadeh , 2017 ) : “ This is one form of epistemic Novel female , model : 99.3 % male ( ours : 53.7 % ) Novel female , model : 99.8 % male ( ours : 85.5 % ) Novel male , model : 99.9 % female ( ours : 84.5 % ) Novel cat , model : 100.0 % dog ( ours : 80.3 % ) Novel cat , model : 100.0 % dog ( ours : 93.3 % ) Novel dog , model : 100.0 % cat Novel fish , model : 99.3 % bird Novel bird , model : 99.7 % mammal Novel herptile , model : 99.4 % bird Novel fish , model : 99.0 % bird uncertainty that is quite relevant to safety because training on a dataset from a different distribution can cause much harm. ” Our trust in a system requires its ability to avoid confident errors . Unfortunately , these novel samples may differ from training in both expected and unexpected ways . This means gathering a set of “ unexpected ” training samples , though required by covariate shift and domain adaptation methods , becomes unviable ( Varshney & Alemzadeh , 2017 ) . One may use novelty detection methods to identify novel samples at test time ( to report an error or seek human guidance ) , but this may be insufficient . These “ outliers ” ( e.g . underrepresented faces that users upload ) are perfectly normal samples in the eyes of a user and they would reasonably expect a prediction . Also , human guidance may not be fast enough or affordable . In this paper , our aim is to reduce confident errors for predictions on samples from a different ( often non-overlapping ) but unknown distribution ( henceforth “ novel ” distribution ) . In contrast with most recent work , we focus on the confidence of the prediction rather than only the most likely prediction ( accuracy ) or the confidence ordering ( average precision or ROC ) . In addition to reviewing the effectiveness of established methods , we propose and evaluate two ideas that are straightforward extensions to ensemble and rejection methods . One is that multiple learners , with different initializations or subsamples of training data , may make different predictions on novel data ( see Lakshminarayanan et al . ( 2017 ) ) . Hence , ensembles of classifiers tend to be better behaved . But ensembles are slow to evaluate . If we distill ( Hinton et al. , 2015 ) an ensemble into a single model using the training data , the distilled classifier will have the original problem of being undefined for novel areas of the feature space . Fortunately , it is often possible to acquire many unlabeled examples , such as faces from a celebrity dataset . By distilling the ensemble on both the training set and on unsupervised examples , we can produce a single model that outperforms , in terms of prediction confidence , single and standard distilled models on both identically and differently distributed samples . Another idea is that if the training set does not provide enough information for the unseen data , therefore it may be desired to simply avoid confident predictions . We can lower their confidence according to the output of an off-the-shelf novelty detector . This means reducing confident errors by reducing confidence , regardless of correctness . It may improve novel sample prediction quality , but in turn degrade performance on familiar samples . Although this idea sounds like a natural extension to novelty detection , we are unaware of any implementation or analysis in the literature . Experimentally , we investigate the confidence problem and perform an extensive study by creating “ familiar ” and “ novel ” test splits , where “ familiar ” are identically distributed with training and −1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0 Ground truth −1 0 1 training data −1.0 −0.5 0.0 0.5 1.0 single model # 1 NLL −1.0 −0.5 0.0 0.5 1.0 single model # 2 NLL −1.0 −0.5 0.0 0.5 1.0 ensemble NLL −1.0 −0.5 0.0 0.5 1.0 distill NLL −1.0 −0.5 0.0 0.5 1.0 G-distill ( ours ) NLL 0 1 2 3 4 5 6 7 ( a ) Demonstration of deep networks ’ generalization behavior with a 2-dimensional toy dataset “ square ” . Column 1 : we design a binary ground truth for classification on a 2-dimensional feature space . Column 2 : for training , we only provide samples on the left and lower portions ( the “ familiar ” distribution ) , and reserve the upper-right only for testing ( the “ novel ” distribution ) . Column 3-7 : we show negative log likelihood ( NLL ) predicted for each point in the feature space . A small NLL indicates correct prediction , while a very large NLL indicates a confident error . Column 3 , 4 : multiple runs of the network have similar performances on familiar regions but vary substantially in novel regions where the training data imposes little or no regulation , due to optimization randomness . Column 5 : an ensemble of 10 such networks can smooth the predictions and reduce confident errors at the sacrifice of test efficiency . Column 6 : distilling the ensemble using the training samples results in the same irregularities as single models . Column 7 : one of our proposals is to distill the ensemble into a single model using both the labeled training data and unsupervised data from a ” general ” distribution . “ novel ” are not . For example , in cat vs. dog classification , the novel examples are from breeds not seen during training , or in gender classification , the novel examples are people that are older or younger than those seen during training . Our evaluation focuses on negative log likelihood ( NLL ) and the fraction of highly confident predictions that are misclassified ( “ E99 ” ) . They measure both prediction accuracy and how often confident errors occur . To summarize , our contributions are : • Draw attention to a counter-intuitive yet important problem of highly confident wrong predictions when samples are drawn from a unknown distribution that is different than training . • Evaluate and compare the effectiveness of methods in related fields , including two proposed methods to reduce such overconfident errors . • Propose an experimental methodology to study the problem by explicitly creating familiar and novel test splits and measuring performance with NLL and E99 . 2 RELATED WORK . To the best of our knowledge , there is no prior work with exactly the same goal and focus as this paper , but we are very similar to several lines of work . To avoid confusion , we compile Table 1 to show our differences from these prior work for your reading convenience . Epistemic uncertainty . The most related works model and reduce epistemic uncertainty ( model uncertainty from incomplete knowledge ) by estimating and evaluating a set or distribution of models that agree with the training data . The intuition is that on familiar test data , performance is boosted by aggregating outputs ; but for novel data , the different models can disagree , indicating a lack of training evidence . In that case , aggregating the outputs results in an appropriately lower confidence . Bayesian approaches ( Bishop , 1995 ; Blundell et al. , 2015 ; Hernández-Lobato et al. , 2016 ) estimate a distribution of network parameters and produce a Bayesian estimate for likelihood . These methods are usually very computationally intensive ( Lakshminarayanan et al. , 2017 ) , limiting their practical application . Gal & Ghahramani ( 2016 ) propose MC-dropout as a discrete approximation of Bayesian networks , adopting the dropout technique for a Monte-Carlo sample of likelihood estimates . They further propose to jointly estimate aleatoric and epistemic uncertainties ( Kendall & Gal , 2017 ) to increase the performance and quality of uncertainty . Lakshminarayanan et al . ( 2017 ) propose to instead use an ensemble to emulate a Bayesian network and achieve better performance . These works reduce uncertainty and improve generalization under i.i.d . assumptions . Our work differs in its focus of improving confidence estimates for novel samples ( differently distributed from training ) , and our method is much more efficient at test time than MC-dropout or ensembles . Hafner et al . ( 2018 ) reduces the confidence towards a prior on unseen data , which is similar to our proposed NCR method . We note that they perform regression on a low-dimensional feature space generalizing into future data , which is a very different problem from our image classification generalizing into unexpected samples . They use perturbed original dataset features and the prior as the labels for reducing confidence . This can be hard to generalize to an 224x224x3 image space . Domain adaptation ( Quionero-Candela et al. , 2009 ) aims at training on a source domain and improving performance on a slightly different target domain , either through unsupervised data or a small amount of supervised data in the target domain . In our settings , it is unviable to sample a representative “ unexpected dataset ” prior to test time , which we consider unknowable in advance . Consequently , the unsupervised samples we use are not from the familiar or novel distribution . Domain generalization ( Muandet et al. , 2013 ; Li et al. , 2017 ; Shankar et al. , 2018 ) is more closely related to our work , aiming to build models that generalize well on a previously unspecified domain , whose distribution can be different from all training domains . Unlike in domain adaptation , the target domain in this case is unknown prior to testing . These models generally build a domaininvariant feature space ( Muandet et al. , 2013 ) or a domain-invariant model ( Shankar et al. , 2018 ) , or factor models into domain-invariant and domain-specific parts ( Li et al. , 2017 ) . Also related are attribute-based approaches , such as Farhadi et al . ( 2009 ) , who build an attribute classifier that generalizes into novel object categories , similar to our experiment set-up . They select features that are discriminative for an attribute within individual classes to build invariance across object classes . These models all require multiple training domains to learn invariant representations , with the intent to improve robustness to variations in the target domain . In contrast , our method is concerned only with novel data and does not require multiple training domains . Some methods refuse to predict by estimating whether the system is likely to fail and outputting a signal requesting external intervention when necessary . Meta-recognition ( Scheirer et al. , 2011 ) estimates the performance based on the model output . Rejection options ( Fumera & Roli , 2002 ; Geifman & El-Yaniv , 2017 ) or failure detection ( Zhang et al. , 2014 ; Wang & Vasconcelos , 2018 ) 1Domain generalization requires several source domains , providing hints of the novel distribution ( see text ) . estimate the risk of misclassification to determine whether to refuse prediction , usually by looking at how close samples are to decision boundaries . Unlike our work , these works are not concerned with the quality of confidence estimates and do not analyze their rejection ability for samples from novel distributions . Outlier detection ( Devarakota et al. , 2007 ; Lee & Grauman , 2012 ; Liang et al. , 2018 ) , novelty detection , and one-class classification ( Tax , 2001 ; Khan & Madden , 2009 ) determine whether a test sample comes from the same distribution as the training data . The downside is that these methods would provide no confidence estimate for rejected samples , even though the models can still provide informative estimates ( e.g . in Section 5 gender classification , baseline accuracy on novel data is 85 % despite bad NLL performance ) . One of our methods naturally extends these methods by using their output to improve confidence estimates . Generalization . Various techniques have been proposed in deep learning literature to minimize the generalization gap , and popular ones include data augmentation , dropout ( Srivastava et al. , 2014 ) , batch normalization ( Ioffe & Szegedy , 2015 ) , and weight decay . Hoffer et al . ( 2017 ) propose better hyperparameter selection strategies for better generalization . Bagging ( Breiman , 1996 ) and other model averaging techniques are also used prior to deep learning . These methods focus on reducing generalization gap between training and validation . They do not address issues with unexpected novel samples and can be used in conjunction with our method . Theoretical analyses for off-training-set error ( Schaffer , 1994 ; Roos et al. , 2006 ) and empirical analysis of generalization for test samples ( Novak et al. , 2018 ) are also available , but these methods measure , rather than reduce , generalization errors . Calibration methods ( e.g . Guo et al . ( 2017 ) ) aim to improve confidence estimates , but since the confidence estimates are learned from familiar samples ( i.i.d . with training ) , risk is not reduced for novel samples . However , we experimentally find that ( Guo et al. , 2017 ) performs surprisingly well on unseen novel data , which the method is not optimized for . Distilling ( Hinton et al. , 2015 ) can be used on unsupervised samples . Radosavovic et al . ( 2017 ) obtain soft labels on transformed unlabeled data and use them to distill for unsupervised learning . Li & Hoiem ( 2017 ) extend models to new tasks without retaining old task data , using the new-task examples as unsupervised examples for the old tasks with a distillation loss . Distillation has also been used to reduce sensitivity to adversarial examples that are similar to training examples ( Papernot et al. , 2016 ) . Our work differs from all of these in the focus on producing accurate confidences for samples from novel distributions that may differ significantly from those observed in training . One-shot learning ( e.g . Vinyals et al . ( 2016 ) ) and zero-shot learning ( e.g . Zendel et al . ( 2017 ) ) aim to build a classifier through one sample or only metadata of the class . They focus on building a new class , while we focus on generalizing existing classes to novel samples within .

Summary: The authors proposed two methods to deal with estimating classification confidence on novel unseen data distributions. The first idea is to use ensemble methods as the base approach that helps in identifying uncertain cases and then using distillation methods to reduce the ensemble into a single model mimicking behavior of the ensemble. They propose a generalization of this idea, that is to also perform distillation on a more generic unsupervised data distribution (than the supervised one that is used in training the ensemble). It is not clear whether this distribution should overlap with the novel distribution as a requirement or not. The second idea is to use a novelty detector classifier and weight the network output by the novelty score.