Towards trustworthy predictions from deep neural networks with fast adversarial calibration

1 INTRODUCTION . To facilitate a wide-spread acceptance of AI systems guiding decision making in real-world applications , trustworthiness of deployed models is key . Not only in safety-critical applications such as autonomous driving or medicine ( Helldin et al. , 2013 ; Caruana et al. , 2015 ; Leibig et al. , 2017 ) , but also in dynamic open world systems in industry it is crucial for predictive models to be uncertaintyaware . Only if predictions are calibrated in the case of any gradual domain shift , covering the entire spectrum from in-domain ( `` known unknowns '' ) to truly out-of-domain samples ( `` unknown unknowns '' ) , they can be trusted . In particular in industrial and IoT settings , deployed models may encounter erroneous and inconsistent inputs far away from the input domain throughout the life-cycle ; in addition , the distribution of the input data may gradually move away from the distribution of the training data ( e.g . due to wear and tear of the assets , maintenance procedures or change in usage patterns ) . A variety of approaches to account for predictive uncertainty exist . They include post-processing steps for trained neural networks , where for example a validation set , drawn from the same distribution as the training data , is used to rescale the logit vectors returned by a trained neural network such that in-domain predictions are well calibrated ( Platt , 1999 ; Guo et al. , 2017 ) . Orthogonal approaches have been proposed where trust scores and other measures for out-of-distribution ( OOD ) detection are derived , typically also based on trained networks ( Liang et al. , 2018 ; Jiang et al. , 2018 ; Papernot & McDaniel , 2018 ) ; however these latter approaches are designed to detect only truly OOD samples and do not consider the continuum of domain shifts from in-domain to truly OOD . Alternative avenues towards intrinsically uncertainty-aware networks have been followed by training probabilistic models . In particular , a lot of research effort has been put into training Bayesian neural networks , where typically a prior distribution over the weights is specified and , given the training data , a posterior distribution over the weights is inferred . This distribution can then be used to quantify predictive uncertainty . Since exact inference is untractable , a range of approaches for approximate inference has been proposed.In particular approaches based on variational approximations have recently received a lot of attention and range from estimators of the fully factorized posterior ( Blundell et al. , 2015 ) , to the interpretation of Gaussian dropout as performing approximate Bayesian inference ( Gal & Ghahramani , 2016 ) and facilitating a complex posterior using normalising flows ( Louizos & Welling , 2017 ) . Since such Bayesian approaches often come at a high computational cost , alternative non-Bayesian approaches have been proposed , that can also account for predictive uncertainty . These include ensemble approaches , where smooth predictive estimates can be obtained by training ensembles of neural networks using adversarial examples ( Lakshminarayanan et al. , 2017 ) , and evidential deep learning , where predictions of a neural net are modelled as subjective opinions by placing a Dirichlet distribution on the class probabilities ( Sensoy et al. , 2018 ) . Both for Bayesian and non-Bayesian approaches , uncertainty-awareness and the quality of predictive uncertainty are typically evaluated by analysing the behaviour of the predictive entropy for out-of-domain predictions in form of gradual perturbations ( e.g . rotation of an image ) , adversarial examples or held-out classes . However , while an increasing predictive entropy for increasingly strong perturbations can be an indicator for uncertainty-awareness , simply high predictive entropy is not sufficient for trustworthy predictions . Models can only be trusted if the confidence of predictions is calibrated , that is if the entropy matches the actual accuracy of the model . For example , if the entropy is too high , the model will yield under-confident predictions and similarly , if the entropy is too low , predictions will be over-confident . Notably , the focus of related work introduced above has been on image data and it remains unclear how these approaches perform for other data modalities , in particular when modelling sequences with long-range dependencies using complex architectures such as LSTMs ( Hochreiter & Schmidhuber , 1997 ) or GRUs ( Cho et al. , 2014 ) . Here , we propose an efficient yet general modelling approach for obtaining calibrated , trustworthy probabilities for both in-domain samples as well as under domain shift that can readily be applied to a wide range of data modalities and model architectures . More specifically , we first introduce a simple loss function to encourage high entropy on wrong predictions and combine this with an adversarial calibration loss term . Since in practical applications it is a priori not clear what type or magnitude of domain drift will occur , we evaluate calibration under doamin drift for 10 different perturbations and 10 different noise levels not seen during training . Our contribution in this paper is three-fold . ( i ) we illustrate the limitations of entropy as measure for trustworthy predictions and motivate the use of the expected calibration error for quantifying technical robustness ( Dawid , 1982 ; DeGroot & Fienberg , 1983 ; Niculescu-Mizil & Caruana , 2005 ; Naeini et al. , 2015 ; Guo et al. , 2017 ) . ( ii ) we introduce a new training strategy combining an entropy-encouraging loss with an adversarial calibration loss term and demonstrate that this results in better calibration and technical trustworthiness of predictions for diverse types of out-of-domain samples and perturbations , compared to the state-of-the-art . ( iii ) We apply the concept of uncertainty-awareness and trustworthiness to sequence models and demonstrate that our approach substantially improves predictive uncertainty over existing approaches when classifying long sequences . While previous studies only compared predictive entropy for one simple architecture ( LeNet ) and typically one type of domain shift ( Sensoy et al. , 2018 ; Louizos & Welling , 2017 ) , we here present an extensive comparison of 4 different architectures across 10 different perturbation strategies . 2 TOWARDS TECHNICALLY TRUSTWORTHY PREDICTIONS . 2.1 LIMITATIONS OF ENTROPY AS MEASURE FOR UNCERTAINTY-AWARENESS . Recent efforts in terms of evaluating predictive uncertainty have focused on entropy as measure for uncertainty-awareness for predictions under domain shift . While entropy quantifies the uncertainty encoded in the model output , it is not clear what absolute entropy is required for a model to be reliable , given a set of samples from an out-of-domain distribution . For example , a popular evaluation strategy consists of computing the absolute entropy for out-of-domain samples generated using perturbation strategies based on the images in the test set ( e.g . gradual rotation of images ) ( Sensoy et al. , 2018 ; Louizos & Welling , 2017 ) . In this case , the entropy should increase with rotation angle , as the accuracy decreases in a coordinated fashion ( since the model was not trained with rotated images ) ( Fig . 1 ) . However , such evaluations alone are not sufficient to determine whether model predictions are technically reliable ( or trustworthy ) , since it is not clear whether accuracy and model confidence/uncertainty are coupled in a meaningful way . Building on prior work utilising the concept of calibration for in-domain predictions , this coupling can be quantified using reliability diagrams ( Guo et al. , 2017 ) , where the model confidence ( i.e . the probability associated with the predicted class label ) is linked to accuracy in a stratified manner . That is , for a given set of samples obtained under domain shift , confidence and accuracy should match for all confidence levels between 1/nclasses and 1.0 . For example , for the subset of samples with confidence between e.g . 60 % and 70 % the average accuracy should lie in that same range ; this relationship should hold for all intervals . Figure 1 illustrates that the accuracy decreases , while the entropy increases if perturbed images are fed to a trained neural network ( top right ) ; however , additional information directly linking the uncertainty or confidence of a model to its accuracy is required to establish whether predictions are calibrated . This is illustrated by reliability diagrams in figure 1 ( bottom row ) , showing accuracy as function of binned confidence and the expected calibration error ( ECE ) curve , summarizing the calibration gap by covering the entire spectrum of domain shifts . ( DeGroot & Fienberg , 1983 ; Niculescu-Mizil & Caruana , 2005 ) . 2.1.1 QUANTIFYING CALIBRATION UNDER DOMAIN SHIFT USING THE ECE CURVE . Let X ∈ RD and Y ∈ { 1 , . . . , C } be random variables that denote the D-dimensional input and labels in a classification task with C classes , respectively . Let h ( X ) = ( Ŷ , P̂ ) be the output of a neural network classifier h predicting a class Ŷ and associated confidence P̂ based on X . We follow Guo et al . ( 2017 ) and formally define perfect calibration such that confidence and accuracy match for all confidence levels : P ( Ŷ = Y |P̂ = p ) = p , ∀p ∈ [ 0 , 1 ] ( 1 ) This directly leads to a definition of miss-calibration mca as the difference in expectation between confidence and accuracy , that is mca = Ê P [ ∣∣P ( Ŷ = Y |P̂ = p ) − p∣∣ ] ( 2 ) mca can be estimated from finite samples , by partitioning predictions into M equally-spaced bins and computing a weighted average of the bins ’ difference between accuracy and confidence . The resulting measure is the expected calibration error ( ECE ) ( Naeini et al. , 2015 ) : ECE = M∑ m=1 |Bm| n ∣∣acc ( Bm ) − conf ( Bm ) ∣∣ ( 3 ) with Bm being the set of indices of samples whose prediction confidence falls into its associated interval Im . conf ( Bm ) and acc ( Bm ) are the average confidence and accuracy associated to Bm respectively and n the number of samples in the dataset . It can be shown that ECE is directly connected to miss-calibration as ECE using M bins converges to the M-term Riemann-Stieltjes sum of eq . 2 Guo et al . ( 2017 ) . Finally , ECE can also be interpreted as a summary measures of reliability diagrams since it quantifies the calibration gap ( red bars in figure 1 ) . ECE was recently popularised when used quantify in-domain calibration for modern neural networks Guo et al . ( 2017 ) . Here , we use an ECE based measure to evaluate the calibration of a predictive model under domain drift . Since the type of domain drift that may occur after training is generally not known a priori , we define a range of distinct perturbation types not seen during training . Each perturbation strategy mimics a scenario where the data a deployed model encounters stems from a distribution that gradually shifts away from the training distribution in a different manner . For each perturbation type we compute the ECE for a range of perturbation strengths . We then generate a ECE-perturbation curve and summarize overall calibration by computing a micro-averaged ECE across all perturbation strengths . ECE curves and micro-averaged ECE are primarily designed to evaluate calibration in domain-drift scenarios where the distribution the inputs are drawn from changes , but the set of labels remains unchanged . For truly OOD samples with labels that are not part of the train classes , entropy-based measure are a useful complementary approach to evaluate predictive uncertainty .

Summary: This paper proposed FALCON, a simple method to produce well-calibrated uncertainty estimation. The idea is to introduce two additional terms, one that directly encourage lower confidence for all negative classes of all data points, and another one that optimizes the ECE for adversarial samples. Experiments show that FALCON outperforms several state-of-the-art methods for calibrating neural network predictions.