Towards trustworthy predictions from deep neural networks with fast adversarial calibration
Authors: Christian Tomani
Florian Buettner
Venue:  (2020)
Keywords: deep learning
uncertainty
calibration
domain shift
robustness

Abstract: To facilitate a wide-spread acceptance of AI systems guiding decision making in real-world applications, trustworthiness of deployed models is key. That is, it is crucial for predictive models to be uncertainty-aware and yield well-calibrated (and thus trustworthy) predictions for both in-domain samples as well as under domain shift. Recent efforts to account for predictive uncertainty include post-processing steps for trained neural networks, Bayesian neural networks as well as alternative non-Bayesian approaches such as ensemble approaches and evidential deep learning. Here, we propose an efficient yet general modelling approach for obtaining well-calibrated, trustworthy probabilities for samples obtained after a domain shift. We introduce a new training strategy combining an entropy-encouraging loss term with an adversarial calibration loss term and demonstrate that this results in well-calibrated and technically trustworthy predictions for a wide range of perturbations. We comprehensively evaluate previously proposed approaches on different data modalities, a large range of data sets, network architectures and perturbation strategies and observe that our modelling approach substantially outperforms existing state-of-the-art approaches, yielding well-calibrated predictions for both in-domain and out-of domain samples. 

Under review as a conference paper at ICLR 2020
TOWARDS TRUSTWORTHY PREDICTIONS FROM DEEP NEURAL NETWORKS WITH FAST ADVERSARIAL CALI- BRATION
Anonymous authors Paper under double-blind review
ABSTRACT
To facilitate a wide-spread acceptance of Al systems guiding decision making real-world applications, trustworthiness of deployed models is key. That is, it is crucial for predictive models to be uncertainty-awee aware and yield well-calibrated (and thus trustworthy) predictions for both in-domain samples well as under domain shift. Recent efforts to account for predictive uncertainty include post-pron past-processing steps for trained neural networks, Bayesian neural networks well as alterna- tive non-Bayesian approaches such as ensemble approaches and evidential deep learning. Here, we propose an efficient yet general modelling approach for obtain- ing well-calibrated, trustworthy probabilities for samples obtained after domain shift. We introduce new training strategy combining an eecruupinnuurgggngg loss term with an adversarial calibration loss term and demonstrate that this re- sults in well-calibrated and technically trustworthy predictions for wide range of perturbations. We comprehensively evaluate previously proposed approaches on different data modalities, a large range of data sets including sequence data, network architectures and perturbation strategies and observe that our modelling approach substantially outperforms existing state-of-the-art approaches, yielding well-calibrated predictions under domain drift.
INTRODUCTION
To facilitate wide-spread acceptance of AI systems guiding decision making in real-world appli- cations, trustworthiness deployed models key. Not only in safety-critical applications such as autonomous driving or medicine (Helldin et al. 2013 Caruana et al. 2015 Leibig et al., 2017). but also in dynamic open world systems in industry aware. Only if predictions are calibrated in the case of any gradual domain shift, covering the entire spectrum from in-domain ("known unknowns") to truly out-of-domain samples ("unknown unknowns"), they can be trusted. In particular in industrial and loT settings, deployed models may encounter erroneous and inconsistent inputs far away from the input domain throughout the life-cycle; in addition, the distribution of the input data may gradually move away from the distribution of the training data (e.g. due to wear and tear of the assets, maintenance procedures or change in usage patterns).
crucial for predictive models be uncertainty-
A variety of approaches to account for predictive uncertainty exist. They include post-processing steps for trained neural networks, where for example validation set, drawn from the same distribution as the training data, is used to rescale the logit vectors returned by a trained neural network such that in-domain predictions are well calibrated (Plati) 1999 Guo et al. [2017). Orthogonal approaches have been proposed where trust scores and other measures for out-of out-of-die distriibbtiio (OOD) detection are derived, typically also based on trained networks (Liang et al.) 2018 8 Jiang et al., 2018: Papernot McDaniel 2018): however these latter approaches are designed to detect only truly OOD samples and do not consider the continuum of domain shifts from in-domain to truly OOD. Alternative avenues towards intrinsically uncertainty-awan networks have been followed by training probabilistic models. In particular, a lot of research effort has been put into training Bayesian neural networks, where typically prior distribution over the weights is specified and, given the training data, posterior distribution over the weights is inferred. This distribution can then be used to quantify predictive uncertainty. Since exact inference is untractable, range of approaches for approximate inference has

---

Under review as a conference paper at ICLR 2020
been proposed.In In particular approaches based on variational approximations have recently received lot of attention and range from estimators of the fully factorized posterior (Blundell et al. 20115), to the interpretation of Gaussian dropout performing approximate Bayesian inference (Gal & Ghahra mani] 2016) and facilitating a complex posterior using normalising flows (Louizos & Welling 2017)
Since such Bayesian approaches often come high computational cost, alternative non-Bayesian approaches have been proposed, that can also account for predictive uncertainty. These include ensemble approaches, where smooth predictive estimates can be obtained by training ensembles of neural networks using adversarial examples [Lakshminarayanaal et al. 2017), and evidential deep learning, where predictions of a neural net are modelled as subjective opinions by placing a Dirichlet distribution on the class probabilities (Sensoy et al.) 2018). Both for Bayesian and non-Bayesian approaches, uncerfainty-avareness and the quality of predictive uncertainty are typically evaluated by analysing the behaviour of the predictive entropy for out-of-domain predictions in form of grad- ual perturbations (e.g. rotation of an image), adversarial examples or held-out classes. However, while an increasing predictive entropy for increasingly strong perturbations can be an indicator for incertainy--awarenee
simply high predictive entropy is not sufficient for trustworthy predictions. Models can only be trusted if the confidence of predictions is calibrated, that is if the entropy matches the actual accuracy of the model. For example, if the entropy is too high, the model will yield under-confident predictions and similarly, if the entropy is too low, predictions will be over-confident. Notably, the focus of related work introduced above has been on image data and it remains unclear how these approaches perform for other data modalities. in particular when modelling sequences with long-range dependencies using complex architectures such as LSTMs (Hochreiter & Schmidhuber 1997) or GRUs (Cho et al.) 2014).
Here, we propose an efficient yet general modelling approach for obtaining calibrated, trustworthy probabilities for both in-domain samples as well as under domain shift that can readily be applied to a wide range of data modalities and model architectures. More specifically, we first introduce simple loss function to encourage high entropy on wrong predictions and combine this with an adversarial calibration loss term. Since in practical applications it is priori not clear what type magnitude of domain drift will occur, we evaluate calibration under doamin drift for 10 different perturbations and 10 different noise levels not seen during training.
Our contribution in this paper is three-fold. (i) we illustrate the limitations of entropy as measure for trustworthy predictions and motivate the use of the expected calibration error for quantifying technical robustness (Dawid. 1982; DeGroot & Fienberg 1983 Niculescu-Mizii & Caruana 2005) Naeini et al.
2015, Guo et al./22177) 2017). (ii) we introduce new training strategy combining an loss with an adversarial calibration loss term and demonstrate that this results in better calibration and technical trustworthiness of predictions for diverse types of out-of-domain samples and perturbations, compared to the state-of-the-an (iii) We apply the concept of uncertainty-awarent thiness to sequence models and demonstrate that our approach substantially improves predictive uncertainty over existing approaches when classifying long sequences. While previous studies only compared predictive entropy for one simple architecture (LeNet) and typically one type of domain shift (Sensoy al.] 2018 Louizos & Welling) 2017), we here present an extensive comparison of 4 different architectures across 10 different perturbation strategies.
and trustwor-
2 TOWARDS TECHNICALLY TRUSTWORTHY PREDICTIONS
2.1 LIMITATIONS OF ENTROPY AS MEASURE FOR UNCERPAINTY--WAREESSS
Recent efforts in terms of evaluating predictive uncertainty have focused on entropy as measure for mncertainty-awarenn for predictions under domain shift. While entropy quantifies the uncertainty encoded in the model output. is not clear what absolute entropy is required for a model to be reliable, given a set of samples from an out-of-domain distribution. For example, popular evaluation strategy consists of computing the absolute entropy for out-of-domain samples generated using perturbation strategies based on the images in the test set (e.g. gradual rotation of images) (Sensoy cet al. 20188] Louizos & Welling 2017). In this case, the entropy should increase with rotation angle, as the accuracy decreases in coordinated fashion (since the model was not trained with rotated images) (Fig. 1). However, such evaluations alone are not sufficient to determine whether model predictions are technically reliable (or trustworthy), since it is not clear whether accuracy and model confidence/uncceraiityy are coupled in a meaningful way. Building on prior work utilising the concept of calibration for in-domain predictions, this coupling can be quantified using reliability diagrams
2

---

Under review as a conference paper at ICLR 2020
(Guo et al. 2017), where the model confidence (i.e. the probability associated with the predicted class label) is linked to accuracy in a stratified manner. That is, for a given set of samples obtained under domain shift, confidence and accuracy should match for all confidence levels between 1/ and 1.0. For example, for the subset of samples with confidence between e.g. 60% and 70% the average accuracy should lie in that same range; this relationship should hold for all intervals. Figure illustrates that the accuracy decreases, while the entropy increases if perturbed images are fed to trained neural network (top right); however, additional information directly linking the uncertainty or confidence of model to its accuracy is required to establish whether predictions are calibrated. This is illustrated by reliability diagrams in figure (bottom row), showing accuracy as function of binned confidence and the expected calibration error (ECE) curve, summarizing the calibration gap by covering the entire spectrum of domain shifts. (DeGroot Fienberg. 1983 Nieulescu-Mizil
Caruana 2005).
Saftwax probabllty
1.0
1.0
0.6
0.8
0.6
Accuraty Entropy
Entropy
0.4
0.2
0.0
0.0
20
40
60
80
Epsilon
20
40
80
Epailon
Epsiion:
Epslion: 60
0.6
0.4
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
20
40
60
80
Confidence
Confidence
Epsilon
Figure 1: Calibration of the predictive uncertainty under domain shift. Here, a LeNet model is trained on MNIST data and calibration of the predictive uncertainty is evaluated on images perturbed with increasing y-zoom. Epsilon denotes the relative perturbation strength. Top: For in-domain samples the model has high accuracy and low entropy, for higher domain shifts wrong predictions are often made with high confidence (left). While increasing domain shift results in a decreased accuracy and increased entropy, it is not clear whether this increased entropy reflects a well calibrated model confidence (right). Bottom: Only reliability diagrams and the expected calibration eтror (ECE) reveal that the decline in accuracy does not match the confidence of the model. Left: Confidence matches accuracy for most bins. Middle: Model makes overconfident predictions (red bars illustrate calibration gap). Right: ECE curve quantifies how miss-calibration changes with increasing perturbation strength.
2.1.1 QUANTIPYING CALIBRATION UNDER DOMAIN SHIFT USING THE ECE CURVE
Let X E R and Y (1...., C} be random variables that denote the D-dimensional input and labels in classification task with C' classes, respectively. Let h(X) (Ý², P)) be the output of neural network classifier predicting class y and associated confidence Í based on X. follow Guo et al. (201)) and formally define perfect calibration such that confidence and accuracy match for all confidence levels:
P(Ý =
Vp E [0.
(I)
This directly leads to a definition of miss-calibration mea as the difference expectation between confidence and accuracy, that is
mca E y -
mca can be estimated from finite samples, by partitioning predictions into M equally-spaced bins and computing weighted average of the bins' difference between accuracy and confidence. The
3

---

Under review as a conference paper at ICLR 2020
resulting measure is the expected calibration error (ECE) (Nacini et al. 2015):
MM ECE = Σ
[mll acc Bmm) co.](Bmm)
with Bm being the set of indices of samples whose prediction confidence falls into its associated interval Im- conf Bm)) and acc Bm) are the average confidence and accuracy associated to Bm respectively and n. the number of samples in the dataset.
It can be shown that ECE is directly connected miss-calibration as ECE using M bins converges the M-term Riemann- Stiille- sum of eq. Guo et al. (2017). Finally, ECE can also be interpreted as summary measures of reliability diagrams since it quantifies the calibration gap (red bars in figure 1).
ECE was recently popularised when used quantify in in-domain calibration for modern neural networks Guo et al. (2017). Here, we use an ECE based measure to evaluate the calibration of a predictive model under domain drift. Since the type of domain drift that may occur after training generally not known a priori, we define range of distinct perturbation types not seen during training. Each perturbation strategy mimics a scenario where the data a deployed model encounters stems from a distribution that gradually shifts away from the training distribution in different manner. For each perturbation type we compute the ECE for a range of perturbation strengths. We then generate a DCE-perturbation curve and summarize overall calibration by computing a micro-averaged ECE across all perturbation strengths.
ECE curves and micro-averaged ECE are primarily designed to evaluate calibration in domain-drift scenarios where the distribution the inputs are drawn from changes, but the set of labels remains unchanged. For truly OOD samples with labels that are not part of the train classes, entropy- -based measure are useful complementary approach to evaluate predictive uncertainty.
2.2
A SIMPLE APPROACH FOR CALIBRATED PREDICTIVE UNCERTAINTY ESTIMATION
2.2. 1
PREDICTIVE ENTROPY
To mitigate overconfident predictions displayed by conventional deep neural networks, we first introduce a loss term encouraging a uniform distribution the scores in case the model "does not know". That is, we distribute the probability mass of false predictions uniformly over C' classes:
C
Ls -E=Σ--
1
log(p1((1 vis) + Mis)
with Pty being the confidence associated to the jth class of sample i. Vú its one-hot encoded label. by encouraging an increased entropy (S) in
This simple loss term increases the presence of high predictive uncertainty without directly affecting reconstruction loss (categorical cross-entropy) and thus accuracy. This has the advantage that our approach - in contrast to state- of-the-art Bayesian neural networks such as those based on multiplicative normalizing flows or evidential deep learning- can be readily applied to complex architectures based on LSTMs or GRUs. In addition, the loss term is parameter free and thus does not require hyperparameter tuning, again facilitating easy usage.
2.2.2 ADVERSARIAL CALIBRATION
While the to introduce additional loss term addressing model calibration directly. Explicitly encouraging calibration for out-of-domanl samples, however e.g. via an ECE-based measure requires knowledge on the type of perturbed or erroneous samples the model is expected to encounter. In many real- world applications it is not clear from which distribution these samples will be drawn and, for model predictions to be truly trustworthy requires robustness against all such potential out-of- domain samples. That is, we would like our model to be technically robust for inputs around an n-eighbouurhood of the in-domain training samples, for a wide range of e and for all 2D directions in 1)P, While inputs from a random direction are unlikely to be representative examples for generic domann samples, by definition adversarial examples are generated along dimension where the loss is high. Lakshminarayana. et al. (2017) show that adversarial training can improve the smoothness of predictions, in particular when training an ensemble of neural networks in an adversarial fashion. Here, we demonstrate that using adversarial samples to directly optimise model
loss term does encourage uncertainty-awaaeeess. we found that it is beneficial
4

---

Under review as a conference paper at ICLR 2020
calibration (rather than the squared error of one-hot encoded labels (Lakshminarayanan al. 2017] results in substantially more trustworthy predictions for out-of-domman samples from a large number
of unrelated directions.
We implement an ECE-inspired calibration loss by minimizing miss ciss-callbbatioon for samples generated using the fast gradient sign method (FGSM) (Goodfellow al. 2014). with ranging from to 0.5 (sampled at 10 equally spaced bins at random). To this end we minimise the L2 norm of the difference between the predicted confidence of sample i, which we denote as conf(i), and its corresponding binned accuracy aco( ). for all samples. This is directly motivated by the definition of ECE (eq. [. and as for the computation of ECE, we partition the predictions of the network into M equally-spaced bins, with my {1,.... MJ being bin into which sample i falls. As for ECE, Bm, is the set of indices of samples falling bin m, and aco( (mm;) the average accuracy of samples Bmn We set M = 10 for all experiments.
7
Lady
(aco (mm)) - ctn)
(5)
The final loss balancing standard reconstruction loss (categorical cross entropy (CCE) against the entropy and adversarial calibration loss can then be written as L = LOCE + Aadv + ÀgLş. The choice of hyperparameters Aadv and Äs is described in the appendix along with a robustness analysis, an ablation study for both loss terms and a summary of the algorithm. Note that we do not use the FGSM samples for adversarial training in the sense that do not try to minimize the reconstruction error (cross entropy) for those samples.
3 EXPERIMENTAL RESULTS
We compare our approach for fast adversarial calibration to both Bayesian and non-Bayesian work and perform extensive set of experiments. We evaluate model trustworthiness by quantifying model calibration for 10 distinct strategies to generate out-of-domain samples. We show that our approach is able to yield technically trustworthy predictions across datasets, model architectures and three data modalities. We assess 9 distinct image-based perturbation types including left rotation, right rotation, shift in x direction, shift in y direction, xy shift, shear, zoom in x direction, zoom in direction and xy zoom for image data. In addition, we investigate robustness to random word swaps for text data. More specifically, perturbation is generated by first drawing a random set of words in corpus. Next each of these words is replaced by word drawn at random from the vocabulary. For all perturbation strategics, perturbed samples were generated at 10 different levels, starting no perturbation, until accuracy reached random levels; relative perturbation strength is denoted by epsilon. The micro-averaged ECE for a specific perturbation strategy was computed by first perturbing each sample in the test set at 10 different levels and then calculating the overall ECE across all samples. By computing this miero-averaged ECE for 100 different perturbation scenarios, we quantify the ability of neural networks to yield well-calibrated, technically robust predictions in diverse circumstances.
We first show that our modelling approach substantially outperforms existing approaches for sequence models and then illustrate improved performance for image data. To evaluate our modelling approach for sequence data, we models on the following datasets:
1. Sequential MNIST. 10 classes of handwritten digits. Images are converted pixel-wise sequences of length 28x28.
2. 20 Newsgroups. News articles partitioned into 20 classes News classes are modelled as sequences of words using word embeddings. We used the 20,000 most common words as vocabulary and a maximum word length of 2500.
We fitted LSTM and GRU models with one hidden layer for all sequence modelling tasks. For the image classification tasks, we fitted a LeNet model to MNIST data in order to establish fair comparison to the ofate-of-the-art (Guo et al., 2017 7 Sensoy et al. 2018). To evaluate the performance for more complex architectures, further fitted deep neural net with VGG19 architecture on the CIFAR10 10 dataset. We used standard splits into training and test set for all datasets.
We compared the following modelling approaches: (i) 1.2 1.2-Dropout, referring to standard neural
5

---

Under review as a conference paper at ICLR 2020
0.6
0.6
L2- Dropout
Deep-Ersembles FALCON (ours)
L2- D-orrpuu
Deep-Ensemblee Ensembbles
MC-Dropout
0.5
Dropout
FALCON lours)
0.5
0.4
ECE
0.4
avg ECE
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
rot-lett
rot- rot-right
shear
xy-shift
x-shift
y-shift
xy-zaom
x-zoom
y-zoom
lot-left
rot-right
shear
xy-shift
x-shift
y-sbift
xy zy-zoom
x-200m
y-20om
(a) LSTM models
(b) GRU models
Figure 2: Technical robustness of sequence models for classifying sequential MNIST data, quantified by computing the micro- averaged expected calibration error (lower is better). FALCON results in consistently well calibrated and robust predictions across 9 different perturbation strategies with substantially lower micro-averaged ECEs compared to existing methods, both for LSTM and GRU models. For fair comparison, only show micro-averaged ECE for models with competitive accuracy, omitting EDL
net with L. regularisation as baseline, (ii) MC-Dropoul corresponding to the modelling approach presented by Gal & Ghahramani (2016), (iii) Deep Ensembles referring to an approach based on an ensemble of neural nets trained using adversarial examples [Lakshminarayanan et al. 2017). (iv)
EDL referring to Evidential Deep Learning (Sensoy et al. 2018). (v) MNF referring to a Bayesian neural network trained using multiplicative normalising flows (Louizos & Welling 2017) and (vi) FALCON, which is our method based on Fast Adversarial. CalibratiON. Additional comparisons to temperature scaling (Guo et al. 2017) and stochasctic variational inference (SVI) based on Flipout (Wen et al. 2018) are shown in the Appendix A.5
3.1
PREDICTIVE UNCERTAINTY FOR SEQUENCE MODELING
We trained LSTM models with one hidden layer of 130 hidden units using the RMSPROP optimizer. GRU models were trained with one hidden layer 250 hidden units to reflect the reduced complexity of GRU cells compared to LSTM cells. The Bayesian neural network based multiplicative normalizing flows (MNF) was developed for convolutional neural networks; since the transfer of such complex modelling approach from convolutional neural networks to recurrent neural networks is out of the scope of this work, we omitted MNF in our comparison of sequence models.
Sequential MNIST For deep ensembles of LSTMs trained on sequential MNIST we found that models did not converge when training the networks with adversarial examples; we therefore also trained ensembles with a reduced e of 0.005 and report performance for this modified Deep Ensemble approach. For the deep ensemble of GRUs sequential MNIST and the deep ensemble of LSTMs on the 20 Newsgroups data, we report performance with standard adversarial training (E = 0.01).
Fitting LSTM models on sequential MNIST is a challenging task (Bai et al. 2018), and it only possible to achieve state-of-the-art predictive power with EDL for shorter sequences (downsampling of images before conversion to sequence). While performance of GRUs approaches, EDL also did not achieve competitive accuracy (Table S2 in appendix A.7). We found that our approach achieved competitive predictive power for LSTM and GRU models and substantially improved calibration of the predictive uncertainty for both models (Figure 2, Table 1). This illustrates that in contrast existing approaches FALCON is able to yield well-calibrate and trustworthy predictions without compromising on accuracy, even for challenging tasks such as classifying long sequences with LSTMS.
better for all modelling
20 Newsgroups
To further evaluate the ability of FALCON to model sequence data, compared the performance of FALCON to existing approaches for an NLP task. To this end, we trained LSTMs to classify news articles into one of 20 classes. We generated vector representations of words using

---

Under review as a conference paper at ICLR 2020
1.0
1.0
0.5
Softmax probablity
0.8
0.4
0.8
MOx Engrpyy
0.6
Accuracy
0.3
C.6
0.4
0.2
0.4
02
0.0
0.0
0:2
0.0
25
50
75
20
40 Epsilon
60
80
Epsilon
0
20
40
60
80
Epsilon
Deep-Emeembles
MC-Drapout
0
WNF
FALCON (ours)
Figure 4: Calibration of the predictive uncertainty under domain shift generated by increasing the y-zoom of each image the test set in 10 steps (MNIST data). Left With increasing domain shift the confidence of predictions with FALCON decreases such that they match accuracy (c.f. overconfident predictions of same samples with L2 in Fig. 1). Middle: expected calibration error at 10 increasingly large levels of y-zoom. Only EDL and FALCON maintain low ECE across all levels of y-zoom. Right: Entropy increases with larger y-zoom for all methods. While EDL starts at the highest entropy, this reflects under-confident predictions for low levels of perturbation (c.f. high ECE middle panel, figure S3 (appendix)). Accuracy decreases with larger zoom to almost random levels.
the pre-trained GLOVE embedding (length 100) and used the first 2500 words of an article as input for an LSTM. We trained LSTMs with one hidden layer of 130 hidden units and evaluated it on perturbation strategy based on random word swaps. To establish a perturbation strategy with gradually increasing perturbations, we varied the fraction of words drawn from each sample between 0% and 45% in 5% steps (gradually decreasing accuracy to random levels).
Similar to the LSTM model trained on sequential MNIST, we found that EDL did not achieve competitive predictive power, with accuracy of 49.3% only. In contrast, FALCON resulted in well-calibrated predictions while maintaining a competitive accuracy of 75.7%, compared to 75.9%, 72.8% and 77.3% for L.2-Dropout, MC-Dropout and Deep Ensemble respectively. As before, the model confidence FALCON was substantially better calibrated than existing methods (Figure 3).
3.2 PREDICTIVE UNCERTAINTY FOR IMAGE CLASSIFICATION
We next evaluated the trustworthiness of predictions for image classifi- cation tasks. To establish a fair comparison with state-of-the-eee the-art models, including Bayesian neural networks, we first trained the existing ap- proaches and evaluated them on 9 different perturbation strategies (not used during training). While with increasingly strong perturbations the predictive entropy increased for all models, this was not necessarily matched by a good calibration across the range of the perturbation. At the typical example of the perturbation y-zoom, it becomes clear that for most methods entropy did not increase sufficiently fast to match the decrease in accuracy, resulting in increasingly overconfident predictions and an increasing BCE for stronger perturbations (Fig. 4. While FAL- CON and EDL yielded well-calibrated predictions that were robust across all perturbation levels, it is worth noting that EDL has a substantially higher ECE for in-domain predictions. reflecting under-confident pre- dictions on the test set (see also Fig. SS4 S5 appendix). We observed this tendency of EDL towards under-confidence when faced with new samples drawn from the same distribution as the training data (known unknowns) also for different dataset and architecture (VGG19 on CI- FARI0: 10; ECEFAL.ON = 0.107. ECEEDL = 0.125 on the test set). While predictions with FALCON were also slightly under- confident for low levels of perturbation, this has arguably a smaller practical relevance than the substantial overconfidence of other baseline methods (c.f appendix A. (A.6 for more in-depth analysis). We observed a similar behaviour across all other 8 perturbation strategies, which reflected in the lowest micro-averaged ECE for FALCON, followed by EDL (Figure Table T
0.6
L2 Drgprp RC 3ropett Desp-Enaemtle
0.5-
FALCON courtt
0.3
0.1-
0.0
character
Figure 3: Expected cali- bration error for 20 News- groups data.
7

---

Under review as a conference paper at ICLR 2020
1.0
0.6
L2- 12-Dropout
L2-Drapout
EDL FALCON tours)
MC Drapout
MC.roppuut
0.5
Deep-Ensemblee
Deep Ensembeeees
avg ECE
0.4
FDI
avg ECE
0.6
FALCON lours)
0.3
0.4
0.2
0.2
0.0
rot-left
rat-right
shear
xy-shift
x-shift
y-shift
xy Ny-zoum
x-zoam
y-zoom
0.0
rat-left
robright
shear
xy sy-shiff
x-shift
y-shift
Xy- xy-zoom
x-zoom
y-zaom
(a) LeNet model MNIST data
(b) LeNet model for CIFARIO data
Figure 5: Technical robustness of image classification models, quantified by computing the micro- averaged expected calibration error (lower is better). FALCON results consistently well calibrated and robust predictions across different perturbation strategies.
To evaluate the technical robustness and calibration of FALCON on more complex architecture for image classification, we trained a VGG19 model on the CIFAR1O 10 dataset. We again observed similar trend as for the MNIST data, with FALCON yielding well calibrated predictions across all perturbation strategies (Figure 5). 'he considerable overconfidence of most baseline methods (except EDL.) when making predictions on OOD samples (epsilon 90) is also reflected by a substantially lower maximum entropy (Fig. [S7] Appendix A.7) and the overall distribution of confidence scores (Fig. S8 Note that we omitted MNF due to the large memory requirements stemming from the use of multipiicative normalising flows.
Table 1: Test accuracy and mean ECE across all perturbation strategies for the LeNet model trained on MNIST and the VGG19 model trained CIFAR10
LeNS-
VGGI9-CIFARI
Test acc.
Mean ECE
Test acc.
Mean ECE
L2-Dropout
0.99
0.243
0.88
0.57
MC-Dropout
0.992
0.179
0.839
0.377
MNF
0.993
0.197
NA
NA
Deep-Ensemble
0.98
0.242
0.847
0.334
EDL
0.989
0.102
0.876
0.197
FALCON
0.991
0.082
0.871
0.146
4 DISCUSSION AND CONCLUSION
We presented a fast, simple and generalizable approach for encouraging well-calibrated uncertainty- awareness of deep neural networks. To this end, combine entropy encouraging loss-term with an adversarial calibration loss and show on diverse data modalities and model architectures that our approach yields well-calibrated predictions for both in-domain and out-of-domain samples generated based on 10 distinct perturbations. We present a detailed analysis of calibration under domain drift for recurrent neural networks and identify major drawbacks of existing methods that were developed for (and evaluated on) image classification tasks. Thus, EDL was only able to result in networks with a high accuracy when trained on short sequences; both for the sequential MNIST and and 20 Newsgroups data, the EDL approach resulted in substantially lower accuracy compared to baseline LSTM and GRU models. While MC dropout is easy to fit and fast, it results only small improvements over the L2- L2-Dropout baseline, especially for sequence data. In contrast, modeling approach fast and robust, with well-calibrated predictive uncertainty across 10 perturbations, datasets, 4 model architectures and three data modalities.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.011 180.00121,, 2018.
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv: 1.505054244 2015.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 1721-1730. ACM, 2015.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. arXiv preprint 1406. 1078, 2014.
A Philip Dawid. The well-calibrated bayesian. Journal of the American Statistical Association, 77 (379):605-610, 1982.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation forecasters. Journal of the Royal Statistical Sociery: Series D (The Statistician), 32(1-2):12-22 1983.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1321-1330. JMLR. org, 2017.
Tove Helldin, Göran Falkman, Maria Riveiro, and Staffan Davidsson. Presenting system uncertainty in automotive uis for supporting trust calibration in autonomous driving. In Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, pp. 210-217. ACM, 2013.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 9(8): 1735-1780, 1997.
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust a classifier. In Advances in Neural Information Processing Systems, pp. 5541-5552, 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp- 6402 6413, 2017.
Christian Leibig, Vaneeda Allken, Murat Seçkin Ayhan, Philipp Berens, and Siegfried Wahl. Lever- aging uncertainty information from deep neural networks for disease detection. Scientific reports, 7716:17816, 2017.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribattio im- age detection neural networks. 2018. URL http:/psss//ppeeeeeeeeeeeeeeeees wttp://wwwwweeeefrfreeeee H1VGkIxRR
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. International Conference on Machine Learning, pp. 2218-2227, 2017. URI.
ngs m press/vi0 lou zos 7a html
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-iint AAAI Conference on Artificial Intelligence, 2015.

---

Under review as a conference paper at ICLR 2020
Alexandru Niculescu- and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pp. ACM, 2005.
Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv: 1803.04765, 2018.
John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In ADVANCES IN LARGE MARGIN CLASSIFIERS, pp. 61-74. MIT Press, 1999.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In Advances in Neural Information Processing Systems, pp. 3179-3189, 2018.
Yeming Wen, Paul Vicol, Jimmy Ba. Dustin Tran, and Roger Grosse. Flipout: Efficient pseudo- independent weight perturbations on mini-batches. arXiv preprint /rXiv: 1803.04366 2018.
A APPENDIX
A.1 PARAMETER AND HYPERPARAMETER SETTINGS
Deep Ensembles, MNF, and EDL were trained with default values for method-specific hyper- parameters (e-g. number of neural networks in a Deep Ensemble). In addition, the following hyperparameters were picked using hyperparameter searches. For all methods, the learning rate was chosen from {le - 5, 5e 5, 1e 4, 5e 4, le 3, 5e 3}. In addition, for the baseline method (L2), our method (FALCON), Deep Ensembles and EDL, dropout was chosen from {0, 0.5) and L2-regularisation from {0.0, 0.001, 0.005, 0.01, 0.05 For EDL we chose the KL regularisation from (0.5, 1., 5., 15., 30., 50.} For fair comparison with this state-of-the the art model, we chose As from this same set of values for FALCON and Xado from {0. 25,25, le 1, le 2, le 3, le 4). We also assessed the effect of the individual loss terms Aado and Äs and the robustness of FALCON with respect to the choice of As and Aodv (section A.4).
We used a batch size of 128 for all models and standard splits in train and test data for all datasets.
For the 20 Newsgroups dataset we used the keras tokenizer to format text sam- ples, converting words into lower case, removing punctuation and special characters !"#$5& ()+, -/:/,,/-//////////11 (')'''"'''''''''
A.2 PERTURBATION STRATEGIES
In practice it is not clear what type of perturbation model may encounter. To assess how neural networks cope in diverse settings, we generated out-of-domain samples based on 10 different perturbation strategies. Each perturbation strategy mimics scenario where the data deployed model encounters stems from a distribution that gradually shifts away from the training distribution in different manner. Samples generated with maximum perturbation strength correspond for example to corrupted or erroneous samples a deployed model may face, unperturbed samples correspond to those drawn from the same distribution as the training data ("known unknowns"). Trustworthy AI models should yield well-calibrated confidence scores in all those settings that it may encounter throughout its life-cycle. We quantify this based on the expected calibration error, micro-averaged across all perturbation strengths. including no perturbation (Tables S6-S10).
For all perturbation strategies we chose 10 levels of perturbation, starting at no perturbation, such that accuracy levels were close to random for maximum perturbation strength (Table (S5). Specific levels of perturbation are listed in Table S4 for visualisation purposes we re-scaled all perturbation-speecfic parameters to range from O to 90 (in steps of 10) and denote this general perturbation strength as epsilon. Perturbations include image transformations (rotation, shift, zoom, shear) as well as a word perturbation (word swap). For sequential MNIST, perturbations performed on the image before transforming the image to a sequence.
10

---

Under review as a conference paper at ICLR 2020
A.3 TRAINING ALGORITHM
Training was performed following Algorithm 1, summarizing the description in section 2.2.
Algorithm FALCON with set of perturbation levels {0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45) (n.b. encourages in-domain calibration). mini batch size number of ECE bins M and training set (X, Y). my (I,..., M) denotes the bin into which sample i falls and Bm; is the set of indices of samples falling in bin m,. acc(Bm.)) the average accuracy of samples Bm..
1: repeat
2:
Read minibatch MB (( Ko}: (Yi..... Yo)) from training set Randomly sample EMB from 3
3:
4:
Generate FGSM minibatch M Bade of size from samples in M B using CMB
5:
Compute LOcF and Ls and do one training step using mini batch MB
6:
B and partition into M equally spaced bins Compute binned accuracy aco( (m:)) for all samples i in M Bade
Compute predictions for samples
7:
8:
Compute Lud based on Bade and do one training step using M Badv
9:
until training converged
A.4 ABLATION STUDY AND SENSITIVITY ANALYSIS
In order to investigate the influence of the individual loss terms on calibration, we performed an ablation study, omitting one of the two loss terms, Lg and Ladv respectively. While either loss term results in an improved calibration compared to the L2-dropout baseline, combining both terms yields consistently better results.
0.6
L2-Dropout
FALCON without Entropy Loss (ours)
0.5
FALCON without Adversarial Loss
FALCON (ours)
avg ECE
0.4
0.3
0.1
0.0
rot-left
rot-right
shear
xy-shift
x-shift
y-shift
xy m
x-z00m
y-zoom
Figure S1: Micro-averaged BCE for FALCON with only one loss term loss term, based on the LeNet model trained on MNIST.
We further performed a robustness analysis in order to quantify the dependence of model performance on calibration under domain drift. To this end, trained LeNet models on MNIST varying both hyperparameters over wide ranges. We first fixed Äs at the optimal value of 50 and varied Aade between 0.0005 and 0.03 and computed the micro-averaged ECE for perturbation yzoom for all hyperparameter combinations. Next, we fixed Aadv at the optimal value of 0.02 and varied Às between 10 and 100. We found that even when varying both hyperparameters over wide range, ECE remained robust and varied by less than 0.04 for Xade and less than 0.06 for Äg. Accuracy was not affected by the choice of either À and remained between 0.985 and 0.991.
11

---

Under review as a conference paper at ICLR 2020
0.20
0.20
ECE - FAUrON(ours)
ECE FALCONNours)
0.15
0.15
ECE
ECE
0.10
0.10
0.05
0.05
0.00
0.00
40
60
80
100
0.05
0.5
1.5
2.5
lambda_s
lambda adv (E-2)
_ambda_adv
(a) Robustness of Às
(b) Robustness of Aado
Figure S2:
Robustness of hyperparameters. Even when varying both hyperparameters, micro-averged
ECE for perturbation yzoom robust.
A.5 ADDITIONAL BASELINES
In addition to the baselines evaluated in depth the main text, we also assessed two additional baselines, namely SVI based on Flipout and temperature scaling. We found that temperature scaling, while improving in-domain calibration, not beneficial for calibration under domain-drift (Fig. [S3] In addition, observe that SVI results in decreased accuracy compared to other basclines and while calibration under domain drift improves in comparison to L.2-droouut is substantially worse than FALCON and also EDL (Table [ST]. Finally, we observe that FALCON has substantially better performance than both SVI and temperature scaling for truly OOD samples (corresponding to strongest perturbations), which is reflected in a higher entropy and more overconfident confidence scores (Figs. S7 S8
ECE
3vg
12-Dropout
MC-Dropout
0.1
Deep Ensembles
SVI
0.0
rat-left
rot rot-right
shear
xy-shift
x-shirt
y-shift
xy-zoom
k-zoom
1-200m
FALCON lours)
Figure S3: Calibration under domain shift of the LeNet model for MNIST data, quantified by computing the micro- averaged expected calibration error (lower is better). FALCON results in consistently well calibrated and robust predictions across 9 different perturbation strategics. L2-TS stands for temperature scaling, SVI stands for Stochastic Variational Inference.
12

---

Under review as a conference paper at ICLR 2020
Table S1:
Test accuracy and mean ECE across all 9 perturbation strategies for the LeNet model trained on MNIST, including additional baselines L2-TS and SVI.
Test acc.
Mean ECE
1.2-Droouut
0.99
0.243
MC-Dropout
0.992
0.179
MNF
0,993
0.197
Deep-Ensembles
0.98
0.242
EDL
0,989
0.102
L2-TS
0.99
0.236
SVI
0.974
0.176
FALCON
0.991
0.082
A.6 UNDER-CONFIDENCE OF FALCON AND EDL FOR SMALL PERTURBATIONS
We observe e.g. in Fig. 4 (middle), that FALCON and EDL have tendency to make slightly under- confident predictions small perturbations. To further investigate the extend and practical relevance of this under-confidence, we generated an empirical CDF of the confidence scores for all approaches. This confirms the issue, but also illustrates that FALCON is consistently less under-confident than EDL: While for FALCON 13.3% of predictions
the test set are made with confidence of less than 90%, this increases to 30.5% for EDL, which is substantially higher than the 7.7% we observe for the well-calibrated Deep Ensembles. In other words, only minority of 5.6% of all predictions made by FALCON are made with a confidence of less than 90% and should have been made with a higher confidence. This can also be seen at the level of individual samples, as illustrated in figure
1.0
12-Drapaut
Mr-Drtt
MNF
Confidence scores
0.8
Doep-Ensembles
EDL
Probability
FALCON (ours)
0.6
0.4
0.2
0.2
0.0
2
EDL
FALCON
0.2
0.4
0.6
0.8
1.0
Confidence
(a) Empirical CDF of confidence levels all ap-(b) Boxplots across all confidence levels. The dashed line indicates the accuracy (I.2-dropout).
proaches.
Figure $4: Distribution of confidence levels for peturbation. EDL makes the most underconfident predictions, which can be seen both in the boxplot as well as in the empirical CDF. While FALCON also makes underconfident predictions, this is notably less severe than for EDL.
EDL makes noticeably under-confident predictions for small domain shifts before the entropy increases and confidence scores match accuracy. While FALCON also makes slightly under-confident predictions for in-domain samples, the corresponding confidence scores are still substantially closer to (matching the near perfect test accuracies for MNIST). Like EDL, FALCON does not make over-confident predictions when moving further away from the training domain (epsilon greater 40). This substantial over-confidence c.g. illustrated in figure S8 has arguably a much bigger effect on decision making process than the slight under-confidence of FALCON affecting only small fraction of samples: an underconfident prediction of 89% rather than well-calibrated 95% is unlikely to affect decision making process real world application the same extent an overconfidence of 80% rather than well-calibrated 10% (maximum entropy).
S5
13

---

Under review as a conference paper at ICLR 2020
L2-Drrpout
1.0
Softmar probabiltt
0.0
0.6
0.2
0.0
0
20
40
60
Epsilon
EDL
FALCON
Softmes probabilty
probabiitty
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0
20
40
60
80
20
80
Epsilon
Epsiton
Figure S5: Softmax probabilities of test sample with increasingly strong perturbation (y-zoom; same sample series as Fig. and Fig. 4). Top: Predictions of L2-Dropout model start with very high confidence, corresponding to good calibration (Fig. 4 Middle). however, for strong perturbations (epsilon greater than 40) false predictions are made with a very high confidence, reflecting the typical overconfident behaviour of the 1.2-Dropout model when moving away from in-domain samples. Bottom: EDL (left) makes noticeably under-confident predictions for small domain shifts before the entropy increases and confidence scores match accuracy. While FALCON (right) also makes slightly under-confident predictions for in-domain samples, the corresponding confidence scores still substantially closer to Like EDL, FALCON does not make over-confident predictions when moving further away from the training domain (epsilon greater 40).
A.7 ADDITIONAL FIGURES AND TABLES
0.8
0.6
Accuracy
0.4
0.2
0.0
20
40
60
80
Epsilon
Figure S6: Test accuracy for the 1.2- 1.2-Dropout model trained the 20 Newsgroups data. Accuracy declines gradually with increasing fraction of swapped words until it reaches random levels.
14

---

Under review as a conference paper at ICLR 2020
2.5
23-
MGGregee
Beep ouseri
FALCOM
0.0
sloar
(a) MNIST data
(b) CIFAR data
HSTM-MMST
McDmmput Dupp
(c) Sequential MNIST (LSTM)
Figure S7: Entropy for OOD predictions based on epsilon 90 on the y-zoom perturbation; with entropy approaching random levels, well calibrated predictions correspond to an entropy close to maximal entropy. The good calibration of FALCON and, for the MNIST data also EDL., is reflected by entropy levels that are substantially higher than for the other baseline methods.
1.0
Confidence scores
0.8
0.6
0.4
L2
L2-TS
MC-Dropout MNF
DeepEns
5VI
EDL
FALCON
Figure S8: Distribution of confidence scores for OOD predictions based on epsilon 90 on the y-zoom perturbation (across all predictions); predictions by baseline models are often severcly overconfident.
Table S2:
Test accuracy and average ECE (lower is better) across all perturbation strategies for LSTM and GRU models.
LSTM
GRU
Test acc.
Mean ECE
Test acc.
Mean ECE
L2-
0.986
0.327
0.991
0.334
MC-Dropout
0.986
0.334
0.98
0.296
Ensemmne
0.99
0.222
0,99
0.168
FALCON
0.978
0.118
0.988
0.108
15

---

Under review as a conference paper at ICLR 2020
Table S3: Accuracy of EDL and the L2- L2-Dropout model for downsampled images. For longer sequences EDL does not achieve competitive predictive power.
LSTM
GRU
Img size
L2-Drp
EDL
L2-Drp
EDL
6x6
0,968
0.8203
0.964
0.9678
10x10
0,982
0,8484
0.987
0,9845
14x14
0.990
0.8223
0.989
0.9865
16x16
0.988
0.7775
0.990
0.9904
20x20
0.986
0.5513
0.991
0.9905
24x24
0.986
0.3688
0.989
0.9323
28x28
0.986
0.3907
0.991
0.8384
Table S4: For each perturbation we varied the perturbation-specife parameter such that it ranged from no perturbation to a maximum perturbation corresponding to an accuracy close random. For rotation, perturbation is the (left right) rotation angle in degrees, shift is measured in pixels in y direction, for shear the perturbation is mcasured as shear angle counter-clockwise direction in degrees, for zoom the perturbation is zoom in direction. Word swap is quantified relative number of swapped words. Only FGSM used during training and measured as the relative amount of noise
Perurbation
Perturbation-speecfic parameter
FGSM rot left
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0
350
340
330
320
310
300
290
280
270
right
10
20
30
50
60
70
90
Shear
0
10
20
30
40
50
60
70
80
90
xyshift
10
14
18
xshift
0
2
4
6
8
10
12
14
16
18
xyshift
2
6
10
12
14
16
18
Xyzoom
1
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
xzoom
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
yzoom
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
word swap
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
Table S5: Test accuracy for the L2-dropout LeNet model trained on MNIST. Accuracy is listed for no perturbation (epsilon = 0) and maximum perturbation (epsilon = 90) on the test set. For all perturbations accuracy declines to almost random levels.
Test Accuracy
Perturbation
No perturbation
Max. perturbation
rot left
0.991
0.19
rot right
0.991
0.184
shear
0.991
0.132
xshift
0.991
0,097
xyshift
0.991
0.095
xyzoom
0.991
0.087
xzoom
0.991
0.188
yshift
0.991
0.14
yzoom
0.991
0.242
16

---

Under review as a conference paper at ICLR 2020
Table S6: Micro-averaged ECE for LeNet model trained on MNIST
rot left
rot right
shear
xyshift
xshift
yshift
Xyzoom
Xzoom
yzoom
L.2-Drooout
0.231
0.301
0.214
0.27
0.391
0.215
0.127
0.281
0.158
MC-Dropout
0.185
0.218
0.183
0.211
0.294
0.12
0.049
0.232
0.126
MNF
0.218
0.256
0.182
0.235
0.278
0.15
0.08
0.228
0.147
Deep-Ensemble
0.261
0.273
0.208
0.253
0.433
0.178
0.12
0.271
0,183
EDL
0,108
0.094
0.121
0.084
0.133
0.075
0.121
0.087
0,095
FALCON
0.074
0.065
0.088
0.106
0.033
0.068
0.117
0.113
0.08
Table S7: Micro-averaged ECE for VGG19 model trained on CIFARIO
rot left
rot right
shear
xyshift
xshift
yshift
xyzoom
Xzoom
yzoom
Dropout
0.551
0.563
0.576
0.582
0.507
0.496
0.669
0.546
0,639
MC-Dropout
0.339
0.343
0.423
0.374
0.307
0.302
0.436
0.364
0.502
Deep-Ensembles
0.321
0.326
0.332
0.325
0,293
0.333
0.373
0.314
0.392
EDL
0.144
0.15
0.262
0.132
0.13
0.164
0.32
0.206
0.267
FALCON
0.132
0.126
0.23
0.098
0.087
0.107
0.148
0.069
0.316
Table Micro-averaged ECE for LSTM model trained on sequential MNIST
rot left
rot right
shear
xyshift
xshift
yshift
xyzoom
XZOom
yzoom
L2-Dropouu Dropout
0.302
0.411
0.346
0.348
0.366
0.226
0.353
0.352
0.242
MC-Dropout
0.281
0.399
0.394
0.282
0.24
0.235
0.454
0.456
0.264
Deep-Ensembles
0.221
0.363
0.24
0.194
0.145
0.206
0.24
0.221
0.172
FALCON
0.092
0.174
0.15
0.036
0.069
0.106
0.221
0.121
0.09
Table S9: Micro-averaged ECE for GRU model trained on sequential MNIST
rot left
rot right
shear
xyshift
xshift
yshift
xyzoom
XZOOm
yzoom
1.2- 1.2-Dropout 0.301
0.435
0.354
0.388
0.332
0.259
0.345
0.327
0.266
MC-Dropout
0.289
0,414
0.379
0.319
0.255
0.253
0.287
0.188
0.279
Deep-Ensembles
0.191
0.301
0.214
0.125
0.049
0.176
0.176
0.169
0.109
FALCON (ours)
0.09
0.165
0.14
0.099
0.132
0.13
0.049
0.081
0.083
Table S10: Micro-averaged ECE for LSTM model trained on 20 Newsgroups data
Character swap
character swap
L2-Dropout
0.449
MC-Dropout
0.375
Deep-Ensebbles
0.218
FALCON (ours)
0.158
17


Reviews:
[{"review_id": "r1leOMl3tH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed FALCON, a simple method to produce well-calibrated uncertainty estimation. The idea is to introduce two additional terms, one that directly encourage lower confidence for all negative classes of all data points, and another one that optimizes the ECE for adversarial samples. Experiments show that FALCON outperforms several state-of-the-art methods for calibrating neural network predictions.\n\nAlthough the first term, L_S, affects only negative predictions, it is still somewhat strange to uniformly operate on all data points. It would help to perform an ablation study to see how L_S affect the results.\n\nSome description in Section 2 could use more mathematical rigor (e.g., when describing L_{adv}).\n\nThe authors use EDL as a baseline, I was wondering why not use the more commonly used, and possibly more effective, temperature scaling (TS) method from Guo et al. 2017. Note that the EDL paper does not seem to explicitly compare EDL and TS. \n\nFigure 4 (middle and right) is confusing. The line style is not consistent in the figures.\n\nThere are a few places where the text is rather vague and confusing. For example, what do you mean by ‘non-misleading evidence’ when describing L_{adv}? It would also be better to provide more insight more L_S to help the readers out. For example, it would help to state that L_S operate only on negative predictions.\n\nMost baselines are rather simple non-probabilistic (non-Bayesian) methods. Besides, MNF, It would also be interesting to see how FALCON compare other probabilistic NN method such as natural parameter networks, where they also explicitly evaluated uncertainty estimation.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1xalm-RKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new loss function for training deep neural networks, which show good performance with respect to well-calibrated, trustworthy probabilities for samples after a domain shift. The authors conduct experiments with multiple datasets and multiple forms of perturbations, where the proposed method achieve superior performance. \n\nI have the following major concerns with the paper:\n\n1. Presentation: The paper's presentation is very weak. It contains repetitive, long, convoluted statements and paragraphs all throughout making it difficult for the reader to understand anything. The first time I read this paper, I couldn't process what is happening. The introduction is more like realted work with less focus on what they are trying to pitch in the paper. However, the latter is the less severe concern.\n\n2. While I appreciate the use of reliability diagrams and the loss function inspired from it, I do not completely understand what are the entities in equations 1 and 2 or how they are used later. Please give them a name, and elaborate on them.\n\n3. While I was reading the paragraph before section 2.2 the first time, it seems to me that ECE is defined for those 10 pertubations specifically. I believe this is not the case; hence, the authors should make it more general and divert the specific details to experiments. \n\n4. I am not sure what the authors mean by, \"... while the loss surface remains largely unchanged\", in paragraph above section 2.2.2. I believe the authors have constructed a new loss function by adding a new term to it. That makes it difficult to understand the advantage the authors are talking about after this statement. Overall, I am not really convinced how in contrast to Bayesian Deep Learning, their approach can be EASILY applied to LSTM's and GRU's.\n\n5. What is L_{adv}? There is no equation number, no discussion around where this is defined. Again a presentation issue.\n\nMinor: Please clarify the reference for supplementary materials. For example, it is not clear that Table S1 is in supplementary material.\n\nOverall, this paper is good and has an interesting idea. The experiments are also extensive useful. However, I have reservations regarding the presentation of this paper at this moment. \n\n--- After Rebuttal  ---\n\nI thank the authors for providing a detailed response to my questions and editing the paper. I am now more positive about the paper; however, I still feel the presentation of the paper could be further improved. At this point, I will not move to acceptance range and keep the same score. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BklNA9L6tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method for calibrating neural networks on in- and out-of-distribution data using two additional loss terms: entropy-encouraging loss term which maximizes softmax probabilities for wrong classes and adversarial calibration loss term which pushes confidences to match accuracy on adversarial examples. The idea of using adversarial calibration training is interesting and promising, however, the clarity of the paper needs significant improvement and there are several issues which need to be addressed; for this reason, I recommend a weak reject for the current version.\n\n1. One of the contributions listed in the paper is that authors “illustrate the limitations of entropy as measure for trustworthy predictions and introduce a new metric to quantify technical trustworthiness based on the concept of calibration”; the section 2.1 discusses this in detail. The proposed metric is expected calibration error (ECE) averaged over different levels of noise perturbations applied to data during test time. This metric assumes that for a given dataset we know in advance what kind of noise perturbations can be implied to cause out-of-domain/domain-shift scenarios during test time. This assumption most likely doesn’t hold in real-world applications since domain shifts may be of various kinds. Moreover, calibration may not make sense at all for out-of-domain data if test inputs don’t belong to any of the train classes (as opposed to entropy of predictive distribution which still can be computed and is expected to be higher for such inputs). The metric is also dependent on the considered noise level range; the plot of ECE vs noise level can be illustrative and informative while the averaged value of ECE over all noise levels can be misleading (at least, we may want to have some discount factor to account for the fact that with noise level 0 we strongly care about calibration, while for very high noise levels calibration doesn’t give us much information since the useful patterns in data may be corrupted and it may be impossible even for humans to classify such objects, e.g. on Figure 1-top-left for noise level 100, the digit 6 is corrupted so much that we wouldn’t care about calibration for such inputs but just want to maximize the entropy).\nThe paper also claims “Recent efforts in terms of evaluating predictive uncertainty have focused on entropy as measure for uncertainty-awareness for predictions under domain shift.” In previous work addressing uncertainty in domain shift [1], not only entropy but other various metrics are considered for out-of-domain detection (Brier score, thresholded confidences, etc), these metrics don’t depend on particular perturbations and are very informative.\nThe introduced metric, ECE for different noise labels, makes sense only in the toy scenarios where we can control the noise level, however, it would still be better to either look at the plots or to use some discounted averaged ECE over different noise levels, but not equal average. So the significance of this contribution (introduction of new metric) is limited.  \n\n2. The clarity of the paper could be significantly improved.\n\n(a) Figure 1: the top left plot suggests that for high noise level “wrong predictions are often made with high confidence” (so the entropy of this distribution is low) while the top right plot shows that on average predictive entropy gradually grows and is high for high noise levels, so the top right plot is probably not a representative example and it might be misleading to claim this is an “often” case.\n\n(b) Could you please clarify what you mean by “after removing non-misleading evidence” in section 2.2.1? In the next sentence, the remaining probability is probably distribution uniformly across C-1, not C, classes. The predictive entropy loss term essentially maximizes the probabilities of wrong classes with a lower coefficient. How would you support the claim that the loss surface is unchanged?\nThe loss term is “parameter-free” but still we need to tune a coefficient lambda_S for it.\n\n(c) The adversarial loss equation in section 2.2.2 is written as L2-norm of a scalar value, why is L2-norm needed? The acc(B_m) is not differentiable so it is probably just considered as constant in the paper? Please comment on that. \n\n(d) Section 3.2 and Figure 4. On Figure 4 Middle and Right plots have different colors than those listed on legend, please, fix this. On the middle plot, both FALCON and EDL have high ECE for noise levels <50 which indicates that they are both highly underconfident for in-distribution data and low noise levels, while other methods have close to 0 ECE on low noise levels. The authors only comment on EDL underconfidence: “it is worth noting that EDL has a substantially higher ECE for in-domain predictions, reflecting under-confident predictions on the test set”, and not on FALCON underconfidence. However, using the proposed score, ECE averaged over all noise levels (Table 2), it may look like the method is doing a good job, while underconfidence problem is revealed when looking at Figure 4. This is also an illustration of the concern about the proposed metric I raised in point 1.\n\n3.  It would help to have an ablation study in the main text of the paper showing the significance of each loss term: in the appendix Figure S1, it is shown that adding adversarial loss to standard and entropy loss helps. Is the entropy loss needed at all? How does performance change if we only have standard and adversarial loss? How sensitive is performance to the choice of lambda_avd and lambda_s?\n\n\n[1] Ovadia, Yaniv, et al. \"Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.\" arXiv preprint arXiv:1906.02530 (2019).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an algorithm to produce well-calibrated uncertainty estimates. The work accomplishes this by introducing two loss terms: entropy-encouraging loss and an adversarial calibration loss to encourage predictive smoothness in response to adversarial input perturbations. 

All reviewers recommended weak reject for this work with a major issue being the presentation of the work. Each reviewer provided specific examples of areas in which the paper text, figures, equations etc were unclear or missing details. Though the authors have put significant effort into responding to the specific reviewer mentions, the reviewers have determined that the manuscript would benefit from further revision for clarity. 

Therefore, we do not recommend acceptance of this work at this time and instead encourage the authors to further iterate on the manuscript and consider resubmission to a future venue. 

Decision: Reject