Title: REDUCING OVER-CONFIDENT ERRORS OUTSIDE THE KNOWN DISTRIBUTION Anonymous

Abstract: Intuitively, unfamiliarity should lead to lack of confidence. In reality, current algorithms often make highly confident yet wrong predictions when faced with unexpected test samples from an unknown distribution different from training. Unlike domain adaptation methods, we cannot gather an "unexpected dataset" prior to test, and unlike novelty detection methods, a best-effort original task prediction is still expected. We compare a number of methods from related fields such as calibration and epistemic uncertainty modeling, as well as two proposed methods that reduce overconfident errors of samples from an unknown novel distribution without drastically increasing evaluation time: (1) G-distillation, training an ensemble of classifiers and then distill into a single model using both labeled and unlabeled examples, or (2) NCR, reducing prediction confidence based on its novelty detection score. Experimentally, we investigate the overconfidence problem and evaluate our solution by creating "familiar" and "novel" test splits, where "familiar" are identically distributed with training and "novel" are not. We discover that calibrating using temperature scaling on familiar data is the best single-model method for improving novel confidence, followed by our proposed methods. In addition, some methods' NLL performance are roughly equivalent to a regularly trained model with certain degree of smoothing. Calibrating can also reduce confident errors, for example, in gender recognition by 95% on demographic groups different from the training data.
Under review as a conference paper at ICLR 2019 Novel female, model: 99.3% male (ours: 53.7%)

1. INTRODUCTION:

In machine learning and computer vision, the i.i.d. assumption, that training and test sets are sampled from the same distribution (henceforth "familiar" distribution), is so prevalent as to be left unwritten. In experiments, it is easy to satisfy the i.i.d. condition by randomly sampling training and test data from a single pool, such as photos of employees' faces. But in real-life applications, test samples are often sampled differently (e.g., faces of internet users) and may not be well-represented, if at all, by the training samples.

Prior work (Lakshminarayanan et al., 2017) has shown networks to be unreliable when tested on semantically unrelated input (e.g. feeding CIFAR into MNIST-trained networks), but users would not expect useful predictions on these data. However, we find this issue extends to semantically related input as well, such as gender classifiers applied to faces of older or younger people than those seen during training, which is a more common occurrence in practice and more problematic from a user's perspective. We demonstrate that, counter to the intuition that unfamiliarity should lead to lack of confidence, current algorithms (deep networks) are more likely to make highly confident wrong predictions when faced with such "novel" samples, both for real-world image datasets (Figure 1 ; see caption) and for toy datasets (Figure 2 ; see subcaption 2(a) and Appendix A). The reason is simple: the classification function, such as a deep network, is undefined or loosely regulated for areas of the feature space that are unobserved in training, so the learner may extrapolate wildly without penalty.

Confident errors on novel samples can be catastrophic. Whether one would ride in a self-driving car with a 99.9% accurate vision system, probably depends on how well-behaved the car is on the 0.1% mistakes. When a trained model labeled a person as a gorilla (Zhang, 2015) , the public trust in that system was reduced. When a driving vision system confidently mistook a tractor trailer (Yadron & Tynan, 2016) , a person died. Scholars that study the impact of AI on society consider differently distributed samples to be a major risk (Varshney & Alemzadeh, 2017) : "This is one form of epistemic uncertainty that is quite relevant to safety because training on a dataset from a different distribution can cause much harm." Our trust in a system requires its ability to avoid confident errors.

Unfortunately, these novel samples may differ from training in both expected and unexpected ways. This means gathering a set of "unexpected" training samples, though required by covariate shift and domain adaptation methods, becomes unviable (Varshney & Alemzadeh, 2017) . One may use novelty detection methods to identify novel samples at test time (to report an error or seek human guidance), but this may be insufficient. These "outliers" (e.g. underrepresented faces that users upload) are perfectly normal samples in the eyes of a user and they would reasonably expect a prediction. Also, human guidance may not be fast enough or affordable.

In this paper, our aim is to reduce confident errors for predictions on samples from a different (often non-overlapping) but unknown distribution (henceforth "novel" distribution). In contrast with most recent work, we focus on the confidence of the prediction rather than only the most likely prediction (accuracy) or the confidence ordering (average precision or ROC). In addition to reviewing the effectiveness of established methods, we propose and evaluate two ideas that are straightforward extensions to ensemble and rejection methods. One is that multiple learners, with different initializations or subsamples of training data, may make different predictions on novel data (see Lakshminarayanan et al. (2017) ). Hence, ensembles of classifiers tend to be better behaved. But ensembles are slow to evaluate. If we distill (Hinton et al., 2015) an ensemble into a single model using the training data, the distilled classifier will have the original problem of being undefined for novel areas of the feature space. Fortunately, it is often possible to acquire many unlabeled examples, such as faces from a celebrity dataset. By distilling the ensemble on both the training set and on unsupervised examples, we can produce a single model that outperforms, in terms of prediction confidence, single and standard distilled models on both identically and differently distributed samples.

Another idea is that if the training set does not provide enough information for the unseen data, therefore it may be desired to simply avoid confident predictions. We can lower their confidence according to the output of an off-the-shelf novelty detector. This means reducing confident errors by reducing confidence, regardless of correctness. It may improve novel sample prediction quality, but in turn degrade performance on familiar samples. Although this idea sounds like a natural extension to novelty detection, we are unaware of any implementation or analysis in the literature.

Experimentally, we investigate the confidence problem and perform an extensive study by creating "familiar" and "novel" test splits, where "familiar" are identically distributed with training and "novel" are not. For example, in cat vs. dog classification, the novel examples are from breeds not seen during training, or in gender classification, the novel examples are people that are older or younger than those seen during training. Our evaluation focuses on negative log likelihood (NLL) and the fraction of highly confident predictions that are misclassified ("E99"). They measure both prediction accuracy and how often confident errors occur. To summarize, our contributions are:

• Draw attention to a counter-intuitive yet important problem of highly confident wrong predictions when samples are drawn from a unknown distribution that is different than training. • Evaluate and compare the effectiveness of methods in related fields, including two proposed methods to reduce such overconfident errors. • Propose an experimental methodology to study the problem by explicitly creating familiar and novel test splits and measuring performance with NLL and E99.

2. RELATED WORK:

To the best of our knowledge, there is no prior work with exactly the same goal and focus as this paper, but we are very similar to several lines of work. To avoid confusion, we compile Table 1 to show our differences from these prior work for your reading convenience.

Epistemic uncertainty. The most related works model and reduce epistemic uncertainty (model uncertainty from incomplete knowledge) by estimating and evaluating a set or distribution of models that agree with the training data. The intuition is that on familiar test data, performance is boosted Bayesian approaches (Bishop, 1995; Blundell et al., 2015; Hernández-Lobato et al., 2016) estimate a distribution of network parameters and produce a Bayesian estimate for likelihood. These methods are usually very computationally intensive (Lakshminarayanan et al., 2017) , limiting their practical application. Gal & Ghahramani (2016) propose MC-dropout as a discrete approximation of Bayesian networks, adopting the dropout technique for a Monte-Carlo sample of likelihood estimates. They further propose to jointly estimate aleatoric and epistemic uncertainties (Kendall & Gal, 2017) to increase the performance and quality of uncertainty. Lakshminarayanan et al. (2017) propose to instead use an ensemble to emulate a Bayesian network and achieve better performance.

These works reduce uncertainty and improve generalization under i.i.d. assumptions. Our work differs in its focus of improving confidence estimates for novel samples (differently distributed from training), and our method is much more efficient at test time than MC-dropout or ensembles. Hafner et al. (2018) reduces the confidence towards a prior on unseen data, which is similar to our proposed NCR method. We note that they perform regression on a low-dimensional feature space generalizing into future data, which is a very different problem from our image classification generalizing into unexpected samples. They use perturbed original dataset features and the prior as the labels for reducing confidence. This can be hard to generalize to an 224x224x3 image space.

Domain adaptation (Quionero-Candela et al., 2009) aims at training on a source domain and improving performance on a slightly different target domain, either through unsupervised data or a small amount of supervised data in the target domain. In our settings, it is unviable to sample a representative "unexpected dataset" prior to test time, which we consider unknowable in advance.

Consequently, the unsupervised samples we use are not from the familiar or novel distribution.

Domain generalization (Muandet et al., 2013; Li et al., 2017; Shankar et al., 2018) is more closely related to our work, aiming to build models that generalize well on a previously unspecified domain, whose distribution can be different from all training domains. Unlike in domain adaptation, the target domain in this case is unknown prior to testing. These models generally build a domaininvariant feature space (Muandet et al., 2013) or a domain-invariant model (Shankar et al., 2018) , or factor models into domain-invariant and domain-specific parts (Li et al., 2017) .

Also related are attribute-based approaches, such as Farhadi et al. (2009) , who build an attribute classifier that generalizes into novel object categories, similar to our experiment set-up. They select features that are discriminative for an attribute within individual classes to build invariance across object classes. These models all require multiple training domains to learn invariant representations, with the intent to improve robustness to variations in the target domain. In contrast, our method is concerned only with novel data and does not require multiple training domains.

Some methods refuse to predict by estimating whether the system is likely to fail and outputting a signal requesting external intervention when necessary. Meta-recognition (Scheirer et al., 2011) estimates the performance based on the model output. Rejection options (Fumera & Roli, 2002; Geifman & El-Yaniv, 2017) or failure detection (Zhang et al., 2014; Wang & Vasconcelos, 2018) estimate the risk of misclassification to determine whether to refuse prediction, usually by looking at how close samples are to decision boundaries. Unlike our work, these works are not concerned with the quality of confidence estimates and do not analyze their rejection ability for samples from novel distributions.

Outlier detection (Devarakota et al., 2007; Lee & Grauman, 2012; Liang et al., 2018) , novelty detection, and one-class classification (Tax, 2001; Khan & Madden, 2009) determine whether a test sample comes from the same distribution as the training data. The downside is that these methods would provide no confidence estimate for rejected samples, even though the models can still provide informative estimates (e.g. in Section 5 gender classification, baseline accuracy on novel data is 85% despite bad NLL performance). One of our methods naturally extends these methods by using their output to improve confidence estimates.

Generalization. Various techniques have been proposed in deep learning literature to minimize the generalization gap, and popular ones include data augmentation, dropout (Srivastava et al., 2014) , batch normalization (Ioffe & Szegedy, 2015) , and weight decay. Hoffer et al. (2017) propose better hyperparameter selection strategies for better generalization. Bagging (Breiman, 1996) and other model averaging techniques are also used prior to deep learning. These methods focus on reducing generalization gap between training and validation. They do not address issues with unexpected novel samples and can be used in conjunction with our method.

Theoretical analyses for off-training-set error (Schaffer, 1994; Roos et al., 2006) and empirical analysis of generalization for test samples (Novak et al., 2018) are also available, but these methods measure, rather than reduce, generalization errors.

Calibration methods (e.g. Guo et al. (2017) ) aim to improve confidence estimates, but since the confidence estimates are learned from familiar samples (i.i.d. with training), risk is not reduced for novel samples. However, we experimentally find that (Guo et al., 2017) performs surprisingly well on unseen novel data, which the method is not optimized for.

Distilling (Hinton et al., 2015) can be used on unsupervised samples. 

3. PROPOSED METHODS:

The goal of this paper is to improve confidence estimate of deep models on unseen data. We focus on a classification setting, although our framework could be naturally extended to tasks similar to classification, such as VQA and semantic segmentation. We assume the probability of label given data P (y|x) is the same where familiar and novel distributions overlap, but unlike covariate shift, we assume no knowledge of the novel distribution other than what is already in the familiar distribution. Distillation of an ensemble. We base our first method on the original distillation from an ensemble (Hinton et al., 2015) , which we briefly summarize. First, train an ensemble f Ens (•) on F tr , which 

F = P (y = c|x F ; f Ens )

is the probability estimate for class c given by the ensemble. Finally, train a new single-model network f θ by taking its probability prediction ŷF and applying the distillation loss L dis ( y F , ŷF ). This is simply a cross-entropy loss between the distribution estimates, with a temperature T applied to the logits (see Hinton et al. (2015) ) to put more focus on relative probability mass.

To further improve distillation performance, a classification loss L cls (cross-entropy is used) over F is added as an auxiliary. The final loss becomes:

EQUATION

As can be seen, distillation is still a method focused on the familiar distribution F, and we have shown that a distilled network is not necessarily well-behaved on N .

Method 1: G-distillation of an ensemble with extra unsupervised data. To improve distillation on novel data, a natural idea would be having the distilled model mimic the ensemble on some kind of novel data. Denote by x G ∼ G an unlabeled general data distribution which ideally encompasses familiar F and any specific novel N . Here "encompass" means that data from F and N can appear in G: ∀x, P G (x) 0 if P F (x) 0 or P N (x) 0. We draw a training set G tr from such general distribution G.

A distribution that has all related images is nearly impossible to sample in practice. Hence, we cannot rely on G tr ∼ G to encompass all possible novel N . We need to pick a dataset that is sufficiently diverse, sufficiently complex, and sufficiently relevant to the task, so our method can use it to generalize. Often, such a set can be obtained through online searches or mining examples from datasets with different annotations. For example, for making a gender classifier robust to additional age groups, we could sample G tr from the CelebA (Liu et al., 2015) dataset (note that CelebA does not have age information and we do not need it; for the sake of the experiment we also discard gender information from CelebA). For making a cat-dog classifier robust to novel breeds, we could sample G tr from ImageNet (Russakovsky et al., 2015) . Note these G tr 's do not encompass N , but only provide a guideline for generalization.

Similar to distillation, we train an ensemble f Ens and obtain the soft labels y F . In addition, we also obtain soft labels y G by evaluating f Ens on x G ∈ G tr . Then we add the samples G tr into the training set, and train using the combined data:

L = - 1 |F tr ∪ G tr | Gtr λ dis L dis ( y G , ŷG ) + Ftr (λ dis L dis ( y F , ŷF ) + λ cls L cls (y F , ŷF )) (2)

Note that G is unsupervised, so we cannot apply L cls on its samples. For test time, we simply evaluate the probability estimation y F against the ground truth in F ts , and y N against those in N ts , respectively. Table 2 demonstrates our training and evaluation procedure.

Method 2: Novelty Confidence Reduction (NCR) with a novelty detector. This method is more straightforward. We make use of a recent off-the-shelf, model-free novelty detector ODIN (Liang et al., 2018) . We run both the single model and the ODIN procedure on the input x to get the probability estimate ŷ and the detection score s 0 (x) (where a smaller s 0 means a more novel input). However, s 0 may not be calibrated and may lie in an arbitrary interval specific to the detector, e.g. [0.5, 0.51]. We normalize it to [0, 1] with a piecewise-linear function s(x) = max 0, min s 95% -s0(x) s 95% -s 5% , 1 where s •% are the •% percentiles of the detection scores for all training samples. s(x) closer to 1 means the confidence should be reduced more.

At test time, we linearly interpolate between ŷ and the prior y 0 :

ŷNCR = (1 -λ s s(x)) ŷ + λ s s(x)y 0 (3)

where y We refer readers to Appendix B for implementation details.

4. EXPERIMENTAL SETUP:

Datasets. To demonstrate effectiveness in appropriately handling unexpected novel data, and reduce confident failures thereof, we perform extensive analysis on four classification datasets mimicking different scenarios. We set up the F, N , and G distributions, and make sure that F and N are completely non-overlapping, unless otherwise noted. Table 3 illustrates our datasets:

• Gender recognition, mimicking a dataset with obvious selective bias.

• ImageNet animal superclass (mammals, birds, herptiles, and fishes) recognition, mimicking an animal classifier being tested on unseen species within these superclasses. (*) We determine the original classes belonging to each superclass, sort the classes by their indices, and use the first half of the class list as familiar and the second half novel. This makes sure F and N do not overlap by definition.

• Cat-dog recognition. Similar in spirit as the above. (*) The first 20 dog breeds and 9 cat breeds are deemed familiar, and the rest are novel. • VOC-COCO recognition (determining presence of object categories). Mimics a model trained on a lab dataset being applied on more complex real world images with unexpected input. tvmonitor is mapped to tv and not laptop.

Note that VOC-COCO is quite different from the others where F and N do not overlap, because VOC and COCO images can be very similar.

Pre-training. When designing our experiments, we should make sure N is still novel even considering information gained from the pre-training dataset. Although using ImageNet (Russakovsky et al., 2015) pre-trained models may improve performance, we note that the dataset almost always contains classes that appear in our "unseen" novel N choices; therefore pre-training on ImageNet would render the novel N not actually novel. Instead, we opt to use Places365-standard (Zhou et al., 2017) as the pre-training dataset. We argue that our method would generalize to ImageNet pre-trained models when N classes are disjoint from ImageNet.

Validation splits. Since we need to report negative log likelihood on both F ts and N ts , datasets with undisclosed test set cannot be directly used. For ImageNet animals, VOC, and COCO, we report performance on the validation set, while splitting the training set for hyperparameter tuning.

For LFW+ and Oxford-IIIT Pets, we split the training set for validation and report on the test set.

For LFW+, we use the first three folds as training, and the last two folds as test since the dataset is smaller after the familiar-novel split. For the general G tr ∼ G dataset we proportionally sample from the corresponding training set, or a training split of it during hyperparameter tuning.

Compared methods. We compare to normal single models ("single"), and standard distillation using F tr images ("distilling") as baselines. We compare to using an ensemble ("ensemble"), which is less efficient in test time. In addition, we compare to Kendall & Gal (2017) where aleatoric and epistemic uncertainties are modeled to increase performance ("uncertainty"). Since ResNet18 does not use dropout, for this method we insert a dropout of p = 0.2 after the penultimate layer. For a fairer comparison, we also include an experiment using DenseNet161 as the base network, which is designed to use dropout. We also compare to (Guo et al., 2017 ) ("T -scaling") where only familiar samples are considered to calibrate the confidences, as well as an ensemble with members with the same calibration ("calib. ens.").

In our experiments, we discover that some methods outperform on novel samples and underperform on familiar, and one may suspect that the trade off is simply an effect of smoothing the probability regardless of the sample being novel or familiar, or the prediction being correct or wrong. To analyze this, we also report the performance of further raising or decreasing the temperature of the prediction logits of each method by τ :

ŷ(c) = (y (c) ) 1/τ i (y (i) ) 1/τ (4)

where ŷ(c) is the original estimate for class c. We use τ ∈ [0.5, 5], and plot the trace for how the familiar and novel NLL change. Note that this is equivalent to using (Guo et al., 2017) with a number of different temperature choices, but the trace can be optimistic (and seen as an oracle), since in practice the choice of τ has to be decided before evaluating on test.

Performance measure. Note that we want to measure how well the model can generalize, and how unlikely the model is to produce confident errors. As discussed in Section 1, we mainly use the negative log likelihood (NLL), a long-standing performance metric (Nasrabadi, 2007; Friedman et al., 2001) , as a measure of the quality of our prediction confidence. If a prediction is correct, we prefer it to be confident, leading to a lower NLL. And for incorrect predictions, we prefer it to be unconfident, which means the likelihood of the correct class should in turn be higher, which also leads to a lower NLL. The metric gives more penalty towards confident errors, suiting our needs.

In summary, the model needs to learn from only familiar labeled data (plus unsupervised data) and produce the correct confidence (or lack of confidence) for novel data to improve the NLL.

During evaluation, we clip the softmax probability estimates given by all models to [0.001, 0.999] in order to prevent a small number of extremely confident wrong predictions from dominating the NLL. We clip these estimates also considering overconfidence beyond this level is risky with little benefit in general. The processed confidences still sum up to 1 for binary classifiers, but may be off by a small amount for multiclass scenarios.

We also report a confident error rate ("E99") where we show the error rate for samples with probability estimate of any class larger than 0.99. Ideally this value should be within [0, 0.01]. Further, we also report accuracy and mean Average Precision (mAP) and test the hypothesis that we perform comparably on these metrics. Due to the relatively high variance of NLL on N ts , we run our experiments 10 times to ensure statistical significance with a p-test, but we run the ensemble method only once (variance estimated using ensemble member performance variance). Our 10 runs of the distillation methods use the same ensemble run.

5. RESULTS:

We first compare G-distill to the baselines on the four scenarios. Tables in Figures 3 and 4 show our results, and the graphs show the trade-off between familiar and novel NLL by smoothing. Note (c) Cat-dog recognition with Pets Figure 3 : Our performance on familiar F and novel data N , for tasks with no overlap between the two. Tables: in terms of novel NLL, calibrated methods perform the best, while our methods outperform the rest of the single-model methods. Methods perform similarly on familiar NLL, while ours and calibration are more reliable with high-confidence estimates (E99), indicating a better generalization. We perform on the same level on accuracy or mAP as other methods except for Gdistill in (c). Graphs: Trade off familiar and novel NLL by smoothing the probability estimates with a higher or lower temperature. Crosses indicate performance without smoothing (τ = 1). Bottomleft is better. Our methods do not outperform single models considering smoothing trade-off. : for NLL, we outperform other single-model based methods on novel data but on familiar. Calibration methods do not make much difference. Graph: considering trade-off between familiar and novel NLL, G-distillation performs similarly to distillation, while NCR underperforms. See Figure 3 for details. Note that this figure is zoomed in more.

that among these experiments, those in Figure 3 have novel distribution N completely disjoint from familiar F, while Figure 4 does not.

Single models and the ensemble: single models perform much worse on novel NLL than familiar. Their familiar E99 is properly small, but novel E99 is far beyond 0.01. This confirms that confident errors in novel data is indeed a problem in our real image datasets. Ensembles not only improve performance on familiar data, but also are much better behaved on novel data.

Calibration: Except in Figure 4 , properly calibrated T -scaling with single models perform as well as, or better than, all methods using a single model at test time. An ensemble of calibrated members is nearly equivalent to a ensemble with further smoothed predictions, and it performs best in terms of novel NLL (at the expense of familiar NLL, and test efficiency). It is very surprising that even when calibrated using familiar data only, single models are able to outperform all methods except ensembles of calibrated members in Figure 3 . These indicate that proper calibration is a key factor when reducing confident errors even on novel data. In Figure 4 , calibration does not make much difference (single models are well-calibrated already).

Our proposed methods: G-distill and sometimes NCR perform reasonably well in familiar-novel balance (except on Figure 3(c) ). In Figure 4 (VOC-COCO), distilling and G-distill outperforms on novel NLL, but note that the difference is much smaller compared to other experiments.

With smoothing trade-off: Looking at predictions with different levels of smoothing, we can tell that uncertainty, distillation, G-distill, and NCR are all equivalent or inferior to a smoothed version of a single model. Considering smoothing with τ is equivalent to tuning T -scaling on test data, this means that (1) these methods may outperform single models in accuracy or mAP, and have bettercalibrated NLL than single models, but are ultimately inferior to better calibrated single models in confidence quality; and (2) even if these methods are calibrated, their confidence may not outperform a calibrated single model trained regularly. We note that in practice, one cannot choose the smoothing τ using a test novel dataset, so the smoothing results are only for analysis purposes.

Confident errors: in Figure 3 , our error rate among novel confident predictions is far below compared non-calibrated methods and much closer to familiar E99. For Figure 4 , we also have a slightly lower novel E99. These indicate a better calibration with confident predictions, especially with a disjoint N . However, the E99 for both NCR and the calibrated methods (T -scaling and ensemble with calibrated members) is usually far lower than 0.01, suggesting often under-confident predictions.

Other metrics: for accuracy or mAP, our methods (especially NCR) remain competitive compared to methods other than the ensemble, in Figures 3(a Novelty detector effectiveness: one interesting phenomenon is that the novelty detector is not very effective in distinguishing familiar and novel samples (AUC for each dataset: LFW+ 0.562, Im-ageNet 0.623, Pets 0.643, VOC-COCO 0.665), but are quite effective in separating out wrongly classified samples (AUC: LFW+ 0.905, ImageNet 0.828, Pets 0.939, VOC-COCO 0.541). We hypothesize that the novelty detector can fail to detect some data in our novel split that are too similar to the familiar. However, this does not affect the prediction performance much since the classifier model are less prone to confident failures on these samples.

Miscellaneous. In Appendix C, we evaluate the impact of some factors on our experiments, namely the choice of ensemble type, and the base network size. It is also possible to train an ensemble of Gdistillation models to boost the performance, at the sacrifice of test time performance. We find that it improves the foreign NLL beyond the original ensemble, while still underperforming calibration.

We also tried to combine G-distillation and NCR, by performing novelty detection on the G-distilled network and further reduce novelty confidence. However, the results show that only when both methods show advantage against baselines, the combined method can outperform both components. Otherwise the combined method may underperform baselines.

6. CONCLUSIONS:

In this paper, we draw attention to the importance of minimizing harm from confident errors in unexpected novel samples different from training. We propose an experiment methodology to explicitly study generalization issues with unseen novel data, and compare methods from several related fields. We propose two simple methods that use an ensemble and distillation to better regularize network behavior outside the training distribution, or reduce confidence on detected novel samples, and consequently reduce confident errors. For future work, it can be beneficial to investigate the ability to handle adversarial examples using this framework, and improve calibration with unexpected novel samples taken into account.

A TOY EXPERIMENT DETAILS:

To better visualize the problem of confident errors when using single neural networks, we demonstrate the generalization behavior of networks on 2-dimensional feature spaces. We construct toy datasets with input x ∈ R 2 and define labels y ∈ {0, 1}. For each dataset we split the feature space into familiar region F and novel region N , train models on F and evaluate on a densely-sampled meshgrid divided into F and N regions. See Figure 2 , column 1 for an illustration of the ground truth and familiar set of our "square" dataset. Intuitively, the "square" dataset seems to be easy, and the network may perform well on the novel data, while for harder datasets the network might either perform badly or produce a low-confidence estimation.

Figures 2 and 5 show the results. In the "square" dataset, the networks do not generalize to the novel corner, but rather draw an irregular curve to hallucinate something smoother. As a consequence, the networks become confidently wrong on a significant portion of the dataset. In multiple runs of the optimization, this behavior is consistent, but the exact regions affected are different. The ensemble gives a much smoother result and (appropriately) less confident estimates on the novel region, and the area of confident errors is largely reduced.

Figure 2 (b) further shows the negative log likelihood (NLL) performance of each method. While the performance for familiar samples is similar across methods, the NLL for novel samples improves dramatically when using an ensemble. This indicates that the ensemble has a more reliable confidence estimate on novel data.

We further test standard distillation (Hinton et al., 2015) and our method (see Section 3 for notations and definitions) against these baselines. We note that this experiment can only be seen as an upper bound of what our method could achieve, because when we sample our unsupervised set from the general distribution in this experiment, we use F and N combined for simplicity. In our later experiments on real images, the novel distribution is considered unknown.

Figure 2 (b) shows that standard distillation performs not much better than single methods, consistent with our hypothesis that distillation using the familiar samples still results in irregularities on novel data. On the other hand, our method performs much closer to the ensemble. This positive result encourages us to further test our method on datasets with real images.

Figure 5 illustrates the irregularities of single models with more toy experiments, and shows the performance of all methods involved. Although the toy datasets can be very simple ("xor" and "circle") or very complex ("checkerboard"), in all cases, the same observations in Figure 2 applies. The single models are more likely to have erratic behaviors outside the familiar region, the ensemble behaves more regularly, and our method can emulate the ensemble in terms of the robustness in novel regions.

B IMPLEMENTATION DETAILS:

For toy experiments. We take 1200 samples for both train and validation, while our test set is simply a densely-sampled meshgrid divided into familiar and novel test sets. We use a 3-hiddenlayer network, both layers with 1024 hidden units and Glorot initialization similar to popular deep networks, to avoid bad local minima when layer widths are too small Choromanska et al. (2015) . Batchnorm Ioffe & Szegedy (2015) and then dropout Srivastava et al. (2014) are applied after ReLU. We use the same hyperparameter tuning, initialization, and training procedures as described in Section 3 implementation details.

For image experiments. We use the following experimental settings, unless otherwise explained. We set λ dis = 2 3 and λ cls = 1 3 . The original paper used 1 and 0.5, but we scale them to better compare with non-distillation methods. Temperature T = 2. We sample G tr to be roughly 1 4 the size of F tr . For the network, we mostly use ResNet18 (He et al., 2016) , but we also perform an experiment on DenseNet161 (Huang et al., 2017) to check that our conclusions are not architecturedependent. To enable training on small datasets, we use networks pre-trained on very large datasets as a starting point for both ensemble members and the distilled network. We use a simple ensemble of 10 members. We also compare to a bagging (Breiman, 1996) ensemble scheme which resamples the dataset. This is more popular prior to deep networks, but we empirically find that bagging undermines the log likelihood on both familiar data and novel data.

:

Domain generalization requires several source domains, providing hints of the novel distribution (see text).

Figures and Tables:

Figure 1: Deep networks can often make highly confident mistakes when samples are drawn from outside the distribution observed during training. Shown are example images that have ages, breeds, or species that are not observed during training and are misclassified by a deep network model with high confidence. Using our methods (shown here is G-distillation, to distill an ensemble of networks on both training and unsupervised examples) makes fewer confident errors for novel examples, increasing the reliability of prediction confidence overall.

Demonstration of deep networks' generalization behavior with a 2-dimensional toy dataset "square". Column 1: we design a binary ground truth for classification on a 2-dimensional feature space. Column 2: for training, we only provide samples on the left and lower portions (the "familiar" distribution), and reserve the upper-right only for testing (the "novel" distribution). Column 3-7: we show negative log likelihood (NLL) predicted for each point in the feature space. A small NLL indicates correct prediction, while a very large NLL indicates a confident error. Column 3, 4: multiple runs of the network have similar performances on familiar regions but vary substantially in novel regions where the training data imposes little or no regulation, due to optimization randomness. Column 5: an ensemble of 10 such networks can smooth the predictions and reduce confident errors at the sacrifice of test efficiency. Column 6: distilling the ensemble using the training samples results in the same irregularities as single models. Column 7: one of our proposals is to distill the ensemble into a single model using both the labeled training data and unsupervised data from a "general" distribution. Feature space partitioning: "familiar" F, "novel" N , and "general" G distributions. (see Appendix A and section 3)

Figure 2: This toy experiment illustrates how classifiers (in this case deep networks) can be confidently wrong when attempting to generalize outside the extent of the training data. See Appendix A for details and extra toy datasets. Figure best read in color.

Denote by (x F , y F ) ∼ F the familiar data distribution, and F tr the training set, F ts the test set drawn from F. Further denote by (x N , y N ) ∼ N a novel data distribution, which satisfies P F (y|x) = P N (y|x) where the input distributions overlap. Denote by N ts the test set drawn from N . The inputs x F and x N may occupy different portions of the entire feature space, with little to no overlap. Later in this section, we introduce and describe an unsupervised distribution x G ∼ G with training set G tr .In our problem setting, the goal is to improve performance and quality of the estimation for P N (y|x). Only F tr (and unsupervised G tr ) are provided in training, while F ts and N ts are used in test time. No training sample from N is ever used, and N should not have been seen by the model, even during pre-training.

F (y = c) is the prior of class c on the familiar F tr , and λ s = 0.15 a hyperparameter. Efficiency comparison. An ensemble(Lakshminarayanan et al., 2017) requires M forward passes where M is the number of ensemble members. MC-dropout(Gal & Ghahramani, 2016) requires ∼ 50× forward passes for the Monte Carlo samples. G-distillation needs only one forward pass, and NCR only needs to evaluate an extra novelty detector (in particular, ODIN needs a forwardbackward pass). Therefore at test time they are much faster. However, G-distillation pays a price at training time by training both an ensemble and a distilled model.

Figure 4: Our performance on VOC-COCO recognition, where familiar F and novel data N have a strong overlap. Table:for NLL, we outperform other single-model based methods on novel data but on familiar. Calibration methods do not make much difference. Graph: considering trade-off between familiar and novel NLL, G-distillation performs similarly to distillation, while NCR underperforms. See Figure3for details. Note that this figure is zoomed in more.

) and 3(b). However, they only perform similarly to distillation in Figure3(c) and slightly underperform the others in Figure4. Calibration with T -scaling and smoothing with temperature do not change accuracy or mAP.

Difference between the goal of our method and several similar lines of research.

Table content: <table><tr><td/><td>assumes (or robust to)</td><td colspan="2">novel distr. unknown novel samples</td></tr><tr><td/><td>different train &amp; test distr.?</td><td>during training?</td><td>in testing:</td></tr><tr><td>Ours (reduce confident errors)</td><td>Yes</td><td>Yes</td><td>Predict</td></tr><tr><td>Novelty &amp; outlier detection</td><td>Yes</td><td>Varies</td><td>Reject</td></tr><tr><td>Rejection / failure detection</td><td>No</td><td>-</td><td>-</td></tr><tr><td>Semi-supervised learning</td><td>No</td><td>-</td><td>-</td></tr><tr><td>Epistemic uncertainty estimation</td><td>Yes</td><td>Yes</td><td>No evaluation</td></tr><tr><td>Active learning</td><td>Varies</td><td>No</td><td>Predict</td></tr><tr><td>Domain generalization</td><td>Yes</td><td>Limited 1</td><td>Predict</td></tr><tr><td>Domain Adaptation</td><td>Yes</td><td>No</td><td>Predict</td></tr><tr><td>Calibration</td><td>No</td><td>-</td><td>-</td></tr><tr><td colspan="4">by aggregating outputs; but for novel data, the different models can disagree, indicating a lack of</td></tr><tr><td colspan="4">training evidence. In that case, aggregating the outputs results in an appropriately lower confidence.</td></tr></table>

Illustration of the data usage of G-distillation on familiar data F, general data G, and some novel data N that we assume no knowledge of.He et al., 2016). Then, for each (x F , y F ) ∈ F tr , obtain a soft label y F , where y

Table content: <table><tr><td>training ensemble</td><td>Familiar F L cls w/ label y F</td><td>General G -</td><td>Novel N -</td></tr><tr><td>distillation evaluation</td><td>L cls w/ label y F L dis w/ soft label y F test on F ts</td><td>-L dis w/ soft label y G -</td><td>-test on N ts</td></tr><tr><td colspan="2">consists of several networks such as ResNet18 ((c)</td><td/><td/></tr></table>

Dataset usage. Each dataset is divided into F and N portions, and gereral G is selected by trying to choose a dataset diverse enough to encompass samples from F and potential N in the wild. (*) Please see text for details.

Table content: <table><tr><td>Task Gender recognition</td><td>familiar F LFW+ (Han &amp; Jain, 2014) age 18-59 (Han et al., 2017)</td><td>novel N LFW+, age 0-17 &amp; 60+</td><td>subsample -</td><td>general G CelebA (Liu et al., 2015)</td></tr><tr><td>ImageNet superclasses</td><td>ILSVRC12, some species*</td><td>ILSVRC12, other species*</td><td>Per superclass: train: 1k, test: 400</td><td>COCO (Lin et al., 2014)</td></tr><tr><td>Cat-dog binary recognition</td><td>Pets, some breeds* (Parkhi et al., 2012)</td><td>Pets, other breeds*</td><td>-</td><td>ILSVRC12 (Russakovsky et al., 2015)</td></tr><tr><td>VOC-COCO recognition</td><td>VOC, 20 classes (Everingham et al., 2015)</td><td>COCO, but ignore non-VOC classes</td><td>-</td><td>Places365-standard (Zhou et al., 2017)</td></tr></table>

Peer review (iclr2019):
 The paper proposes two ideas for reducing overconfident wrong predictions: - Method 1: “G-distillation” of an ensemble with extra unsupervised data - Method 2: Novelty Confidence reduction (NCR) using novelty detector The paper is well-written and was a pleasure to read. In particular, I really enjoyed reading the introduction and related work. My main concern is that some of the contributions claimed were already shown in previous work (see method 1 below for details), and the novelty feels a bit limited. That said, I like the simplicity of the method and think that the extensive experiments on a variety of datasets and architectures is useful to the community. Method 1: - The paper claims “Draw attention to a counter-intuitive yet important problem of highly confident wrong predictions when samples are drawn from a unknown distribution that is different than training” as one of the contributions. Note that previous work has already shown that single models are overconfident on unknown classes and ensembles are less overconfident, e.g. see Section 3.5 of the paper: Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles https://arxiv.org/pdf/1612.01474.pdf - If I understand correctly, the key difference is that the proposed method 1 also uses ensemble prediction on unlabeled data for distillation, which could make the distilled model more robust. Ensembling on unlabeled data for robustness does seem novel to me, however, the text needs to be updated to clarify the novelty. Method 2: - By off-the-shelf, do you mean a pre-trained network released by ODIN? Or did you train ODIN-based novelty detector on your dataset? - There was a recent paper that proposed to reduce confidence on novel inputs, which might be worth discussing: Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors https://arxiv.org/pdf/1807.09289.pdf Minor issues: - Figures 3,4 are a bit small and hard to see