Title: IMPROVING DIRICHLET PRIOR NETWORK FOR OUT-OF-DISTRIBUTION EXAMPLE DETECTION Anonymous

Abstract: Determining the source of uncertainties in the predictions of AI systems are important. It allows the users to act in an informative manner to improve the safety of such systems, applied to the real-world sensitive applications. Predictive uncertainties can originate from the uncertainty in model parameters, data uncertainty or due to distributional mismatch between training and test examples. While recently, significant progress has been made to improve the predictive uncertainty estimation of deep learning models, most of these approaches either conflate the distributional uncertainty with model uncertainty or data uncertainty. In contrast, the Dirichlet Prior Network (DPN) can model distributional uncertainty distinctly by parameterizing a prior Dirichlet over the predictive categorical distributions. However, their complex loss function by explicitly incorporating KL divergence between Dirichlet distributions often makes the error surface ill-suited to optimize for challenging datasets with multiple classes. In this paper, we present an improved DPN framework by proposing a novel loss function using the standard cross-entropy loss along with a regularization term to control the sharpness of the output Dirichlet distributions from the network. Our proposed loss function aims to improve the training efficiency of the DPN framework for challenging classification tasks with a large number of classes. In our experiments using synthetic and real datasets, we demonstrate that our DPN models can distinguish the distributional uncertainty from other uncertainty types. Our proposed approach significantly improves DPN frameworks and outperform the existing OOD detectors on CIFAR-10 and CIFAR-100 dataset while also being able to recognize distributional uncertainty distinctly.

1. INTRODUCTION:

Deep neural networks (DNNs) have achieved impeccable success to address various real world tasks (Simonyan & Zisserman, 2014a; Hinton et al., 2012; Litjens et al., 2017) . However, despite impressive, and ever-improving performance in various supervised learning tasks, DNNs tend to make over-confident predictions for every input. Predictive uncertainties of DNNs can be confronted from three different factors such as model uncertainty, data uncertainty and distributional uncertainty (Malinin & Gales, 2018) . Model or epistemic uncertainty captures the uncertainty in estimating the model parameters, conditioning on training data (Gal, 2016) . This uncertainty can be explained away given enough training data. Data or aleatoric uncertainty is originated from the inherent complexities of the training data, such as class overlap, label noise, homoscedastic and heteroscedastic noise (Gal, 2016) . Distributional uncertainty or dataset shift arises due to the distributional mismatch between the training and test examples (Quionero-Candela et al., 2009; Malinin & Gales, 2018) . In this case, as the network receives unfamiliar out-of-distribution (OOD) test data, it should not confidently make predictions. The ability to separately model these three types of predictive uncertainty is important, as it enables the users to take appropriate actions depending on the source of uncertainty. For example, in the active learning scenario, distributional uncertainty indicates that the classifier requires additional data for training. On the other hand, for various real-world applications where the cost of an error is high, such as in autonomous vehicle control, medical, financial and legal fields, the source of uncertainty informs whether an input requires manual intervention.

Recently notable progress has been made to detect OOD images. Bayesian neural network based approaches conflate the distributional uncertainty through model uncertainty (Hernandez-Lobato & Adams, 2015; Gal, 2016) . However, since obtaining the true posterior distribution for the model parameters are intractable, the success of these approaches depends on the chosen prior distribution over parameters and the nature of approximations. Here, the predictive uncertainties can be measured by using an an ensemble of multiple stochastic forward passes using dropouts from a single DNN (Monti-Carlo Dropout or MCDP) (Gal & Ghahramani, 2016) or by ensembling results from multiple DNNs (Lakshminarayanan et al., 2017) and computing their mean and spread. On the other hand, most of the non-Bayesian approaches model their distributional uncertainty with data uncertainty. These approaches can explicitly train the network in a multi-task fashion incorporating both in-domain and OOD examples to produce sharp and flat predictive posteriors respectively (Lee et al., 2018a; Hendrycks et al., 2019) . However, none of these approaches can robustly determine the source of uncertainty. Malinin & Gales (2018) introduced Dirichlet Prior Network (DPN) to distinctly model the distributional uncertainty from the other uncertainty types. A DPN classifier aims to produce sharp distributions to indicate low-order uncertainty for the in-domain examples and flat distributions for the OOD examples. However, their complex loss function, using the Kullback-Leibler (KL) divergence between Dirichlet distributions, results in the error surface to become poorly suited for optimization and makes it difficult efficiently train the DNN classifiers for challenging datasets with a large number of classes (Malinin & Gales, 2019) .

In this work, we aim to improve the training efficiency of the DPN framework by proposing a novel loss function that also allows the distributional uncertainty to be modeled distinctly from both data uncertainty and model uncertainty. Instead of explicitly using Dirichlet distributions in the loss function, we propose to apply the standard cross-entropy loss on the softmax outputs along with a novel regularization term for the logit (pre-softmax activation) outputs. We train the models in a multi-task function by leveraging both in-domain training images and OOD training images. The proposed loss function can be also viewed from the perspective of the non-Bayesian frameworks (Lee et al., 2018a; Hendrycks et al., 2019) where the proposed regularizer presents an additional term to control the sharpness of the output Dirichlet distributions. In our experiments, we demonstrate that our proposed regularization term can effectively control the sharpness of the output Dirichlet distributions from the DPN to detect distributional uncertainties along with making the framework scalable for more challenging datasets. We also find that the OOD detection performance of our DPN models often improves by augmenting Gaussian noises with the OOD training images to train the DPN. Our experimental results on CIFAR-10 and CIFAR-100 suggest that our proposed approach significantly improves the performance of the DPN framework for OOD detection and out-performs the recently proposed OOD detection techniques.

2. RELATED WORKS:

In Bayesian frameworks, the predictive uncertainty of a classification model, trained on a finite dataset, D in = {x i , y i } N i=1 ∼ P in (x, y), is expressed in terms of data (aleatoric) and model (epistemic) uncertainty (Gal, 2016) . For an input x * , the predictive uncertainty is expressed as:

EQUATION

Here, x and y represents the images and the corresponding class labels, sampled from an underlying probability distribution p in (x, y). Here, the data uncertainty, p(ω c |x * , θ) is described by the posterior distribution over class labels given model parameters, θ and model uncertainty, p(θ|D in ), is given by the posterior distribution over parameters given the data, D in .

The expected distribution for predictive uncertainty, p(ω c |x * , D in ) is obtained by marginalizing out θ. However, true posterior for p(θ|D in ) is intractable. Approaches such as Monte-Carlo dropout (MCDP) (Gal & Ghahramani, 2016) , Langevin Dynamics (Welling & Teh, 2011 ), explicit ensembling (Lakshminarayanan et al., 2017) approximate the integral in eq. 1 as:

EQUATION

where, θ (m) is sampled from an explicit or implicit variational approximation, q(θ) of the true posterior p(θ|D in ).

Each p(ω c |x * , θ (m) ) represents a categorical distribution, µ = [µ 1 , • • • , µ k ] = [p(y = ω 1 ), • • • , p(y = ω K )]

over the class labels and the ensemble can be visualized as a collection of points on the simplex. While for a confident prediction, the ensemble is expected to sharply appear in one corner of the simplex, the flatly spread ensembles cannot determine whether the uncertainty is due to data or distributional uncertainty. Furthermore, for standard DNNs, with millions of parameters, it becomes even harder to find an appropriate prior distribution and inference scheme to estimate the posterior distribution of the model. Dirichlet Prior Network (DPN) is introduced to explicitly model the distributional uncertainty by parameterizing a Dirichlet distribution over a simplex (Malinin & Gales, 2018) . More discussions about DPN is presented in section 3.1.

Alternatively, non-Bayesian frameworks derive their measure of uncertainties using the predictive posteriors obtained from DNNs. Lee et al. (2018a) and Hendrycks et al. (2019) introduce new components in their loss functions to explicitly incorporate OOD data for training. DeVries & Taylor (2018) append an auxiliary branch onto a pre-trained classifier to derive the OOD score. Shalev et al. (2018) uses multiple semantic dense representations as the target label to train the OOD detection network. Several recent works such as (Lee et al., 2018b; Liang et al., 2018) have demonstrated that by tweaking the input images during inference using adversarial perturbations can enhance the performance of a DNN for OOD detection (Goodfellow et al., 2014b) . However, their discriminative scores are achieved by tailoring the parameters for each OOD distributions during test time, which is not possible for real-world OOD examples. Hein et al. (2019) propose an adversarial training (Madry et al., 2018) like approach to produce lower confident predictions for OOD examples. However, while these models can identify the total predictive uncertainties, they can not robustly determine whether the source of uncertainty is due to an in-domain input in a region of class overlap or an OOD example far away from the training distribution.

3. PROPOSED METHODOLOGY:

This section first describes the DPN framework and the difficulties of the existing modeling techniques to scale DPNs for challenging datasets. We then present our improved version DPN by proposing a novel loss function to address these difficulties while allowing to model the distributional uncertainty distinctly from the model and data uncertainty.

3.1. DIRICHLET PRIOR NETWORK:

A DPN for classification directly parametrizes a prior Dirichlet distribution over the categorical output distributions on a simplex (Malinin & Gales, 2018) . For in-domain examples, a DPN attempts to produce sharp Dirichlet in one corner of the simplex, when it is confident in its predictions (Figure 1a ). It should produce a sharp distribution in the middle of the simplex to indicate the data (loworder) uncertainty for the in-domain example with a high degree of noise or belongs to a class overlapping region (Figure 1b ). In contrast, for OOD examples, a DPN should produce a Dirichlet that spreads over the simplex or across the edge of the simplex to indicate the distributional uncertainty (Figure1c). Here, the data uncertainty is expressed by the point-estimate categorical distribution µ while the distributional uncertainty is described using the distribution over the predictive categorical i.e p(µ|x * , θ). The overall predictive uncertainty is expressed as:

EQUATION

This expression forms a three layered hierarchy of uncertainties: a large model uncertainty, p(θ|D) would induce a large variation in distributional uncertainty in p(µ|x * , θ) and a large degree of uncertainty for µ leads to higher data uncertainty. DPN framework is consistent with the existing approaches, where an additional layer of uncertainty is included to capture the distributional uncertainty. For example, marginalization of µ in Eqn. 3 will reproduce Eqn. 1 while loose the control over the sharpness of the output Dirichlet distributions. The marginalization of θ produces the expected estimation of data and distributional uncertainty given model uncertainty as:

EQUATION

However, similar to eq. 1 marginalizing θ is eq. 4 is also intractable. Since the model uncertainty is reducible given large training data, for simplicity, here we assume a dirac delta estimation, θ for θ:

EQUATION

Constructing a DPN. A DPN constructs a Dirichlet distribution as a prior over the categorical distributions, which is parameterized by the concentration parameters,

EQUATION

where, α 0 is called the precision of the Dirichlet. A larger value of α 0 produces sharper distributions to indicate low order uncertainties (fig 1a  and 1b ). A DPN, f θ produces the concentration parameters, α and the posterior over class labels, p(ω c |x * ; θ), is given by the mean of the Dirichlet.

EQUATION

A standard DNN with the softmax activation function can be represented as a DPN where the concentration parameters are α c = e zc(x * ) ; z c (x * ) is the pre-softmax (logit) output corresponding to the class, c for an input x * . The expected posterior probability of class label ω c is given as:

p(ωc|x * ; θ) = αc α0 = e zc(x * ) K c=1 e zc(x * ) (8)

However, the mean of the Dirichlet is now insensitive to any arbitrary scaling of α c . Hence, the precision of the Dirichlet, α 0 , degrades under the standard cross-entropy loss.

Malinin & Gales (2018) instead introduced a new loss function that explicitly minimizes the KL divergence between the output Dirichlet and a target Dirichlet to produce a predefined target precision value for the output Dirichlet distributions. For in-domain examples, the target distribution is chosen to be a sharp Dirichlet, Dir(µ| αy ), focusing on their ground truth classes. While for OOD examples, a flat Dirichlet, Dir(µ| α) is selected that spreads over the whole simplex.

EQUATION

where, P in and P out are the underlying distribution of in-domain and OOD training examples respectively. However, learning the model using sparse 1-hot continuous distributions for class labels, which are effectively a delta function, is challenging due to their complex loss function (eq. 9).

Here, the error surface becomes poorly suited for optimization using the back-propagation algorithm (Malinin & Gales, 2019) . Malinin & Gales (2018) tackle this problem by using label smoothing (Szegedy et al., 2016) or teacher-student training (Hinton et al., 2015a) to redistribute a small amount of probability density to each corner of the Dirichlet. This technique is found to work well for datasets with a fewer number of class labels. However, for more challenging datasets with a large number of classes, even these techniques cannot efficiently redistribute the probability densities at each corner and results in the target distribution to tend to a delta function. Hence, it becomes difficult to train the DPN to achieve competitive performances. Malinin & Gales (2019) have recently proposed to reverse the terms within the KL divergence in eq. 9 to improve the training efficiency of DPN models. This approach still requires to explicitly constrain the precision of the output Dirichlet distributions using an appropriately chosen hyper-parameter for training.

3.2. IMPROVED DIRICHLET PRIOR NETWORK:

We now propose an improved technique to model DPN by proposing a novel loss function using the standard cross-entropy loss along with a regularization term to control the precision of the output Dirichlet distribution from the network. As we have seen in equation 8, the precision of a Dirichlet distribution produced by the standard DNN is given as

K c=1 exp z c (x *

). Hence, we can control the sharpness of the distribution by designing a regularization term that increases the sum of logit (presoftmax) outputs for the in-domain examples to produce sharp distributions to indicate their lower uncertainties. For the OOD examples, the regularization term aims to decrease the sum of logit (presoftmax) outputs to produce flat distributions to indicate distributional (higher-order) uncertainties. Hence, instead of explicitly constraining the precision the output Dirichlet with a specific hyperparameter, we allow the network to appropriately produce the precision values for different input. In this paper, we propose the regularization term as 1 K K c=1 sigmoid(z c (x)) to control the sharpness of the output Dirichlet distribution along with the standard cross-entropy loss for classification. Here, the sigmoid function is chosen for the regularization term as it always produces a value within the range of (0, 1) for any z c (x). For in-domain training examples, the loss function is given as:

EQUATION

For OOD training examples, the loss function is given as:

EQUATION

where, U denotes the uniform distribution over the class labels. H c is the cross-entropy function.

We train the network in a multi-task fashion with the overall loss function as:

EQUATION

In this loss function, we have incorporated three user-defined hyper-parameters: λ in , λ out and λ in Eq. 10, Eq. 11, and Eq. 12 respectively. λ balances between the loss values for in-domain examples and OOD examples. The hyper-parameters λ in and λ out controls the sharpness of the output Dirichlet from a DPN. By choosing λ in > 0, we enforce the network to produce positive logit values for in-domain examples that lead to producing sharper Dirichlet distributions (Fig. 2 (a)). We choose λ in > λ out to ensure that the density is either spread over the the simplex or across the boundary (Fig 1c ). The choice of λ in > λ out > 0 will lead the network to produce comparatively flatter Dirichlet distributions for OOD examples (Fig. 2b ). In contrast, by choosing λ out < 0, we enforce the network to produce negative values for z c (x * ) and hence fractional values for α c 's (i.e α c ∈ (0, 1)) for OOD examples. This will cause the densities of the Dirichlet to be distributed in the edges of the simplex and produces an extremely sharp distribution, as shown in Fig 2c .

The proposed loss function in Eq. 12 is also very closely related to non-Bayesian approaches, where by choosing λ in , λ out to zero we re-obtain similar loss functions as proposed by Lee et al. (2018a) ; Hendrycks et al. (2019) . However, by setting λ in , λ out to zero, we lose control over the precision of the Dirichlet distribution that distinguishes distributional uncertainty from data uncertainty.

Our multi-task loss function (eq. 12) requires training samples from the in-domain distribution, P in as well as from OOD P out . However, since P out is unknown, Lee et al. (2018a) propose to synthetically generate the OOD training samples from the boundary of in-domain region, P in using generative models such as GAN (Goodfellow et al., 2014a) . Alternatively, a different, easily available, real datasets can be used as OOD training examples. In practice, the latter approach is to found to be more effective for training the OOD detectors and has been applied for our experiments on vision datasets (Hendrycks et al., 2019) .

4. EXPERIMENTAL STUDY:

We demonstrate the effectiveness of the DPN framework using the proposed loss function by conducting two sets of experiments. First, we experiment on a synthetic dataset. Next, we present a comparative study of our proposed framework with the existing approaches on CIFAR10 and CI-FAR100, and show the advantages over the original DPN framework (Malinin & Gales, 2018) .

4.1. SYNTHETIC DATASET:

We construct a simple dataset with three classes where the instances are sampled from three different isotropic Gaussian distributions as shown in Figure 3 (a). We select isotropic co-variances, σ 2 I with σ = 4, to ensure that the classes are overlapping. We train a small DPN with 2 hidden layers of 50 Entropy of the predicted distribution, H[p(ωc|x * , Din)] = -K c=1 p(ωc|x * , Din) ln p(ωc|x * , Din), can be also applied as a total uncertainty measure that produces low scores when the model is confident in its prediction (Figure 3 (d) and 3(e) for DPN(β = 0.0) and DPN(β = 0.5) respectively).

Max probability and entropy are the most frequently used uncertainty measures used by the existing OOD detection models. However, since the predicted distribution is obtained by marginalizing µ (eq. 3), these measures cannot capture the sharpness of the output Dirichlet for a DPN. Hence, they cannot distinguish between misclassified examples and OOD examples. This observation also indicates the limitation of the existing non-DPN approachesto differentiate between data and distributional uncertainties (Malinin & Gales, 2018) . A DPN framework address this limitation by using the differential entropy (D-Ent) as an uncertainty measure that produces lower scores for sharper Dirichlet distributions (eq. 13). Appendix D presents the expression to compute D-Ent scores for a given data point. Since we explicitly constrain the logit outputs to produce smaller values for OOD examples, it is meaningful to define the sum of the exponential of logits, K c=1 e zc(x * ) , as a new measure of predictive uncertainty. In Figure 3 (d) and 3(h), we visualize this uncertainty measure for our DPN(β = 0.0) and DPN(β = 0.5), respectively. We found that our DPN models produce very high logit values. Hence, we scaled down these values by a factor of 100 before computing this uncertainty measure for better visualization. We can see, K c=1 e zc(x * ) produces high scores for all in-domain data points to distinguish them from OOD examples.

EQUATION

4.2. EXPERIMENTS ON CIFAR-10 AND CIFAR-100:

Experimental setup. In this section, we present our experiments on CIFAR-10 and CIFAR-100 datasets (Krizhevsky, 2009) . CIFAR-10 images belong to 10 different classes while CIFAR-100 is a more challenging dataset, containing 100 image classes. Both of these datasets contain 32 × 32 natural colored images of 50, 000 training and 10, 000 testing examples. In Appendix-A, we have also experimented on TinyImageNet (TIM) that consists of 64 × 64 natural images from 200 classes.

The CIFAR-10 classifiers are trained using CIFAR-10 training images as in-domain and CIFAR-100 training images as OOD dataset (Simonyan & Zisserman, 2014b) . We use VGG-16 architecture for this case. For CIFAR-100, we use CIFAR-100 training images as in-domain and CIFAR-10 training images as the OOD dataset. Here, we consider a DenseNet with depth = 55 and growth rate = 12 (Huang et al., 2017) .

We present in-domain misclassification detection and OOD detection experiments to evaluate the performance of our models. For our misclassification detection experiments, we take the in-domain test set and attempt to distinguish the correctly classified examples from misclassified examples (see Table 2 ). The OOD detection experiments attempt to distinguish the in-domain test images from unknown OOD images. In Table 3 , we present a smaller set of results where we consider TinyImageNet (TIM) as the OOD dataset and attempt to distinguish them from in-domain test data points. In Appendix A, we present an expanded version of this comparative table for a wide range of OOD examples. Note that, the OOD test images are selected from datasets different from the datasets used for training. The description of our training and test datasets are presented in Table 1 . Please refer to Appendix C for further details.

Evaluation of Predictive Uncertainty Estimation.:

To evaluate the performance of our model for misclassification detection, we consider the misclassified examples as the positive class and correctly classified examples as the negative class. For the OOD detection task, we treat the OOD examples as the positive class and in-domain examples as a negative class. The detection performance for these tasks are measured using two metrics: area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPR) (Hendrycks & Gimpel, 2016) . The AUROC can be interpreted as the probability of an OOD example to produce a higher detection score than an in-domain example (Davis & Goadrich, 2006) . Hence, a higher AUROC is desirable, and an uninformative detector produces an AUROC ≈ 50%. The AUPR is more informative when the positive class and negative class have greatly differing base rates. It can take these base rates into account (Manning & Schütze, 1999) During training, we set λ = 0.5 (eq. 12), similar to Hendrycks et al. (2019) . We train multiple DPN models for CIFAR-10 and CIFAR-100 classifiers using both positive and negative values for λout. We choose λin and λout as λin = (1 -β) and λout = ( 1 #class -β). We train two different sets of DPN models using 0.0 (i.e positive λout) and β = 0.5 (i.e negative λout), denoted as DPN sof t (β : 0.0) and DPN sof t (β : 0.5), respectively.

Detecting the source of uncertainty. [1] DPN model with λout > 0: By choosing a positive value for λout, we enforce the network to produce flat Dirichlet distributions over the simplex for OOD examples. Hence, the D-Ent measure should produce higher scores for OOD examples and lower scores for in-domain examples. Therefore, given a test example with high scores for both entropy as well as D-Ent, it indicates distributional uncertainty. In contrast, if the test example achieves high entropy scores and lower D-Ent score, it indicates data uncertainty. Our DPN sof t (β : 0.0) models achieve high AUROC and AUPR scores using D-Ent to detect OOD examples for both CIFAR-10 and CIFAR-100 (see Table 3 ) while achieves lower AUROC and AUPR scores for D-Ent while detecting misclasification for CIFAR-100 (see Table 2 ). However for CIFAR- 2 and 3 ). This indicates that both DPNDir and our DPN sof t (β : 0.0) models may be ineffective to detect distributional uncertainty for CIFAR-10. In contrast, our DPN sof t (β : 0.5) models are always found to be effective to identify the source of uncertainties.

[2] DPN model with λout < 0: In contrast, if we choose a negative value for λout, our DPN network produces very sharp Dirichlet distributions across the edge of the simplex for OOD examples. Hence, it produces very low D-Ent scores. Notably, these distributions are even sharper than the Dirichlet distributions obtained for confidently predicted examples. Hence, for a test example, we can detect distributional uncertainty if it achieves high entropy and low D-Ent scores. As we can see, our DPN sof t (β : 0.5) models produce very low AUROC and AUPR scores for D-Ent measure to detect OOD examples (Table 3 ) while achieve relatively higher AUROC and AUPR scores for detecting misclassified examples (Table 2 ).

Data Augmentation with white-noise for Training. After training our DPN models using clean training images, we also fine-tune them using noisy OOD training images for a few epochs. We add minor Gaussian noise sampled from isotropic Gaussian distributions N (0, σ 2 I) with our OOD training images. Here, the idea is to add minor perturbation without distorting the features of the OOD training images. It further exposes the network into the out of distribution space. Note that, the DPN models are fine-tuned only using OOD training images. The OOD test examples remain unknown. In practice, this technique may often further improve the OOD detection performance of our DPN sof t models as we choose σ = 0.01 (see Table 3 ). However, for larger perturbation, performance degrades for the DPN sof t (σ = 0.05).

Comparative Study. In Table 2 and 3 , we compare the performance of our approach with several baselines, such as standard DNN (Hendrycks & Gimpel (2016) ), MCDP (Gal & Ghahramani, 2016) , DPNDir (Malinin & Gales, 2018) , ODIN (Liang et al., 2018) , and OE (Hendrycks et al., 2019) . We use the same architecture as our DPN sof t models for all the competitive models. OE models are trained using the set of in-domain and OOD training images with their proposed loss function (Hendrycks et al., 2019) . Note that, since non-DPN methods do not explicitly model the logit outputs, the D-Ent or K c=1 e zc(x * ) measures are not meaningful (Malinin & Gales, 2018) .

We also compare our results with the existing DPN framework, namely DPNDir Malinin & Gales (2018) for CIFAR-10. Note that, the DPNDir framework fails to scale for the CIFAR-100 dataset that consists of 100 classes. Due to the unavailability of codes and results (under the same settings), we could not compare our model with Malinin & Gales (2019) . Overall, our DPN sof t models significantly improved the performance of the DPN framework and consistently out-performed the existing OOD detection models. In addition, it is able to distinguish distributional uncertainty from other uncertainty types.

with the standard cross-entropy loss function. Our experiments on synthetic and real datasets demonstrate that our proposed framework can efficiently distinguish the distributional uncertainty from other uncertainty types. We demonstrate that the OOD detection performance of our DPN models can be often improved by training with noisy OOD examples. Experiments show that our proposed approach significantly improves DPN frameworks, and outperforms the existing OOD detectors on both CIFAR-10 and CIFAR-100 datasets. (Huang et al. (2017) ). We trained multiple DPN sof t models using our proposed loss functions with different hyper-parameters. examples cannot be known. Hence, for our comparisons, we always set the perturbation size to 0.002 and the temperature to 1000.

A EXPANDED RESULTS:

OE models are trained using the proposed loss function by Hendrycks et al. (2019) . Here, we use the same training set up as applied for our DPN sof t models: CIFAR-10 classifiers are trained using CIFAR- 

Note that, αc is a function of x * . Γ and ψ denotes the Gamma and digamma functions respectively.

:

CONCLUSIONIn this paper, we propose a novel framework to improve the training efficiency of DPN models for challenging classification tasks with large number of classes. We also propose a novel regularization term that allows controls the sharpness of the Dirichlet distributions. We show that the proposed regularizer can be easily integrated

The three classes of our synthetic dataset are constructed by sampling from three different isotropic Gaussian distributions with means of (-4, 0), (4, 0) and (0, 5) and isotropic variances of σ = 4. We sample 200 training data points from each distribution for each class. We also sample 600 OOD training examples from an uniform distribution of U([-15, 15],[-13, 17]).

as the OOD training data(Simonyan & Zisserman (2014b)). For CIFAR-100, we train a DenseNet with

Figures and Tables:

Figure 1: Desired behavior of a DPN to indicate the three different uncertainties.

Figure 2: Dirichlet distributions become flatter as the precision is reduced (from (a) to (b)) and the densities move to the edges as the concentration parameters become fractional i.e αc ∈ (0, 1).

Figure 3: Visualizing the uncertainty measures for different data-points for DPN(β = 0.0) in the top row and for DPN(β = 0.5) in the bottom row.

Figure 3(f) and 3(g) demonstrate that D-Ent can distinguish between in-domain and OOD examples. DPN(β = 0.0) produces smaller positive logit values, resulting in flat Dirichlet distributions for OOD examples. Hence, we obtain relatively lower D-Ent scores for OOD examples compared to the in-domain examples (fig 3 (d)). Subsequently, DPN(β = 0.5) produces negative logit values, resulting in sharp Dirichlet distributions across the edge of the simplex for OOD examples (Fig. 2(c)). As we can see in Figure 3(g), the D-Ent scores for OOD examples are even smaller than the in-domain confident predictions, indicating that the output Dirichlet distributions for OOD examples are even sharper.

CIFAR-10, we use the VGG-16 network. Here, we use CIFAR-10 training images (50, 000 images) as our in-domain training data and CIFAR-100 training images (50, 000 images) as our OOD training data.For CIFAR-100, Densenet(55, 12) is trained using the same setup as proposed byHuang et al. (2017). Here, we use CIFAR-100 training images (50, 000 images) as our in-domain training data and CIFAR-10 training images (50, 000 images) as our OOD training data.

10 training images as in-domain examples and CIFAR-100 training images as OOD examples. For CIFAR-100, the OE models are trained using CIFAR-10 training images as OOD examples.D DIFFERENTIAL ENTROPY MEASURE FOR DIRICHLET PRIOR NETWORKDifferential Entropy of a Dirichlet distribution can be calculated as follows(Malinin & Gales, 2018):H[p(µ|x * , Din)] = -S K-1p(µ|x * , Din) ln p(µ|x * , Din)

Training details for our proposed DPN models

Table content: <table><tr><td colspan="3">Classification Task Input Shape #Classes</td><td colspan="2">Details of Training data In-Domain OOD</td><td>Details of Test data</td></tr><tr><td>CIFAR-10</td><td>32 × 32</td><td>10</td><td>CIFAR-10 training set (50,000 images)</td><td>CIFAR-100 training set (50,000 images)</td><td>In-Domain: CIFAR-10 test set (10,000 Images) OOD: TIM, LSUN, etc.</td></tr><tr><td>CIFAR-100</td><td>32 × 32</td><td>100</td><td>CIFAR-100 training set (50,000 images)</td><td>CIFAR-10 training set (50,000 images)</td><td>In-Domain: CIFAR-10 test set (10,000 Images) OOD: TIM, LSUN, etc.</td></tr></table>

.

Table content: <table><tr><td>D test in</td><td>Methods</td><td colspan="2">Max.P Ent.</td><td colspan="5">AUROC e zc(x  *  ) D-Ent Max.P Ent.</td><td colspan="2">AUPR e zc(x  *  ) D-Ent</td><td>Acc.(%)</td></tr><tr><td/><td>Baseline</td><td>93.2</td><td colspan="2">93.3</td><td>-</td><td>-</td><td>43.0</td><td>46.6</td><td>-</td><td>-</td><td>94.1</td></tr><tr><td/><td>MCDP</td><td>93.6</td><td colspan="2">93.6</td><td>-</td><td>-</td><td>46.1</td><td>46.3</td><td>-</td><td>-</td><td>94.1</td></tr><tr><td>CIFAR-10</td><td>ODIN OE DPNDir DPNsoft(β : 0.0, σ : 0.0)</td><td>91.1 92.1 92.2 91.7</td><td colspan="2">-91.6 92.1 91.3</td><td>---90.1</td><td>--90.9 90.2</td><td>48.5 36.4 52.7 37.0</td><td>-34.8 51.0 35.7</td><td>---32.2</td><td>--45.5 32.5</td><td>94.1 94.0 92.5 94.0</td></tr><tr><td/><td>DPNsoft(β : 0.0, σ : 0.01)</td><td>91.3</td><td colspan="2">90.8</td><td>88.5</td><td>88.9</td><td>35.0</td><td>33.3</td><td>29.1</td><td>29.7</td><td>93.6</td></tr><tr><td/><td>DPNsoft(β : 0.0, σ : 0.05)</td><td>93.4</td><td colspan="2">93.2</td><td>91.4</td><td>91.8</td><td>44.7</td><td>42.5</td><td>35.1</td><td>36.4</td><td>93.7</td></tr><tr><td/><td>DPNsoft(β : 0.5, σ : 0.0)</td><td>92.0</td><td colspan="2">91.6</td><td>89.9</td><td>60.3</td><td>36.8</td><td>34.8</td><td>31.5</td><td>17.0</td><td>94.1</td></tr><tr><td/><td>DPNsoft(β : 0.5, σ : 0.01)</td><td>91.3</td><td colspan="2">90.8</td><td>87.7</td><td>61.9</td><td>37.1</td><td>35.3</td><td>28.9</td><td>17.0</td><td>94.1</td></tr><tr><td/><td>DPNsoft(β : 0.5, σ : 0.05)</td><td>93.0</td><td colspan="2">92.8</td><td>90.1</td><td>88.8</td><td>39.8</td><td>38.3</td><td>29.8</td><td>32.8</td><td>94.1</td></tr><tr><td/><td>Baseline</td><td>86.8</td><td colspan="2">87.0</td><td>-</td><td>-</td><td>63.6</td><td>64.7</td><td>-</td><td>-</td><td>76.3</td></tr><tr><td>CIFAR-100</td><td>MCDP ODIN OE DPNsoft(β : 0.0, σ : 0.0) DPNsoft(β : 0.0, σ : 0.01)</td><td>87.7 79.5 86.0 86.5 86.9</td><td colspan="2">87.7 -85.0 85.4 86.2</td><td>---74.4 75.1</td><td>---75.0 75.6</td><td>65.7 55.3 59.3 60.7 61.5</td><td>65.8 -55.5 57.1 58.5</td><td>---42.6 43.3</td><td>---41.8 43.1</td><td>76.9 76.3 76.3 75.7 75.6</td></tr><tr><td/><td>DPNsoft(β : 0.0, σ : 0.05)</td><td>87.3</td><td colspan="2">86.7</td><td>76.7</td><td>76.4</td><td>64.6</td><td>61.7</td><td>45.8</td><td>43.3</td><td>75.7</td></tr><tr><td/><td>DPNsoft(β : 0.5, σ : 0.0)</td><td>86.7</td><td colspan="2">85.6</td><td>73.5</td><td>73.1</td><td>61.3</td><td>57.3</td><td>41.4</td><td>41.9</td><td>76.0</td></tr><tr><td/><td>DPNsoft(β : 0.5, σ : 0.01)</td><td>86.5</td><td colspan="2">85.7</td><td>74.1</td><td>74.6</td><td>61.0</td><td>57.5</td><td>41.4</td><td>43.2</td><td>76.2</td></tr><tr><td/><td>DPNsoft(β : 0.5, σ : 0.05)</td><td>87.6</td><td colspan="2">87.0</td><td>76.3</td><td>77.1</td><td>64.6</td><td>61.9</td><td>45.4</td><td>46.1</td><td>76.3</td></tr></table>

Comparative results of misclassified image detection for CIFAR-10 and CIFAR-100. Hyper-parameters for Training Loss. Unlike Liang et al. (2018); Lee et al. (2018a), and similar toMalinin & Gales (2018); Hendrycks et al. (2019), we do not require to tune any hyper-parameters during testing for different datasets. In other words, the OOD test examples remain unknown to our DPN classifiers, as in a real-world scenario.

Table content: <table/>

Comparative results of OOD example detection for CIFAR-10 and CIFAR-100. Expanded version of this table along with a wide range of OOD datasets are provided in Appendix A. 10, our DPN sof t (β : 0.0) models fail to produce low AUROC and AUPR scores under D-Ent measure for misclassification detection. This indicates that these models achieve high D-Ent scores for both misclassified and OOD examples and make it difficult to distinguish distributional uncertainty from data uncertainty for CIFAR-10.Note that, similar to our DPN sof t (β : 0.0), DPNDir(Malinin & Gales, 2018) also produces high AUPR and AUROC scores under D-Ent measure for for both misclassified examples and OOD examples for CIFAR-10 model (Table

Table content: <table><tr><td>D test in</td><td>D test out</td><td>Methods</td><td colspan="2">Max.P Ent.</td><td colspan="5">AUROC e zc(x  *  ) D-Ent Max.P Ent.</td><td colspan="2">AUPR e zc(x  *  ) D-Ent</td></tr><tr><td/><td/><td>Baseline</td><td>88.8</td><td colspan="2">89.4</td><td>-</td><td>-</td><td>85.1</td><td>86.7</td><td>-</td><td>-</td></tr><tr><td/><td/><td>MCDP</td><td>88.5</td><td colspan="2">89.2</td><td>-</td><td>-</td><td>84.7</td><td>86.1</td><td>-</td><td>-</td></tr><tr><td>CIFAR-10</td><td>TIM</td><td>ODIN OE DPNDir DPNsoft(β : 0.0, σ : 0.0)</td><td>94.4 98.0 94.3 97.6</td><td colspan="2">-98.0 94.3 97.7</td><td>--97.6</td><td>--94.6 97.6</td><td>93.8 97.9 94.0 97.5</td><td>-97.9 94.0 97.6</td><td>--97.5</td><td>--94.2 97.5</td></tr><tr><td/><td/><td>DPNsoft(β : 0.0, σ : 0.01)</td><td>98.5</td><td colspan="2">98.5</td><td>98.4</td><td>98.5</td><td>98.4</td><td>98.5</td><td>98.3</td><td>98.4</td></tr><tr><td/><td/><td>DPNsoft(β : 0.0, σ : 0.05)</td><td>97.1</td><td colspan="2">97.4</td><td>97.6</td><td>97.7</td><td>97.2</td><td>97.5</td><td>97.7</td><td>97.8</td></tr><tr><td/><td/><td>DPNsoft(β : 0.5, σ : 0.0)</td><td>98.7</td><td colspan="2">98.8</td><td>96.7</td><td>6.8</td><td>98.6</td><td>98.7</td><td>92.8</td><td>32.5</td></tr><tr><td/><td/><td>DPNsoft(β : 0.5, σ : 0.01)</td><td>99.0</td><td colspan="2">99.1</td><td>96.3</td><td>6.8</td><td>98.5</td><td>98.9</td><td>90.7</td><td>32.2</td></tr><tr><td/><td/><td>DPNsoft(β : 0.5, σ : 0.05)</td><td>97.9</td><td colspan="2">98.2</td><td>98.2</td><td>30.9</td><td>97.9</td><td>98.2</td><td>98.1</td><td>54.1</td></tr><tr><td/><td/><td>Baseline</td><td>74.9</td><td colspan="2">76.3</td><td>-</td><td>-</td><td>71.1</td><td>73.1</td><td>-</td><td>-</td></tr><tr><td>CIFAR-100</td><td>TIM</td><td>MCDP ODIN OE DPNsoft(β : 0.0, σ : 0.0) DPNsoft(β : 0.0, σ : 0.01)</td><td>78.9 83.8 86.5 88.9 96.5</td><td colspan="2">81.0 -88.0 90.3 97.4</td><td>---91.1 98.8</td><td>---90.7 98.0</td><td>75.4 81.4 82.8 86.1 97.1</td><td>78.0 -83.0 86.4 97.8</td><td>---85.4 98.9</td><td>---83.6 95.2</td></tr><tr><td/><td/><td>DPNsoft(β : 0.0, σ : 0.05)</td><td>95.8</td><td colspan="2">96.7</td><td>98.0</td><td>96.7</td><td>96.4</td><td>97.2</td><td>98.2</td><td>92.9</td></tr><tr><td/><td/><td>DPNsoft(β : 0.5, σ : 0.0)</td><td>98.9</td><td colspan="2">99.1</td><td>99.5</td><td>5.0</td><td>98.9</td><td>99.0</td><td>99.5</td><td>31.8</td></tr><tr><td/><td/><td>DPNsoft(β : 0.5, σ : 0.01)</td><td>99.1</td><td colspan="2">99.3</td><td>99.7</td><td>5.9</td><td>99.2</td><td>99.3</td><td>99.7</td><td>32.4</td></tr><tr><td/><td/><td>DPNsoft(β : 0.5, σ : 0.05)</td><td>95.8</td><td colspan="2">96.9</td><td>98.8</td><td>41.8</td><td>96.4</td><td>97.4</td><td>98.9</td><td>59.8</td></tr></table>

Results of OOD image detection for CIFAR-10. Description of these OOD datasets are provided in Appendix C.2.

Table content: <table><tr><td>D test out</td><td>Methods</td><td colspan="2">Max.P Ent.</td><td colspan="5">AUROC e zc(x  *  ) D-Ent Max.P Ent.</td><td colspan="2">AUPR e zc(x  *  ) D-Ent</td></tr><tr><td/><td>Baseline</td><td>73.0</td><td colspan="2">73.3</td><td>-</td><td>-</td><td>70.4</td><td>71.3</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>74.0</td><td colspan="2">74.0</td><td>-</td><td>-</td><td>70.9</td><td>71.2</td><td>-</td><td>-</td></tr><tr><td>Gaussian</td><td>ODIN OE DPN sof t (β : 0.0, σ : 0.0) DPN sof t (β : 0.0, σ : 0.01)</td><td>72.6 81.9 82.3 96.0</td><td colspan="2">-82.0 82.4 96.2</td><td>--82.7 96.4</td><td>--82.7 96.4</td><td>70.1 82.5 83.9 96.3</td><td>-82.6 84.1 96.7</td><td>--84.0 96.7</td><td>--84.0 96.7</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.05)</td><td>89.2</td><td colspan="2">89.6</td><td>90.8</td><td>90.7</td><td>91.6</td><td>92.0</td><td>93.0</td><td>92.9</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>82.9</td><td colspan="2">83.0</td><td>83.3</td><td>37.3</td><td>83.8</td><td>83.9</td><td>82.6</td><td>48.7</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>95.8</td><td colspan="2">96.0</td><td>94.6</td><td>13.0</td><td>96.4</td><td>96.7</td><td>90.9</td><td>35.1</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>90.2</td><td colspan="2">90.6</td><td>92.4</td><td>28.1</td><td>92.4</td><td>92.8</td><td>94.0</td><td>47.3</td></tr><tr><td/><td>Baseline</td><td>88.8</td><td colspan="2">89.4</td><td>-</td><td>-</td><td>85.1</td><td>86.7</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>88.5</td><td colspan="2">89.2</td><td>-</td><td>-</td><td>84.7</td><td>86.1</td><td>-</td><td>-</td></tr><tr><td/><td>ODIN</td><td>94.4</td><td>-</td><td/><td>-</td><td>-</td><td>93.8</td><td>-</td><td>-</td><td>-</td></tr><tr><td>TIM</td><td>OE DPN Dir</td><td>98.0 94.3</td><td colspan="2">98.0 94.3</td><td>--</td><td>-94.6</td><td>97.9 94.0</td><td>97.9 94.0</td><td>-</td><td>-94.2</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.0)</td><td>97.6</td><td colspan="2">97.7</td><td>97.6</td><td>97.6</td><td>97.5</td><td>97.6</td><td>97.5</td><td>97.5</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.01)</td><td>98.5</td><td colspan="2">98.5</td><td>98.4</td><td>98.5</td><td>98.4</td><td>98.5</td><td>98.3</td><td>98.4</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.05)</td><td>97.1</td><td colspan="2">97.4</td><td>97.6</td><td>97.7</td><td>97.2</td><td>97.5</td><td>97.7</td><td>97.8</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>98.7</td><td colspan="2">98.8</td><td>96.7</td><td>6.8</td><td>98.6</td><td>98.7</td><td>92.8</td><td>32.5</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>99.0</td><td colspan="2">99.1</td><td>96.3</td><td>6.8</td><td>98.5</td><td>98.9</td><td>90.7</td><td>32.2</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>97.9</td><td colspan="2">98.2</td><td>98.2</td><td>30.9</td><td>97.9</td><td>98.2</td><td>98.1</td><td>54.1</td></tr><tr><td/><td>Baseline</td><td>90.2</td><td colspan="2">91.0</td><td>-</td><td>-</td><td>86.6</td><td>88.6</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>90.3</td><td colspan="2">91.1</td><td>-</td><td>-</td><td>86.8</td><td>88.9</td><td>-</td><td>-</td></tr><tr><td>LSUN</td><td>ODIN OE DPN Dir</td><td>96.6 97.9 94.4</td><td colspan="2">-98.0 94.4</td><td>--</td><td>--94.6</td><td>96.2 97.6 93.3</td><td>-97.7 93.4</td><td>--</td><td>--93.3</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.0)</td><td>97.7</td><td colspan="2">97.8</td><td>97.7</td><td>97.7</td><td>97.3</td><td>97.4</td><td>97.3</td><td>97.3</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.01)</td><td>98.7</td><td colspan="2">98.7</td><td>98.6</td><td>98.6</td><td>98.5</td><td>98.5</td><td>98.3</td><td>98.3</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.05)</td><td>97.3</td><td colspan="2">97.7</td><td>97.9</td><td>98.0</td><td>97.2</td><td>97.6</td><td>97.7</td><td>97.8</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>98.8</td><td colspan="2">98.8</td><td>97.1</td><td>6.2</td><td>98.6</td><td>98.6</td><td>93.8</td><td>32.4</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>99.2</td><td colspan="2">99.2</td><td>96.7</td><td>5.5</td><td>98.7</td><td>98.9</td><td>91.8</td><td>31.8</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>97.9</td><td colspan="2">98.2</td><td>98.4</td><td>33.0</td><td>97.7</td><td>98.1</td><td>98.3</td><td>56.0</td></tr><tr><td/><td>Baseline</td><td>89.4</td><td colspan="2">90.0</td><td>-</td><td>-</td><td>95.6</td><td>96.2</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>89.3</td><td colspan="2">90.2</td><td>-</td><td>-</td><td>95.6</td><td>96.2</td><td>-</td><td>-</td></tr><tr><td>Places365</td><td>ODIN OE DPN sof t (β : 0.0, σ : 0.0) DPN sof t (β : 0.0, σ : 0.01)</td><td>95.4 97.9 97.7 98.8</td><td colspan="2">-98.0 97.8 98.8</td><td>--97.7 98.7</td><td>--97.7 98.7</td><td>98.5 99.4 99.3 99.6</td><td>-99.4 99.3 99.6</td><td>--99.3 99.6</td><td>--99.3 99.6</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.05)</td><td>97.3</td><td colspan="2">97.6</td><td>97.8</td><td>97.8</td><td>99.2</td><td>99.3</td><td>99.4</td><td>99.4</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>98.7</td><td colspan="2">98.7</td><td>96.7</td><td>6.7</td><td>99.6</td><td>99.6</td><td>97.9</td><td>60.7</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>99.2</td><td colspan="2">99.3</td><td>96.5</td><td>5.4</td><td>99.6</td><td>99.7</td><td>97.3</td><td>59.5</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>97.9</td><td colspan="2">98.2</td><td>98.3</td><td>29.5</td><td>99.4</td><td>99.5</td><td>99.5</td><td>77.1</td></tr><tr><td/><td>Baseline</td><td>88.6</td><td colspan="2">89.0</td><td>-</td><td>-</td><td>74.9</td><td>77.0</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>87.6</td><td colspan="2">88.0</td><td>-</td><td>-</td><td>73.7</td><td>75.8</td><td>-</td><td>-</td></tr><tr><td>Textures</td><td>ODIN OE DPN sof t (β : 0.0, σ : 0.0) DPN sof t (β : 0.0, σ : 0.01)</td><td>94.9 99.2 99.5 99.5</td><td colspan="2">-99.7 99.5 99.5</td><td>--99.5 99.3</td><td>--99.5 99.4</td><td>90.5 98.6 98.8 98.9</td><td>-98.5 99.0 98.9</td><td>--99.0 98.6</td><td>--99.0 98.7</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.05)</td><td>99.4</td><td colspan="2">99.5</td><td>99.5</td><td>99.6</td><td>99.0</td><td>99.1</td><td>99.2</td><td>99.3</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>99.5</td><td colspan="2">99.5</td><td>96.7</td><td>4.6</td><td>99.1</td><td>99.1</td><td>85.6</td><td>21.3</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>99.4</td><td colspan="2">99.5</td><td>96.5</td><td>5.5</td><td>98.0</td><td>98.8</td><td>84.3</td><td>21.4</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>99.1</td><td colspan="2">99.3</td><td>98.7</td><td>24.7</td><td>98.3</td><td>98.7</td><td>97.3</td><td>38.3</td></tr></table>

Results of OOD image detection for CIFAR-100. Description of these OOD datasets are provided in Appendix C.2.

Table content: <table><tr><td>D test out</td><td>Methods</td><td colspan="2">Max.P Ent.</td><td colspan="5">AUROC e zc(x  *  ) D-Ent Max.P Ent.</td><td colspan="2">AUPR e zc(x  *  ) D-Ent</td></tr><tr><td/><td>Baseline</td><td>73.2</td><td colspan="2">74.0</td><td>-</td><td>-</td><td>67.7</td><td>68.5</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>74.4</td><td colspan="2">74.7</td><td>-</td><td>-</td><td>68.9</td><td>69.3</td><td>-</td><td>-</td></tr><tr><td>Gaussian</td><td>ODIN OE DPN sof t (β : 0.0, σ : 0.0) DPN sof t (β : 0.0, σ : 0.01)</td><td>80.4 72.9 71.7 94.4</td><td colspan="2">-74.0 72.6 95.2</td><td>--71.9 97.6</td><td>--71.9 96.4</td><td>76.2 64.6 63.6 95.3</td><td>-64.9 63.9 96.0</td><td>--61.7 97.9</td><td>--61.5 94.0</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.05)</td><td>96.1</td><td colspan="2">96.7</td><td>98.7</td><td>97.2</td><td>97.0</td><td>97.5</td><td>98.9</td><td>93.5</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>91.9</td><td colspan="2">92.8</td><td>96.3</td><td>37.6</td><td>91.8</td><td>92.4</td><td>95.8</td><td>50.8</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>98.3</td><td colspan="2">98.6</td><td>99.6</td><td>3.4</td><td>98.6</td><td>98.8</td><td>99.7</td><td>31.0</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>96.9</td><td colspan="2">97.4</td><td>99.6</td><td>9.3</td><td>97.7</td><td>98.1</td><td>99.7</td><td>33.6</td></tr><tr><td/><td>Baseline</td><td>74.9</td><td colspan="2">76.3</td><td>-</td><td>-</td><td>71.1</td><td>73.1</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>78.9</td><td colspan="2">81.0</td><td>-</td><td>-</td><td>75.4</td><td>78.0</td><td>-</td><td>-</td></tr><tr><td/><td>ODIN</td><td>83.8</td><td>-</td><td/><td>-</td><td>-</td><td>81.4</td><td>-</td><td>-</td><td>-</td></tr><tr><td>TIM</td><td>OE DPN sof t (β : 0.0, σ : 0.0)</td><td>86.5 88.9</td><td colspan="2">88.0 90.3</td><td>-91.1</td><td>-90.7</td><td>82.8 86.1</td><td>83.0 86.4</td><td>-85.4</td><td>-83.6</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.01)</td><td>96.5</td><td colspan="2">97.4</td><td>98.8</td><td>98.0</td><td>97.1</td><td>97.8</td><td>98.9</td><td>95.2</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.05)</td><td>95.8</td><td colspan="2">96.7</td><td>98.0</td><td>96.7</td><td>96.4</td><td>97.2</td><td>98.2</td><td>92.9</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>98.9</td><td colspan="2">99.1</td><td>99.5</td><td>5.0</td><td>98.9</td><td>99.0</td><td>99.5</td><td>31.8</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>99.1</td><td colspan="2">99.3</td><td>99.7</td><td>5.9</td><td>99.2</td><td>99.3</td><td>99.7</td><td>32.4</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>95.8</td><td colspan="2">96.9</td><td>98.8</td><td>41.8</td><td>96.4</td><td>97.4</td><td>98.9</td><td>59.8</td></tr><tr><td/><td>Baseline</td><td>78.9</td><td colspan="2">80.4</td><td>-</td><td>-</td><td>74.2</td><td>75.9</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>83.2</td><td colspan="2">85.4</td><td>-</td><td>-</td><td>79.0</td><td>81.5</td><td>-</td><td>-</td></tr><tr><td>LSUN</td><td>ODIN OE DPN sof t (β : 0.0, σ : 0.0)</td><td>87.8 90.6 91.6</td><td colspan="2">-91.6 92.6</td><td>--93.3</td><td>--92.6</td><td>84.6 86.5 88.3</td><td>-86.1 88.3</td><td>--87.3</td><td>--85.4</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.01)</td><td>98.9</td><td colspan="2">99.2</td><td>99.7</td><td>98.9</td><td>99.0</td><td>99.3</td><td>99.7</td><td>96.1</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.05)</td><td>96.0</td><td colspan="2">96.8</td><td>98.2</td><td>96.8</td><td>96.6</td><td>97.2</td><td>98.4</td><td>93.0</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>99.7</td><td colspan="2">99.7</td><td>99.9</td><td>1.4</td><td>99.6</td><td>99.7</td><td>99.8</td><td>30.8</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>99.7</td><td colspan="2">99.7</td><td>99.9</td><td>2.8</td><td>99.7</td><td>99.7</td><td>99.9</td><td>31.1</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>96.8</td><td colspan="2">97.5</td><td>99.1</td><td>37.1</td><td>97.3</td><td>98.0</td><td>99.2</td><td>56.2</td></tr><tr><td/><td>Baseline</td><td>76.6</td><td colspan="2">78.1</td><td>-</td><td>-</td><td>90.2</td><td>91.0</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>80.9</td><td colspan="2">83.1</td><td>-</td><td>-</td><td>92.2</td><td>93.3</td><td>-</td><td>-</td></tr><tr><td>Places365</td><td>ODIN OE DPN sof t (β : 0.0, σ : 0.0) DPN sof t (β : 0.0, σ : 0.01)</td><td>86.4 88.8 90.5 97.8</td><td colspan="2">-90.1 91.3 98.4</td><td>--92.6 99.3</td><td>--92.0 98.5</td><td>94.7 95.1 96.0 99.4</td><td>-95.0 96.0 99.6</td><td>-95.9 99.8</td><td>--94.9 98.6</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.05)</td><td>95.9</td><td colspan="2">96.7</td><td>98.1</td><td>96.7</td><td>98.9</td><td>99.1</td><td>99.5</td><td>97.6</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>99.3</td><td colspan="2">99.4</td><td>99.7</td><td>3.4</td><td>99.8</td><td>99.8</td><td>99.9</td><td>59.1</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>99.4</td><td colspan="2">99.5</td><td>99.8</td><td>4.8</td><td>99.8</td><td>99.9</td><td>99.9</td><td>60.3</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>96.0</td><td colspan="2">96.9</td><td>98.8</td><td>40.0</td><td>98.9</td><td>99.2</td><td>99.7</td><td>80.9</td></tr><tr><td/><td>Baseline</td><td>60.0</td><td colspan="2">60.2</td><td>-</td><td>-</td><td>43.4</td><td>43.4</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>64.0</td><td colspan="2">64.3</td><td>-</td><td>-</td><td>46.0</td><td>45.6</td><td>-</td><td>-</td></tr><tr><td>Textures</td><td>ODIN OE DPN sof t (β : 0.0, σ : 0.0) DPN sof t (β : 0.0, σ : 0.01)</td><td>63.4 73.6 80.5 84.3</td><td colspan="2">-74.8 81.8 86</td><td>--85.0 92.9</td><td>--83.7 90.9</td><td>48.9 58.3 66.6 81.3</td><td>-58.4 66.9 83.3</td><td>--71.3 90.9</td><td>--67.2 84.1</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.05)</td><td>89.2</td><td colspan="2">90.7</td><td>94.5</td><td>92.8</td><td>86.2</td><td>88.3</td><td>93.1</td><td>84.5</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>89.7</td><td colspan="2">91.2</td><td>96.2</td><td>10.6</td><td>86.1</td><td>87.8</td><td>95.1</td><td>22.4</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>92.2</td><td colspan="2">93.6</td><td>97.4</td><td>8.9</td><td>89.9</td><td>91.6</td><td>96.9</td><td>22.1</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>86.7</td><td colspan="2">88.9</td><td>96.6</td><td>29.6</td><td>83.7</td><td>86.8</td><td>96.1</td><td>35.1</td></tr><tr><td colspan="11">We train a neural network with 2 hidden layers with 50 nodes each and relu activation function. The network</td></tr><tr><td colspan="11">is trained for 2, 500 epochs using stochastic gradient descent (SGD) optimization with a constant learning rate</td></tr><tr><td>of 0.01.</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Results for misclassification detection for TinyImageNet.

Table content: <table><tr><td>D test in</td><td>Methods</td><td colspan="2">Max.P Ent.</td><td colspan="5">AUROC e zc(x  *  ) D-Ent Max.P Ent.</td><td colspan="2">AUPR e zc(x  *  ) D-Ent</td><td>Acc.(%)</td></tr><tr><td/><td>Baseline</td><td>85.4</td><td colspan="2">85.2</td><td>-</td><td>-</td><td>78.3</td><td>77.8</td><td>-</td><td>-</td><td>57.8</td></tr><tr><td>TinyImageNet</td><td>MCDP OE DPN sof t (β : 0.0, σ : 0.0) DPN sof t (β : 0.0, σ : 0.01) DPN sof t (β : 0.0, σ : 0.05) DPN sof t (β : 0.5, σ : 0.0)</td><td>85.8 85.2 85.6 85.7 85.4 85.9</td><td colspan="2">85.8 85.4 85.6 85.8 85.6 86.3</td><td>--81.7 81.8 81.8 81.8</td><td>--80.0 80.2 80.3 77.6</td><td>77.9 77.8 78.8 79.0 78.4 79.7</td><td>77.9 77.9 78.6 78.7 78.5 80.1</td><td>--73.5 73.6 73.6 73.6</td><td>--73.5 73.6 73.6 72.7</td><td>59.0 57.7 57.8 57.8 57.8 57.5</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>85.8</td><td colspan="2">86.2</td><td>81.9</td><td>77.5</td><td>79.5</td><td>80.1</td><td>73.7</td><td>72.8</td><td>57.5</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>85.7</td><td colspan="2">86.1</td><td>81.8</td><td>77.9</td><td>79.2</td><td>80.0</td><td>73.7</td><td>73.1</td><td>57.6</td></tr></table>

Results of OOD image detection for TinyImageNet classifiers. Description of these OOD datasets are provided in Appendix C.2. growth rate = 12 and CIFAR-100 as the in-domain and CIFAR-10 as the OOD training data

Table content: <table><tr><td>D test out</td><td>Methods</td><td colspan="2">Max.P Ent.</td><td colspan="5">AUROC e zc(x  *  ) D-Ent Max.P Ent.</td><td colspan="2">AUPR e zc(x  *  ) D-Ent</td></tr><tr><td/><td>Baseline</td><td>62.9</td><td colspan="2">61.7</td><td>-</td><td>-</td><td>51.4</td><td>50.0</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>63.8</td><td colspan="2">63.6</td><td>-</td><td>-</td><td>52.2</td><td>51.8</td><td>-</td><td>-</td></tr><tr><td>STL-10</td><td>OE DPN sof t (β : 0.0, σ : 0.0) DPN sof t (β : 0.0, σ : 0.01) DPN sof t (β : 0.0, σ : 0.05)</td><td>98.9 98.4 98.1 97.0</td><td colspan="2">99.0 98.5 98.5 98.1</td><td>-98.9 98.6 97.9</td><td>-98.8 98.6 97.9</td><td>99.0 98.6 98.4 97.5</td><td>99.1 98.8 98.6 97.7</td><td>-99.1 98.9 98.4</td><td>-99.0 98.8 98.3</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>98.6</td><td colspan="2">98.7</td><td>99.0</td><td>4.3</td><td>98.8</td><td>98.9</td><td>99.2</td><td>28.3</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>98.3</td><td colspan="2">98.4</td><td>98.8</td><td>5.2</td><td>98.5</td><td>98.7</td><td>99.0</td><td>28.8</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>97.5</td><td colspan="2">97.7</td><td>98.4</td><td>7.6</td><td>97.8</td><td>98.0</td><td>98.6</td><td>30.3</td></tr><tr><td/><td>Baseline</td><td>64.9</td><td colspan="2">65.6</td><td>-</td><td>-</td><td>59.7</td><td>59.5</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>65.9</td><td colspan="2">66.1</td><td>-</td><td>-</td><td>60.3</td><td>60.2</td><td>-</td><td>-</td></tr><tr><td>LSUN</td><td>OE DPN sof t (β : 0.0, σ : 0.0) DPN sof t (β : 0.0, σ : 0.01)</td><td>98.2 97.8 97.3</td><td colspan="2">98.5 98.2 97.7</td><td>-98.9 98.7</td><td>-98.8 98.6</td><td>98.7 98.4 98.1</td><td>99.0 98.7 98.4</td><td>-99.2 99.0</td><td>-99.2 99.0</td></tr><tr><td/><td>DPN sof t (β : 0.0, σ : 0.05)</td><td>95.2</td><td colspan="2">95.9</td><td>97.7</td><td>97.5</td><td>96.5</td><td>97.0</td><td>98.3</td><td>98.1</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>98.2</td><td colspan="2">97.6</td><td>99.1</td><td>6.8</td><td>99.1</td><td>98.9</td><td>99.4</td><td>35.0</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>97.6</td><td colspan="2">98.0</td><td>98.8</td><td>9.1</td><td>98.3</td><td>98.6</td><td>99.1</td><td>36.8</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>96.1</td><td colspan="2">96.6</td><td>98.0</td><td>14.4</td><td>97.1</td><td>97.1</td><td>98.6</td><td>40.8</td></tr><tr><td/><td>Baseline</td><td>67.8</td><td colspan="2">68.9</td><td>-</td><td>-</td><td>85.8</td><td>86.0</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>68.5</td><td colspan="2">68.9</td><td>-</td><td>-</td><td>86.1</td><td>86.2</td><td>-</td><td>-</td></tr><tr><td>Places365</td><td>OE DPN sof t (β : 0.0, σ : 0.0) DPN sof t (β : 0.0, σ : 0.01) DPN sof t (β : 0.0, σ : 0.05)</td><td>95.8 95.0 94.1 90.5</td><td colspan="2">96.5 95.8 94.9 91.8</td><td>-97.3 96.7 94.4</td><td>-97.1 96.5 94.1</td><td>98.9 98.7 98.5 97.4</td><td>99.1 98.9 98.7 97.4</td><td>-99.3 99.1 98.5</td><td>-99.3 99.1 98.5</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>95.6</td><td colspan="2">96.2</td><td>97.6</td><td>17.0</td><td>98.9</td><td>99.0</td><td>99.4</td><td>69.6</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>94.4</td><td colspan="2">95.1</td><td>96.8</td><td>21.3</td><td>98.6</td><td>98.7</td><td>99.2</td><td>72.1</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>91.6</td><td colspan="2">92.6</td><td>95.0</td><td>30.8</td><td>97.8</td><td>98.0</td><td>98.7</td><td>77.0</td></tr><tr><td/><td>Baseline</td><td>68.8</td><td colspan="2">70.9</td><td>-</td><td>-</td><td>51.0</td><td>53.3</td><td>-</td><td>-</td></tr><tr><td/><td>MCDP</td><td>69.1</td><td colspan="2">69.7</td><td>-</td><td>-</td><td>51.0</td><td>51.6</td><td>-</td><td>-</td></tr><tr><td>Textures</td><td>OE DPN sof t (β : 0.0, σ : 0.0) DPN sof t (β : 0.0, σ : 0.01) DPN sof t (β : 0.0, σ : 0.05)</td><td>87.5 85.1 83.9 81.4</td><td colspan="2">89.1 86.8 85.8 83.5</td><td>-88.4 87.6 85.6</td><td>-89.0 88.2 86.2</td><td>85.6 82.6 81.1 77.3</td><td>87.5 84.7 83.3 79.9</td><td>-86.0 85.0 81.9</td><td>-86.6 85.6 82.6</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.0)</td><td>85.7</td><td colspan="2">87.0</td><td>88.1</td><td>45.0</td><td>83.3</td><td>84.8</td><td>85.8</td><td>46.6</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.01)</td><td>84.5</td><td colspan="2">85.8</td><td>87.1</td><td>47.5</td><td>81.6</td><td>83.3</td><td>84.4</td><td>48.4</td></tr><tr><td/><td>DPN sof t (β : 0.5, σ : 0.05)</td><td>82.3</td><td colspan="2">83.8</td><td>85.3</td><td>53.0</td><td>78.5</td><td>80.4</td><td>81.8</td><td>52.0</td></tr><tr><td colspan="2">depth = 55,</td><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
This work studies the predictive uncertainty issue of deep learning models. In particular, this work focuses on the distributional uncertainty which is caused by distributional mismatch between training and test examples. The proposed method is developed based on the existing work called Dirichlet Prior Network (DPN). It aims to address the issue of DPN that its loss function is complicated and makes the optimization difficult. Instead, this paper proposes a new loss function for DPN, which consists of the commonly used cross-entropy loss term and a regularization term. Two loss functions are respectively defined over in-domain training examples and out-of-distribution (OOD) training examples. The final objective function is a weighted combination of the two loss functions. Experimental study is conducted on one synthetic dataset and two image datasets (CIFAR-10 and CIFAR-100) to demonstrate the properties of the proposed method and compare its performance with the relevant ones in the literature. The issue researched in this work is of significance because understanding the predictive uncertainty of a deep learning model has its both theoretical and practical value. The motivation, research issues and the proposed method are overall clearly presented. The current recommendation is Weak Reject because the experimental study is not convincing or comprehensive enough. 1. Although the goal of this work is to deal with the inefficiency issue of the objective function of existing DPN with the newly proposed one, this experimental study does not seem to conduct sufficient experiments to demonstrate the advantages (say, in terms of training efficiency & the capability in making the network scalable for more challenging dataset) of the proposed objective function over the existing one; 2. Table 1 compares the proposed method with ODIN. However, as indicated in this work, ODIN is trained with in-domain examples only. Is this comparison fair? Actually, ODIN*s setting seems to be more practical and more challenging than the setting used by the propose methods. 3. The evaluation criteria shall be better explained at the beginning of the experiment, especially how they can be collectively used to verify that the proposed method can better distinguish distributional uncertainty from other uncertainty types. 4. In addition, the experimental study can be clearer on the training and test splits. How many samples from CIFAR-10 and CIFAR-100 are used for training and test purpose, respectively? Also, since training examples are from CIFAR-10 and CIFAR-100 and the test examples are also from these two datasets, does this contradict with the motivation of “distributional mismatch between training and test examples” mentioned in the abstract? 5. The experimental study can have more comparison on challenging datasets with more classes since it is indicated that DPN has difficulty in dealing with a large number of classes. Minor: 1. Please define the hat	heta in Eq.(5). Also, is the dirac delta estimation a good enough approximation here? 2. The lambda_{out} < lambda_{in} in Eq.(11) needs to be better explained. In particular, are the first terms in Eq.(10) and Eq.(11) comparable in terms of magnitude? Otherwise, lambda_{out} < lambda_{in} may not make sense. 3. The novelty and significance of fine-tuning the proposed model with noisy OOD training images can be better justified.