Title: PRIOR NETWORKS FOR DETECTION OF ADVERSARIAL ATTACKS Anonymous

Abstract: Adversarial examples are considered a serious issue for safety critical applications of AI, such as finance, autonomous vehicle control and medicinal applications. Though significant work has resulted in increased robustness of systems to these attacks, systems are still vulnerable to well-crafted attacks. To address this problem several adversarial attack detection methods have been proposed. However, system can still be vulnerable to adversarial samples that are designed to specifically evade these detection methods. One recent detection scheme that has shown good performance is based on uncertainty estimates derived from Monte-Carlo dropout ensembles. Prior Networks, a new method of estimating predictive uncertainty, have been shown to outperform Monte-Carlo dropout on a range of tasks. One of the advantages of this approach is that the behaviour of a Prior Network can be explicitly tuned to, for example, predict high uncertainty in regions where there are no training data samples. In this work Prior Networks are applied to adversarial attack detection using measures of uncertainty in a similar fashion to Monte-Carlo Dropout. Detection based on measures of uncertainty derived from DNNs and Monte-Carlo dropout ensembles are used as a baseline. Prior Networks are shown to significantly out-perform these baseline approaches over a range of adversarial attacks in both detection of whitebox and blackbox configurations. Even when the adversarial attacks are constructed with full knowledge of the detection mechanism, it is shown to be highly challenging to successfully generate an adversarial sample.

1. INTRODUCTION:

Neural Networks (NNs) have become the dominant approach to addressing computer vision (CV) (Girshick, 2015; Simonyan & Zisserman, 2015; Villegas et al., 2017) , natural language processing (NLP) (Mikolov et al., 2013b; a; 2010) , speech recognition (ASR) (Hinton et al., 2012; Hannun et al., 2014) and bio-informatics (Caruana et al., 2015; Alipanahi et al., 2015) tasks. However, as observed by (Szegedy et al., 2013) , they are susceptible to adversarial attacks -small perturbations to the input which are almost imperceptible to humans, yet which drastically affect the predictions of the neural network. It was found that adversarial attacks have several properties which make them a serious security concern. Firstly, adversarial attacks are transferable -an adversarial attack computed on one network may be able to successfully attack a different network (Szegedy et al., 2013; Goodfellow et al., 2015) . Secondly, there exists a plethora of adversarial attacks which are quite easy to construct (Goodfellow et al., 2015; Kurakin et al., 2016; Dong et al., 2018; Carlini & Wagner, 2016; Papernot et al., 2016a; b; Liu et al., 2016) . Thirdly, it is possible to craft adversarial attacks within the physical world, such as putting stickers on paper (Kurakin et al., 2016) . To compound the issue, it was found that adversarial attacks are hard to defend against (Carlini & Wagner, 2017) . Specifically, it was found that while adversarial training (Szegedy et al., 2013) and adversarial distillation (Papernot et al., 2016c) are able to improve the robustness of a network, it is still possible to craft successful adversarial attacks against these networks. Indeed, currently, there are far more methods of successfully attacking networks than there are of defending networks (Carlini & Wagner, 2017; 2016) . Altogether, this raises serious concerns about how safe it is to deploy neural networks for high-stakes applications, such as autonomous vehicle control, financial and medical applications.

While much work focuses on constructing neural networks which are robust to adversarial attacks, (Carlini & Wagner, 2017) investigate detection of adversarial attack and shows that adversarial attacks can be detectable using a range of approaches. However, it turns out that attacks can then be crafted to fool the proposed detection schemes. However, (Carlini & Wagner, 2017) singles out detection of adversarial attacks using uncertainty measures derived from Monte-Carlo dropout as being successful. Detection of adversarial attack using Monte-Carlo dropout was further investigated in (Smith & Gal, 2018) . Work by (Smith & Gal, 2018) suggests that adversarially modified inputs can be interpreted as inputs which lie off the manifold of natural images. In other words, adversarial samples can be seen as a special type of out-of-domain or out-of-distribution input which can be detected by measures of uncertainty derived via Monte-Carlo Dropout.

Recently, (Malinin & Gales, 2018) proposed Prior Networks -a new approach to modelling uncertainty which has been shown to outperform Monte-Carlo dropout on a range of tasks. Prior Network parameterize a distribution over output distributions which allows them to separately model data uncertainty and distributional uncertainty. Furthermore, they are explicitly taught to learn decision boundaries between an in-domain and an out-of-domain region, unlike Monte-Carlo dropout, where it is not possible to specify the behaviour of measures of uncertainty in different regions of the input space explicitly. Finally, Prior Networks allow measures of uncertainty to be obtained analytically, without the need for expensive Monte-Carlo sampling. Thus, this work investigates the detection of adversarial attacks using measures of uncertainty derived from Prior Networks. The contributions of this work are as follows:

• Measures for assessing the success of adversarial attacks in the context of detection are proposed.

• It is shown that whitebox and blackbox adversarial attacks with no knowledge of the detection scheme can be successfully detected based on measures of uncertainty.

• It is shown that whitebox detection evading adversarial attacks are difficult to construct for Prior Networks, and detection evading attack fail entirely against Prior Networks.

2. UNCERTAINTY FOR DEEP LEARNING:

In (Smith & Gal, 2018) adversarially modified inputs are interpreted as inputs which lie off the manifold of natural images -the stronger the adversarial perturbation, the further is the input from the manifold. Thus, adversarial samples can be seen as out-of-distribution inputs. 

P(y|x * , D) = P(y|x * , θ)p(θ|D)dθ (1)

Unfortunately, both the integral in eq. 1 and calculation of the model posterior are intractable for neural networks. Typically the model posterior distribution is approximated using either an implicit or explicit variational approximation q(θ) and the integral is approximated via sampling (eq. 2), using approaches such as Monte-Carlo dropout (Gal & Ghahramani, 2016) :

EQUATION

Bayesian approaches model distributional uncertainty through model uncertainty. By selecting an appropriate approximate inference scheme and model prior p(θ) Bayesian approaches aim to craft a model posterior p(θ|D) such that the ensemble {P(ω c |x * , θ (i) )} M i=1 is consistent in-domain and becomes increasingly diverse the further away x * is from the region of training data. The entropy of the expected distribution P(ω c |x * , D) will indicate the total uncertainty in predictions. Measures of the diversity of the ensemble, such as Mutual Information, assess uncertainty in predictions due to model uncertainty, which yields distributional uncertainty.

MI[y, θ|x * , D] M odel U ncertainty = H[E p(θ|D) [P(y|x * , θ)]] T otal U ncertainty -E p(θ|D) [H[P(y|x * , θ)]] Expected Data U ncertainty (3)

In practice, however, for deep, distributed models with tens of million parameters, such as DNNs, it is difficult to select an appropriate approximate inference scheme to craft a model posterior which induces a distribution over distributions with the desired properties.

2.2. UNCERTAINTY ESTIMATION VIA PRIOR NETWORKS:

Unlike Bayesian approaches, which indirectly specify a conditional distribution over output distributions, a Prior Network p(π|x * ; θ), proposed by (Malinin & Gales, 2018) directly parametrizes a prior distribution over categorical output distributions. In this work the Dirichlet distribution (eqn 5) is chosen due to its tractable analytic properties.

EQUATION

A Dirichlet distribution is parameterized by its concentration parameters α, where α 0 , the sum of all α c , is called the precision of the Dirichlet distribution. Higher values of α 0 lead to sharper, more confident distributions:

Dir(π|α) = Γ(α 0 ) K c Γ(α c ) K c=1 π αc-1 c , α c > 0, α 0 = K c α c (5)

The predictive distribution is given by the expected categorical distribution under the Dirichlet prior: A Prior Network is trained to display these behaviors by minimizing the following multi-task loss functions, as described in (Malinin & Gales, 2018) . A flat Dirichlet is chosen as the out-of-distribution target distribution p out (π), as it is the maximum entropy Dirichlet distribution. The target in-domain distribution p in (π|x) is constructed by specifying the mean and precision of the Dirichlet, as described in (Malinin & Gales, 2018) . In order to train with this loss function, it is necessary to have out-of-distribution training data. One way of doing this is to chose, in addition to the target training data, an 'out-of-distribution' dataset. For example, if a model is trained on CIFAR-10 ( Krizhevsky, 2009) , it is possible to use CIFAR-100 as the out-of-distribution dataset, as they don't have overlapping classes.

EQUATION

EQUATION

Given a trained Prior Network it is possible to calculate the Mutual Information using an expression similar to equation 3:

MI[y, π|x * ; θ] Distributional U ncertainty = H[E p(π|x * ; θ) [P(y|π)]] T otal U ncertainty -E p(π|x * ; θ) [H[P(y|π)]] Expected Data U ncertainty (8)

This expression, as in the Bayesian case, allows total uncertainty, given by the entropy of the expected distribution, to be decomposed into data uncertainty and distributional uncertainty

3. MEASURES OF PERFORMANCE:

In order to investigate detection of adversarial attacks, it is necessary to discuss how to assess the effectiveness of an adversarial attack in a scenario where detection of the attack is possible. Previous work on detection of adversarial examples (Gong et al., 2017; Grosse et al., 2017; Metzen et al., 2017; Carlini & Wagner, 2017; Smith & Gal, 2018) assesses the performance of detection methods separately from whether an adversarial attack was successful, and use the standard measures of adversarial success and detection performance. However, in a real deployment scenario, an attack can only be considered successful if it both affects the predictions and evades detection. In this section we develop a measure of performance to assess this.

An adversarial input x adv will be defined as the output of an adversarial attack generation process A adv applied to a natural input x:

EQUATION

This process is typically an optimization problem which tries to minimize a loss L(•), which is a function of the input x, the model P(y|x; θ) and a target class t, with respect to the input while also minimizing some distance δ(•, •) between the original image and the generated adversarial perturbation. This loss is typically the negative log-likelihood of a particular target class, in case of targeted adversarial attacks, or the (positive) log-likelihood of the predicted class, in case of an untargeted attack:

L tgt P(y|x; θ), t = -ln P(ω t |x; θ)

EQUATION

The distance δ(•, •), typically the L 1 , L 2 or L ∞ norm, is minimized so that the adversarial attack is still perceived to be a natural input to a human observer. The best adversarial attack is one which minimizes the chosen loss and has the minimal deviation from the original image x. There are multiple ways in which this optimization problem can be solved (Szegedy et al., 2013; Goodfellow et al., 2015; Kurakin et al., 2016; Dong et al., 2018) .

While the process A adv will always generate some kind of perturbation, it will not always yield a successful attack, as it may fail to affect the prediction. For the purposes of this discussion the adversarial generation process A adv will be defined to either yield a successful adversarial attack x adv or an empty set ∅.

A adv (x, t) ∈ {x adv , ∅} (11) It is necessary to point out that here we take the perspective of the attacker -a successful attack, in the case of targeted attacks, is one which yields the target class. However, consider the perspective of the system designer who does not know whether the attacker attempts a targeted attack or not. From the point of view of the system designer, any attack which is able to affect the predictions of the network may be considered successful, regardless of whether the attack produced the attacker's desired prediction. In the case of non-targeted attacks, however, the attacker's and defender's definition of success are the same.

In a standard scenario, where there is no detection, the efficacy of an adversarial attack on a model1 can be summarized via the success rate S of the attack:

EQUATION

)

Typically S is plotted against the total maximum perturbation from the original image, measured as either the L 1 , L 2 or L ∞ distance from the original image.

Now consider using a threshold-based detection scheme where a sample is labelled 'positive' if some measure of uncertainty, such as entropy H(x), is less than a threshold T and 'negative' if it is higher than a threshold:

EQUATION

The performance of such a scheme can be evaluated at every threshold value using the true positive rate t p (T ) and the false positive rate f p (T ):

EQUATION

The whole range of such trade offs can be visualized using a Receiver-Operating-Characteristic (ROC) and the quality of the trade-off can be summarized using area under the ROC curve.

However, a standard ROC curve does not give credit to the system for being robust to adversarial attacks -it doesn't account for situations where the process A adv (•) fails to produce a successful attack. In fact, if an adversarial attack is made against a system which has a detection scheme, it can only be considered successful if it both affects the predictions and evades detection. This condition can be summarized in the following indicator function:

ÎT (x) =    1, T > H(x) 0, T ≤ H(x) 0, x = ∅ (15)

Given this indicator function, a new false positive rate fP (T ) can be defined as:

EQUATION

This false positive rate can now be seen as a new Joint Success Rate which measures how many attacks were both successfully generated and evaded detection, given the threshold of the detection scheme. The Joint Success Rate can be plotted against the standard true positive rate on an ROC curve to visualize the possible trade-offs. This ROC curve will now give credit to the system when the attacker fails to generate a succesful attack. One possible operating point is where the false positive rate is equal to the false negative rate, also known as the Equal Error-Rate point:

EQUATION

Throughout the remainder of this work the EER false positive rate will be quoted as the Joint Success Rate.

4. DETECTING ADVERSARIAL ATTACKS:

The previous sections discussed how to obtain estimates of uncertainty using several different approaches and how to assess the performance of adversarial attacks in the scenario where a detection scheme is in place. In this section detection of adversarial samples using measures of uncertainty is investigated. Four threat models are evaluated:

• Whitebox attack with no knowledge of the detection scheme;

• Blackbox attack with no knowledge of the detection scheme;

• Whitebox attack with complete information about the detection scheme;

• Blackbox attack with complete information about the detection scheme.

Experiments were conducted on the CIFAR-10 ( Krizhevsky, 2009) dataset. Four models were constructed: standard classification network DNN, Monte-Carlo dropout (MCDP) ensemble derived from the same DNN, a 'standard' prior network 'PN' and an 'adversarially trained' prior network 'PN-ADV'. DNN and MCDP were used as baselines in this work. The baseline prior network PN was trained on CIFAR-10 as in-domain data and CIFAR-100 as out-of-distribution data in exactly the same setup as (Malinin & Gales, 2018) . The adversarially trained Prior Network PN-ADV was additionally trained on un-targeted Fast Gradient Sign Method (Goodfellow et al., 2015) attacks which were dynamically generated during training. The perturbation values were randomly sampled from a normal distribution for each mini-batch. Standard VGG-16 (Simonyan & Zisserman, 2015) architecture was used for all networks. Entropy of the predictive distribution was used as the measure of uncertainty of a DNN and Mutual Information was used as the measure of uncertainty for MCDP (eqn 3) and Prior Networks (eqn 8). For each model type, 5 model-instantiations with different random seeds were trained.

Three types of non-targeted adversarial attacks are considered: Fast-Gradient Sign Method (FGSM) attacks (Szegedy et al., 2013) , Basic-Iterative Method (BIM) Attacks (Kurakin et al., 2016) and Momentum-Iterative Method (MIM) (Dong et al., 2018) attacks. These attacks are chosen because they were the ones evaluated in (Smith & Gal, 2018) and MIM attacks are considered to be quite strong. The L ∞ norm version of the FGSM, BIM and MIM attacks were investigated. Each of the iterative attacks was run for 10 iterations of gradient descent with their 'default' settings , as described in (Kurakin et al., 2016; Dong et al., 2018) . Further details on the training and construction of all models can be found in appendix A.

4.1. ATTACKS WITH NO KNOWLEDGE ABOUT THE DETECTION SCHEME:

Figure 2 displays plots of the Joint Success Rate (eqn 16) at the EER threshold against the perturbation for all attacks averaged across different seed values with ±σ bounds.

Figures 2a-c show that a deterministic DNN and MC Dropout achieve nearly identical performance across all attacks. Curiously, they are both able to detect FGSM attacks, but are almost completely ineffective against BIM and MIM attacks, which achieve joint success rates in excess of 90%2 for small values of epsilon. The standard prior network PN is robust FGSM attacks and is more robust against BIM and MIM attacks than the baselines -the highest success rate of BIM and MIM attack is now 80% at a higher epsilon value of 30. The adversarially trained prior network PN-ADV achieves the best performance by a large margin across all attacks. It is totally robust to FGSM attacks and significantly more robust to BIM and MIM attacks, which now achieve a peak joint success rate of 15% in a narrow range of perturbations.

Performance against black-box attacks is described in figures 2d-f. Whitebox attacks against each random seed are used as blackbox attacks against all the other seeds for a model type. All models, even the DNN and MCDN, are able to achieve very low joint success rates and are robust to all attacks considered. Curiously, a standard Prior Network PN has the worst performance. However, an adversarially trained prior network achieves by far the best performance against all considered black-box adversarial attacks.

These results indicate that a standard prior network learns a better decision boundary between the in-domain and out-of-domain regions than Monte-Carlo dropout and is able to yield better measures of uncertainty. However, as it was trained using only 'on-manifold' out-of-distribution data, it still is susceptible to adversarial attacks which are far from the manifold. Training a prior network on FGSM data allows it to learn a better decision boundary, allowing the prior network to generalize the out-of-domain region beyond the manifold of natural data. Interestingly, the results suggest that the 'holes' in a model's understanding of where the in-domain/out-of-domain region discovered using whitebox attacks fail to generalize as blackbox attacks.

4.2. ATTACKS WITH COMPLETE KNOWLEDGE ABOUT DETECTION SCHEME:

The results of section 4.1 show that prior networks are able to detect both whitebox and blackbox adversarial attacks successfully. However, as discussed in (Carlini & Wagner, 2017) , it is necessary to evaluate whether is is possible to construct adversarial attacks specifically to evade detection. This section investigate whether is is possible craft adversarial attacks which avoid detection by measures of uncertainty derived from DNNs, Monte-Carlo Dropout and Prior Networks.

Conceptually, the best approach to avoiding detection using measures of uncertainty is to generate an adversarial attack which changes a model's prediction while leaving the measures of uncertainty (entropy, mutual information) unchanged. In the case of a DNN or Monte-Carlo dropout, the simplest approach to do this is to simply permute the predicted distribution over classes so that the probability of the max class is assigned to the target class, and the probability of the target class is assigned to the max class. The loss function minimized by the adversarial generation process will be the KL divergence between the predicted distribution over class labels P(y|x; θ) and the target permuted distribution P t (y): A adv applied to a natural input x:

EQUATION

For prior networks the equivalent approach would be to permute the values of α and to minimize KL divergence to the permuted target Dirichlet distribution:

EQUATION

In the following experiments, MIM adversarial attacks are generated using the loss functions given by equations 18 and 19 for DNN/MCDP and Prior Network models, respectively. The iterative approaches are run for a range of iterations, ranging from 10 to 100. The second most likely class is selected as the target class, as this should be a more aggressive attack than switching to the least likely class. Figure 3 shows the Joint Success Rate, ROC AUC and success rate S (eqn. 12) vs the number of iterations for MIM attack generation at a perturbation of 40. The change of x-axis from perturbation to iteration is used to illustrate the added computational difficulty in generating detection evading attacks. The figures show that a detection evading whitebox attack against a DNN is completely successful.

Figure 3c shows that in almost 100% of cases the attack yields the target class. Monte-Carlo dropout performs better in terms of success rate and AUC. However, whitebox detection evading attacks against PN and PN-adv were far less successful, yielding the target class 45% and 40% of the time after far more iterations, respectively. Furthermore, AUC only fell by about 10% absolute for prior network models, and and only after many iterations. Interestingly, blackbox detection evading attacks failed altogether and were detectable using all considered models. This suggests that it is very difficult to generalize the weaknesses a detection evading attack discovers in a model's detection scheme.

The experiments in this section show that it is non-trivial to successfully construct whitebox adversarial attacks which are able to evade detection by measures of uncertainty for appropriately secured prior networks. This suggests that the optimization problem in equation 9 becomes more difficult to solve for Prior Networks, which, in turn, suggests that the manifold hypothesis of adversarial attacks is correct. Specification of the behaviour of the network in-domain and out-of-domain on-manifold as well as out-of-domain off-manifold greatly constrains the space of solutions to equation 9 where the attack both yields the target class and avoids changing properties of predicted distribution over distributions. Using measures of uncertainty in the prediction, uncertainty in the model's understanding of the data, allows the space of solutions to the adversarial optimization problem to be constrained in a way which methods proposed in (Metzen et al., 2017; Gong et al., 2017; Grosse et al., 2017) do not.

5. CONCLUSION:

This work shows that it is possible to detect FGSM, BIM and MIM adversarial attacks using measures of uncertainty derived from DNNs, MC Dropout and Prior Networks. Prior Network models are the most robust to FGSM, BIM and MIM L ∞ adversarial attacks and outperform all the other methods by a large margin in both detection of whitebox and blackbox adversarial attacks. In section 4.2 it is shown that it is possible to construct targeted adversarial attacks which also avoid detection by directly attacking the measure of uncertainty derived from DNNs and Monte-Carlo dropout ensembles. However, results in section 4.2 show that it difficult to construct detection avoiding attacks for adversarially trained prior networks. The results in this work are encouraging and show that is appropriate measures of uncertainty are used, then adversarial attacks are not a great security concern. However, further work in evaluating the robustness of uncertainty estimates derived from Prior Networks and Monte-Carlo dropout against stronger detection-evading adversarial attacks must be done. 

:

Given an evaluation dataset Dtest = {xi, yi} N i=1

Note -a joint success rate larger than 50% means that the AUC is less than 0.5 and can be improved by using a reversed detection scheme

P(ωc|x; θ) is a shorthand for P(y = ωc|x; θ)

Figures and Tables:

6) The desired behaviors of the Prior Network can be visualized on a simplex (fig 1), where figure 1:a describes confident behavior (low-entropy prior focused on low-entropy output distributions), figure 1:b describes uncertainty due severe class overlap and figure 1:c describes the behaviour for an out-of-distribution input.

Figure 1: Desired Behaviors of a Dirichlet distribution over categorical distributions.

Figure 2: Plots of joint success rate at EER threshold for whitebox (a-c) and blackbox (d-f) attacks. Note difference in y-axis scale of plots d-f.

Figure 3: Summaries of performance against whitebox and blackbox detection evading MIM attack. Note different y-axis scale of plot d.

Figure 4: Plots of Joint Success Rates, ROC AUC and Success rates against perturbation ( ) for whitebox FGSM, BIM and MIM attacks.

Figure 7: Plots of uncertainty distributions for real (in-domain) and adversarial (out-of-domain) data for MIM attack at iteration 100

Peer review (iclr2019):
 This paper proposes a new detection method for adversarial examples, based on a prior network, which gives an uncertainty estimate for the network*s predictions. The idea is interesting and the writing is clear. However, I have several major concerns. A major one of these is that the paper considers "detection of adversarial attacks" to mean detecting adaptive and non-adaptive attacks, while the latter are (a) unrealistic and (b) a heavily explored problem, with solutions ranging from clustering activations, to denoising the image via projection (in particular, once can use any of the circumvented ICLR 2018 defenses which all work in the non-adaptive sense, and check the prediction of the denoised image vs the original). Thus, the paper should focus on the regime of adaptive attacks. Within this regime: Motivation: - This work seems to suggest that dropout-based detection mechanisms are particularly successful. While Carlini & Wagner finds that the required distortion (using a specific attack) increases with randomization, the detection methods which used dropout were still completely circumvented in this paper. - The claim that adversarial examples are "points off of the data manifold" is relatively unmotivated, and is not really justified. Justification for this point is needed, as it forms the entire justification for using Prior Networks. - Detecting adversarial examples is not the same problem to detecting out-of-distribution samples, and the writing of the paper should be changed to reflect this more. Evaluation: - 100 iterations is not nearly enough for a randomization-based or gradient masking defense, so the attacks should be run for much longer. In particular, some of the success rate lines appear to be growing at iteration 100. - There is no comparison to any other method (in particular, just doing robust prediction via Madry et al or something similar); this should be added to contextualize the work. - The term "black-box" attacks can take on many meanings/threat models. The threat models in the paper need to be more well-defined, and in particular "black-box attacks" should be more accurately defined. If black-box attack refers to query-based attacks, the success rate should be equal to those of white-box attacks (or very close to it), as then the attack can just estimate the gradient through the classifier via queries. - The fact that the attacks do not reach 100% on the unprotected classifier is concerning, and illustrates the need for stronger attacks. Smaller comments: Page 1: Abstract: Line 4: missing , at the end of the line Page 1: Abstract: Line 5: “However, system can“ missing a before system Page 1: Abstract: Line 10: “have been shown” should be “has” instead of “have” Page 1: Abstract: Line 13: “In this work” missing a , after Page 1: Last paragraph: Line 2: “investigate” missing an s and should be “investigates’ Page 2: Section 2: Line 5: “in other words” missing a , after Page 2: Section 2: Line 8: “In order to capture distributional uncertainty” missing a , after Page 2: Last paragraph: Line 2: “Typically” missing a , after Page 3: Paragraph 2: Line 1: “In practice, however, for deep,” no need for the last , Page 3: Section 2.2: paragraph 1: Line 2: “a Prior Network p(π|x∗; θˆ), “ no need for the last , Page 3: Section 2.2: paragraph 1: Line 3: “In this work“ missing a , after Page 3: second last paragraph: Line 1: refers to figure as fig and figure (not consistent) Page 3: second last paragraph: Line 3: “uncertainty due severe class“ missing “to” before “severe” Page 4: Paragraph 1: Line 2: “to chose” should be “to choose” Page 4: Paragraph 2: Line 1: “Given a trained Prior Network“ missing a , after Page 4: Paragraph 2: two extra , Page 6: paragraph 1: Last line: 5 needs to be written in words and same for 10 in the next paragraph Page 7: section 4.2: paragraph 3: “For prior networks” need a , after Page 8: Paragraph 1: Line 6: “and and” Page 8: Conclusion: Line 4: “In section 4.2” needs a , after Page 8: Conclusion: Line 7: “it difficult” missing “is” Page 8: Conclusion: Line 9: “is appropriate” should be “if” instead of “is”