Regression Prior Networks
Authors: Andrey Malinin
Sergey Chervontsev
Ivan Provilkov
Mark Gales
Venue:  (2021)
Keywords: uncertainty
prior networks
regression
ensemble distribution distillation
depth estimation.

Abstract: Prior Networks are a recently developed class of models which yield interpretable measures of uncertainty and have been shown to outperform state-of-the-art ensemble approaches on a range of tasks. They can also be used to distill an ensemble of models via \emph{Ensemble Distribution Distillation} (EnD$^2$), such that its accuracy, calibration and uncertainty estimates are retained within a single model. However, Prior Networks have so far been developed only for classification tasks. This work extends Prior Networks and EnD$^2$ to regression tasks by considering the Normal-Wishart distribution. The properties of Regression Prior Networks are demonstrated on synthetic data, selected UCI datasets and a monocular depth estimation task, where they yield performance competitive with ensemble approaches.

Under review as a conference paper at ICLR 2021
REGRESSION PRIOR NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Prior Networks a class of models which yield interpretable measures of uncer- tainty and have been shown to outperform state-of-the-ar ensemble approaches on a range of tasks. They can also be used to distill an ensemble of models via Ensemble Distribution Distillation (EnD²), such that its accuracy, calibration, and uncertainty estimates are retained within a single model. However, Prior Networks have so far been developed only for classification tasks. This work extends Prior Networks and EnD² to regression tasks by considering the Normal- Wishart distri- bution. The properties of Regression Prior Networks are demonstrated on synthetic data, selected UCI datasets, and two monocular depth estimation tasks. They yield performance competitive with ensemble approaches.
INTRODUCTION
Neural Networks have become the standard approach to addressing a wide range of machine learning
tasks Girshick 2015 Simonyan & Zisserman 2015 Villegas et al. 2017 Mikolov et al. 2013b
2010 Hinton et al. 2012 Hannun et al. [2014] Caruana et al. 2015 Alipanahi et 2015). However, in order to improve the safety of AI systems (Amodei et al. 2016 and avoid costly mistakes in high-risk applications, such as self-driving cars, it is desirable for models to yield estimates of uncertainty in their predictions. Ensemble methods are known yield both improved predictive performance and robust uncertainty estimates (Gal & Ghahramani] 2016 Lakshminarayanan et al. 2017 Maddox et al. 2019). Importantly, ensemble approaches allow interpretable measures of uncertainty to be derived via a mathematically consistent probabilistic framework. Specifically, the overall total uncertainty can be decomposed into data uncertainty, or uncertainty due inherent noise in the data, and knowledge uncertainty, which is due to the model having limited uncertainty of the test data (Malinin) 2019). Uncertainty estimates derived from ensembles have been applied to the detection of misclassifications, out-of-domain inputs and adversarial attack detection (Carlini & Wagner 2017 Smith & Gal 2018), and active learning (Kirsch et al. 2019). Unfortunately, ensemble methods may be computationally expensive to train and are always expensive during inference.
A class of models called Prior Networks (Malinin & Gales 2018 2019 Malinin 2019, Sensoy et al.,
2018) was proposed an approach modelling uncertainty in classification tasks by emulating an ensemble using a single model. Prior Networks parameterize a higher order conditional distribution over output distributions, such the Dirichlet distribution. This enables Prior Networks to efficiently yield the same interpretable measures of total, data and knowledge uncertainty as an ensemble. Unlike ensembles, the behaviour of Prior Networks' higher-order distribution is specified via a loss function, such as reverse KL-divergence (Malinin & Gales), 2019), and training data. However, such Prior Networks yield predictive performance consistent with that of single model trained via Maximum Likelihood, which is typically worse than that of an ensemble. This can be overcome via Ensemble Distribution Distillation (EnD?) (Malinin et al. 2020). which is an approach that allows distilling ensemble into Prior Network such that measures of ensemble diversity are preserved. This enables to retain both the predictive performance and uncertainty estimates of an ensemble at low computational and memory cost. Finally, it is important to point out that a related class of evidential methods has concurrently appeared (Sensoy et al. 2018 Amiii et al. 2020). Structurally they yield models similar to Prior Networks, but trained in different fashion.
While Prior Networks have many attractive properties, they have only been applied to classification tasks. In this work we develop Prior Networks for regression tasks by considering the Normal- Normal-Wishart distribution a higher-order distribution over the parameters of multivariate normal distributions. Specifically, extend theoretical work from (Malinin 2019), where such models considered, but

---

Under review as a conference paper at ICLR 2021
never evaluated. We derive all measures of uncertainty, the reverse KL-divergence training objective, and the Ensemble Distribution Distillation objective in closed form. Regression Prior Networks are then evaluated on synthetic data, selected UCI datasets and the NYUv2 and KITTI monocular depth estimation tasks, where they are shown to yield comparable or better performance to state-of-the-art single-model and ensemble approaches. Crucially, they enable, via EnD?, to retain the predictive performance and uncertainty estimates of an ensemble within a single model.
2 REGRESSION PRIOR NETWORKS
In this section we develop Prior Network models for regression tasks. While typical regression models yield point-estimate predictions, we consider probabilistic regression models which parameterizes a distribution p x, 0) over the target y E R² Typically, this a normal distribution:
(1) where j is the mean, and the precision matrix, a positive-definite symmetric matrix. While normal
p/y/ 0) = My A), (n, = f(x;
distributions are usually defined in terms of the covariance matrix Σ = A-1 parameterization using the precision is more numerically stable during optimization (Bishop) 2006; Goodfellow et al. 2016).
While range of distributions over continuous random variables can be considered, we will consider the normal as makes the least assumptions about the nature of and is mathematically simple.
As in the case for classification, can consider ensemble of networks which parameterize multivariate normal distributions (MVN) (p(y|ær, (m)) ) This ensemble can be interpreted as set of draws from a higher-order implicit distribution over normal distributions. A Prior Network for regression would, therefore, emulate this ensemble by explicitly parameterizing higher-order distribution over the parameters ir and A of a normal distribution. One sensible choice is the formidable Normal- Wishart distribution (Murphy) 2012) Bishop 2006), which is conjugate prior to the MVN. This parallels how the Dirichlet distribution, the conjugate prior to the categorical, was used in classification Prior Networks. The Wishart distribution is defined follows:
NW A |m, L, N. v) = (N m, (A),,)
where m and L are the prior mean and inverse of the positive-definite prior scatter matrix, while and v are the strengths of belief in each prior, respectively. The parameters k and v are conceptually similar to precision of the Dirichlet distribution ag. The Normal- Wiihart a compound distribution which decomposes into product of conditional normal distribution over the mean and Wishart distribution over the precision. Thus, a Regression Prior Network (RPN) parameterizes the Normal- Wishart distribution over the mean and precision of normal output distributions follows:
p(u, Alx, 0) = (W Alm, K,,,, {m, L, v) Ω f(( 0)
(3)
where Ω = (m, L, K. w is the set of parameters of the The posterior predictive of this model is the multivariate Student's ss TT distribution (Murphy 2012). which is the heavy-tailed generalization of the multivariate normal distribution:
predicted by neural network.
+
P æ. 0) o)) μ. A)] = Tyymm, x(v xKv-K +1+ K(U-K+11 b.
(4)
In the limit, as v - oc, the T distribution converges to a normal distribution. The predictive posterior of the Prior Network given in equation Jontiyo//oolyyy has a defined mean and variance when v > K + 1.
Figure T depicts the desired behaviour of an ensemble of normal distributions sampled from Normal Wishart distribution. Specifically, the ensemble should be consistent for in-domain inputs in regions of low/high data uncertainty, in figures b. and highly diverse both in the location of the mean and in the structure of the covariance for ofttt--distribution inputs, as in figure D.. Samples of continuous output distributions from a regression Prior Network should yield the same behaviour.
Measures of Uncertainty Given an RPN which displays these behaviours, we can compute closed- form expression for all uncertainty measures previously discussed for ensembles and Dirichlet Prior Networks (Malinin 2019). We can obtain measures of knowledge, total and data uncertainty by considering the mutual information between y and the parameters of the output distribution ( A
Ily, [1,, AHH H [ppp 0z (ply) A)]] (p(r.A)a,0) [H(p(y A)]]
(5)
Knowledge Uncertiitty
Total Unsertainty
Expocted Data Uncertainty
2

---

Under review as a conference paper at ICLR 2021
(a) Low uncertainty
(b) Data uncertainty
(c) Knowledge Uncertainty
0²
a²
μ
(d) Low uncertainty
(c) Data uncertainty
(f) Knowledge Uncertainty
Figure Desired behaviors of an ensemble of regression models. The bottom row displays the desired Wissar Distribution and the top row depicts Normal Distributions samples from it.
This expression consists of the difference between the differential entropy of the posterior predictive and the expected differential entropy of draws from the Normal -Wisart prior. We can also consider the expected pairwise KL-divergence (EPKL) between draws from the Normal-Wishan prior:
Kly, {H.AH AH Epul (Pp(..AA.. In ( μr A)]] (p(,.A [H[p(y) μ, A)]]
(6)
This is an upper bound mutual information (Malinin 2019). Notably, estimates of data uncertainty are unchanged. One practical use of EPKL is comparison with ensembles, as it is not possible to obtain a tractable expression for the mutual information of regression ensemble (Malinin, 2019).
Alternatively, we can consider measures of uncertainty derived via the law of total variance:
P(vl lyj
EpinA (a.0) lA|AA|A
Knowledge Uncertuinty
Total Uncertainty
Expected Data Uncertainty
This yields similar decomposition mutual information, but only first and second moments considered. We provide closed-form expressions of (5)(() in appendix AA] Note, however, that these barsence-based measures are not scale-invariant, and are therefore sensitive the scale of predictions which the model makes. This is relevant to applications such as depth-estimation, where we are likely encounter images with a wide range of possible depths, and therefore varying scale of predictions. We omit the closed-fone expressions for all terms here and instead provide them in appendix A
RKL. training objective Having discussed how to construct Prior Networks for regression, we now discuss how they can be trained. Prior Networks are trained using multi-task loss, where an in-domain loss Lin and an -ut-of-dist (OOD) loss Lout are jointly minimized:
C(O, Dtr. Dout) Exp..(u..e) [on (y, (y, 2r, 0)] + 7 Repo (2)) Cout (ax 0)]
The OOD loss is necessary teach the model the limit of its knowledge (Malinin 2019) and define a decision boundary between the in-domain and out-of-domain regions. In some applications the choice of OOD data can be particularly difficult. Normal distributions sampled from Prior Network should be consistent and reflect the correct level of data uncertainty in in-domain, and diverse in both Achieving the former is challenging, as the training data only
mean and precision consists of samples of inputs æ and targets y. there is no access to the underlying distribution, and associated data uncertainty, represented by the precision A. Effectively, we are attempting to train a Normal-Wishart distribution from targets sampled from normal distribution that are sampled from the Normal- Normal-Wishart, rather than on the normal distribution samples themselves, which is challenging. However, it shown that for Dirichlet Prior Networks minimizing the reverse KL-divergence between the model and an appropriate target Dirichlet induces in expectation the correct estimate of data uncertainty (Malinin & Gales] 2019). As the Normal-Wishanl like the Dirichlet, is conjugate
3

---

Under review as a conference paper at ICLR 2021
prior and exponential family member, the precision can be induced in expectation by considering the reverse KL-divergence between the model ((,,, A|x, 0) and a target Normal-Wisher p(t,, A ()) corresponding to each (1)), The appropriate Normal-Wishan is specified via Bayes's s rule:
A(2 x p(y Ir, P(H, A|S2o)
(9)
where p(y(() μ.. A) is a normal distribution and Pzy = {ma. Lu, Ku. u0) are the parameters of the prior p(,, A (2) defined follows:
N ME mo = N
N E0 - mo)", No = c, v -+1+c (10) N
G L
In other words, we consider a semi-informative prior which corresponds to the mean and seatter matrix marginal distribution p(y), and we see each sample the training data 3 times. The hyper-parameter 3 allows us to weigh the effect of the prior and the data. is a small value, like 10-2 so that Kg and vo yield maximally un-informative, but proper predictive posterior. The reason to use semi-infone anmi-informative prior is that in regression tasks, unlike classification tasks, uninformative priors are improper and lead to infinite differential entropy. Furthermore, we do know something about the data purely based on the marginal distribution, and it is sensible to use that as the prior. The reverse Kl--divergence loss can then expressed as:
Cy x. 0, KL A 0)Ilp(H, ())((
(1I)
= 3 (p Imp u. A)] + KL|p(H, A x, 0)l A|S20)] + z
where is normalization constant independent of parameters 0. For in-domain data, 3 can be set to large value, and for oft-of-d training data 8 0, so that the model regresses to the prior. In-domain, the prior will add a degree of smoothing, which may prevent over over-fitting and improve performance small datasets. A large value of / means the model will yield a larger K, for in-domain data. This will result the first term of (TT) being very close to the expected negative eiglliklihhod of the predictive posterior p ax 0), thereby avoiding degradation of predictive performance. The derivation and closed-form expression for this loss is provided in appendix [A]
Ensemble Distribution Distillation An exciting task which Prior Networks can solve is Ensemble Distribution Distillation (EnD²) (Malinin et al., 2020), where the distribution of an ensemble's 's predictions is distilled into single model. EnD enables retaining an ensemble's improved predictive performance and uncertainty estimates within a single model at low cost. In contrast, standard Ensemble Distillation (EnD) minimizes the KL-divergence between model and the ensemble:
M Σ m==1
N
LEAD ( Dtrr)
KL (L(p(ylx
((m) (p x (1) .b)]]
(12)
NM
This loses information about ensemble diversity. A complication that occurs in probabilistic regression models is that the student model will try to fit single normal distribution top of mixture distribution, spreading itself across each model. This may result in both poor predictive performance and poor estimates of uncertainty. An approach about diversity was considered in (Tran et al.) 2020; Wu et al.] 2020). where the student model parameterizes mixture distribution, where each component of the mixture models particular model in the ensemble, We refer to this as Mixture-Densin Ensemble Distillation (MD-EnD))
both overcome this and retain information
M
plyl,, x)
(e A (m)), m=1
Σ
{m) A (m) (m)) M Im=1 f(x; Φ))
M
(13)
M KL[p(y) ( (00m))))))))). ((ylaa) (m))))
CMD-EDD (Φ, Dern)
NM
This clearly overcomes the issue of distributional mismatch, resolving any issues with poor predictive performance. It also allows the model to, theoretically, retain information about ensemble diversity. However, may be challenging fully replicate the behaviour of each ensemble member in detail
4

---

Under review as a conference paper at ICLR 2021
by having only multiple output heads thus, splitting the model at an earlier point in the network may be necessary, which increases computational cost. EnD avoids the problem by directly modelling the bulk behaviour of the ensemble, rather than the behavior of each individual model.
EnD2 can be implemented for regression via RPNs as follows. Consider an ensemble {p(y)æ: æ, ) where each model yields the mean and precision of a normal distribution. We can define an empirical distribution over the mean and precision as follows:
p(1,, A, a:) = { {L (m)) (mi))
1m=1: (+) j==1 = Dern
(14)
EnD² can then be accomplished by minimizing the negative log- ligeliiclhoood of the ensemble's mean and precision under the Normal-Wishart prior:
LEADE (Ø, Dern)= = Ep(..A.)) In p(1,, A x Φ)] = -ppaa) [KL| A A|x))|p((e, A A +)]
+
(15)
This is equivalent minimizing the KL-diveegencc between the model and the empirical distribution of the ensemble. Note that here, unlike in the previous section, the parameters of a normal distribution are available for every input æ, making forward KI--divergence the appropriate loss function. However, while this is theoretically sound approach, the optimization might be numerically challenging. Similarly to (Malinin et al. (2000) we propose a lemperature-annealing trick to make the optimization process easier. First, the ensemble is reduced to it's mean:
(m))
2
1 (ms)
T+ T+1
T 1
1
Σ m=1
M
(mi)
T
M
(16)
M
AT
((m))
2 A 1(m))
T A-10) T
1(1)
A A-1(0
1
A (
+
M
We use inverses of the precision matrix A because we are interpolating the covariance matrices Σ. Secondly, the predicted Tk and Tv are multiplied by order to make the Normal-Wiste sharp around the mean. The loss is divided by T to avoid scaling the gradients by T, yielding:
xx Φ) = NW m L, TR, Tv),
{m, L, k. v} = f(x:
CEAD= Denn; T) - "Pr(u....)) [- In ( A T)]
(17)
This splits learning into two phases. First, when the temperature is high, the model learns to match the ensemble's 's mean (first moment). Second, as the temperature is annealed down to the model will gradually focus on learning higher moments of the ensemble's 's distribution. This trick may be necessary, as the ensemble may have a highly non-Normal- -Wisaar distribution, which may be challenging to learn. Note that for EnD² it may be better to parameterize the -nneae--vvese- Wishart distribution over the mean and covariance due to numerical stability concerns. However, for consistency, we describe EnD in terms of the -ishart Finally, we emphasize that EnD does not require OOD training data, unlike the RKL objective above. This eliminates the non-trivial challenge of finding appropriate OOD data.
Related Approaches It is necessary to mention evidential approaches. specifically Deep Evidential Regression (Amini et al. 2020). Structurally, it considers models which are similar to RPNs, and is
thus compared to in section 5] However, they do not use OOD training data nor attempt to emulate ensemble's behaviour to enforce high-uncertainty behaviour in OOD regions.
and generalize
Rather, they are trained by maximizing the likelihood of the T distribution (equation (4) together with an evidence regularizer which forces the model yield high v, Λ on regions with low absolute error, and low evidence in regions of high L1 error. However, this seems susceptible to pathologies, such as making the model overconfident if over-fits the training data and achieves zero MAE, or introducing an inverse correlation between K and the scale of predictions in datasets where the scale of predictions range widely. Thus, while they yield encouraging results, their principle of action remains unclear. Another related recent research direction is in developing computationally efficient ensembles (Wen et al. 2020). Here, rather than emulating an ensemble using a single model, the goal is to generate high diversity in predictions while re-using large portions of a neural network model, thereby making for compact ensembles. While such approaches are more computationally efficient than using multiple independent models and use only little more memory than a single model on disk, they still require using M times as much GPU memory at run time, which can be a significant limitation in resource constrained applications, like mobile devices autonomous vehicles.
5

---

Under review as a conference paper at ICLR 2021
3
EXPERIMENTS ON SYNTHETIC DATA
EREN NSPR
Ground Trucs
ENSM NAPM
ENSN
MW
EnD'
DxD
1.0
3.0
0.8
0.8
0.5
0.6
0.0
0.0
0.4
0.4
6.2
6B15
-10
10
6.15
-10
-15
(a) Input Data
(b) Total Uncertainty
(c) Data Uncertainty
(d) Know. Uncertainty
Figure 2: Comparison of different models on synthetic data y NNiin 10 1
0.01). Gray
area indicates training data region.
We first examine Regression Prior Networks on a synthetic one-dimensional dataset with addi- tive heteroscedastic noise. We compare Regression Prior Network trained via RKL (NWPN) and of an ensemble (EnD²) ) to Deep Ensembles (ENSM). The ensemble consists of 10 models that yield Normal output distribution and are trained via maximum likelihood. All models use the same 2-layer architecture with 30 ReLU units. Details of the setup available in appendix B The NWPN was trained via reverse KL-divergence (TT). and Ensemble Distribution Distillation using the loss in equation (17). In-domain training data for all models is sampled between [-10,10], and OOD training data for the NWPN model is sampled from [-25, 2 [200 25). Measures of total, data and knowledge uncertainty obtained via the law of total variance
distribution-dissillattt
The results presented in Figure 2 show several trends. Firstly, the total uncertainty of all models is high in the region of high heteroscedastic noise well as out-of-domain. Secondly, total uncertainty decomposes into data uncertainty and knowledge uncertainty. The former is high in the region of high heteroscedastic noise and has undefined behavior out-of-domain, while the latter is low in-domain and large out-of-disse -it-o--diitribution.. Third, EnD² successfully replicates the ensemble's estimates of uncertainty, though they are consistently larger, especially estimates of data uncertainty out-of-domain. This is consequence of the ensemble being non Non-Normal- -Wisart distributed when it is diverse, leading the EnD² Prior Network to over-estimate support. Thus, these results validate the principle claims that Regression Prior Networks can emulate an ensemble's behavior via multi-task training using the RKL objective via EnD2 and that they yield interpretable measures of uncertainty.
4 EXPERIMENTS ON UCI DATA
In this section, evaluate Normal-Wishart Wishart Prior Networks trained via reverse-KL divergence (NWPN) and Ensemble Distribution Distillation (EnD?) relative to a Decp-Ensemble (ENSM) baseline selected UCI datasets. Other ensemble-methods
not considered, Deep Ensembles have been shown to consistently outperform them using fewer ensemble members (Ashukha et al.
2020 Ovadia al. 2019 Fort et al. (019). We follow the experimental setup of {Lakshminarayandal ct al. 2017) with several changes, detailed in appendix C Out-of-distribttion training data for NWPN is generated using a factor analysis model. This model is linear generative model that learns to approximate in-domain data with & ~ (H. WWT + V) where [W, Y are model parameters. The out-of-domain training examples are then sampled from N "W,, 3 such that they are further from the in-domain region. Table showws comparison of all models in terms of NI.I. and RMSE. Single Gaussian model, ENSM is an ensemble of Gaussian models. Unsurprisingly, ensembles yield the best RMSE, though both NWPN and EnD? generally give comparable NLL scores. Furthermore, EnD² comes close to or matches the performance of the ensemble and outperforms NWPN.
In Table we compare uncertainty measures derived from all models on the tasks error detection and OOD detection. To evaluate error detection Prediction Rejection Ratio (PRR) is used. It shows what part of the best possible error-detection performance our algorithm covers and is defined in appendix C For the evaluation of OOD-detection performance, took parts of other UCI datasets as

---

Under review as a conference paper at ICLR 2021
Table 1: RMSE and NLL of models UCI datasets. Datasets listed in order of increasing size. Results on remaining UCI datasets are available in appendix
RMSE (U)
NLL (c)
Data
Single
ENSM
Ean
NWPN
Single
ENSM
Ean
NWPN
wine
0.65 0.01
0.63 + 0.01
0.63
0.63 + 0.01
+
0/66 + 0.00
0.91 +0::00000 0.02
0,93 0.02
power
0.07
4.00 0.07
4.09 0.07
2.82
2.79 + 0,02
2.79 0.01
281 0.01
MSD
9.08 1 0.00
8.92 + 0.00
8,94 + 0.00
9.07 = 0.00
3.51 + 0.00
3,39 + 0.00
3,99 - 0.00
3.41 = 0.00
OOD data. We made sure that the OOD-data comes from different domains and feature distributions different. Columns of each OOD dataset are normalized using statistics derived from the in- domain training dataset. Details are available in appendix C The results show that all models achieve comparable error-detection using measures of total uncertainty. In terms of OOD detection, EnD² generally reproduces the ensemble's behavior, while NWPN usually performs worse. However, on the MSD dataset, NWPN yields the best performance. This may be due to the nature of the OOD training data- it may simply be better suited to MSD OOD detection. Furthermore, the UCI datasets are generally small and have low input dimensionality MSD, the largest, has 95 features. Therefore, it is difficult to assess the superiority of any particular model on these simple datasets all we can say that they generally perform comparably. In the next section, we validate Regression Prior Networks on more complex, larger-scale task.
Table 2: PRR and OOD detection scores
PRR
AUC-ROC
Data
Model
[[3]
V[s]
I
K.
Vi4
ENSM
0.32 =
0.58 0.01
4:66:00
Wine
EnD
0.30 :: 0.02
0.30 : 0.02
0.65 0.01
0.65 + 0.01
0.65 1: 0.03
NWPN
0.30 0.00
0:30
0.64
ENSM
0.23 :1 0.01
0.61 1: 0.02
0.62 0.02
power
EaD
0.23 +: 0.01
0.23 + 0.01
0.66 0.01
0.66 + 0.01
0.50 + 0.02
NWPN
0.20 4: 0.02
0.20 :: 0.02
0,56 0.01
0,56 1: 0.01
0.31 1: 0.02
ENSM
0.64 ::: 0.00
0.55 0:00
0.62 + 0.0
med
Foly
0.63 + 0.00
0.63
0.50 0.0
0.65 +
NWPN
0.64 1: 0.00
0.64 :1: 0.00
0.73 1: 0.0
0.73 1: 0.0
0.75 :1 0.0
5 MONOCULAR DEPTH ESTIMATION EXPERIMENTS
Having established that the proposed methods work on par with or better than ensemble methods on the UCI datasets, we now examine them on the large-scale NYU Depth v2 (Nathan Silberman
& Fergus 2012) and KITTI (Menze & Geiger 2015) depth-estimation tasks. In this section, the base model is Depsh (DD), which defines a U-Net like architecture on top of DenseNet-169 features (Alhashim & Wonka 2018). The original approach trains it on inverted targets using combination LI, SSIM, and Image-Gradics Gaussian model, which yields mean and precision for each pixel (Single). We also use original targets from the dataset. The rest of the data pre- rre-processing, augmentation. optimization, and evaluation protocol is kept unchanged. On the challenging KITTI benchmark (Geiger et al. 2013) all models are evaluated on the split proposed by (Eigen et al. 2014).
losses. We replace this with NLL training using a
We consider the following baselines: a single Gaussian model (Single), Deep-Ensemble of Gaus- sian models (ENSM). ensemble distillation (EnD) (Hinton et al. 2015), mixture density ensemble distillation (MD-EnD) Tran et al.) (2020): WW et al. (2020) and Deep Evidential Regression (Amini et al. 2020). We distribution-distll the ensemble into a Regression Prior Network (EnDD) with per-pixel Normal-Wishenn
distribution. We also examine training an RPN with the RKI. loss function (NWPN). On the NYU dataset, we take KITTI as OOD training data, and on KITTI,
use NYU as OOD training data. We retrain all models 4 times with different random seeds and report mean. Distribution-disiillatioo is done with temperature annealing (T = 1.0 vs. T = 10.0). For T 10.0 we train with initial temperature for 20% of epochs, linearly decay it to 1.0 during 60% of epochs, fine-tune with T 1.0 for remaining epochs. We found that this greatly stabilized training.
We report standard predictive performance metrics for depth estimation (Eigen et al. (014), a detailed description can be found in appendix [ Results in table 3 show that all probabilistic models either
7

---

Under review as a conference paper at ICLR 2021
outperform work par with the original DenseDepth both datasets. EnD? and MD-EnD achieve performance closest to the ensemble, with End doing marginally better than MD-Enl on Kitti. At the same time both DER and NWPN achieve performance comparable to single model, though NWPN does marginally worse on NYU. In terms of test-set negative log-likelihood, a metric of calibration, DER, NWPN and EnD2 outperform all other approaches, including the ensemble. With regards to EnD² this may be due to the ensemble being poorly modeled by a Normal-Wishart distribution, leading the Prior Network to overestimate the distribution's ' support, and therefore yield less overconfident prediction. NWPN and DER may simply be well-regularized and smoothed. Finally, both EnD and MD-EnD are significantly worse in terms of calibration (NLL) on both datasets. EnD also achieves both poor predictive and calibration performance, which is likely an artifact of trying to model a mixture of Gaussians with a single Gaussian.
Table 3: Predictive Performance comparison
Method
NYUv2 Predictive Performzace
KITTI Predictive Performance
(1)
(1))
6g(1)
me((4)
mmse(4)
(L))
NLL(L)
(1)
62(1)
6a(
rel(C)
mL)((L)
1os
NLL(J)
DD
0.847
0.972
0.993
0.124
0.468
0,054
0.886
0.965
0.986
0.093
4,170
ENSM
0,862
0.975
0.994
0.117
0.438
0,051
0.76
0
0.989
0.998
0,073
3,355
0.032
1.94
Single
0882
0.971
0.001
0.122
0.456
0.053
5.74
0.924
0.967
0.907
0.078
1,545
0.034
2,98
NWPN
0.842
0.968
0.992
0.126
0.172
0,055
-1.60
0.920
0.986
0.997
0.077
3.525
0.035
1.52
BaD
0.851
0.901
0.123
0.053
0.915
0.984
3.9%6
3.27
MD-Fab>
0858
0.922
0.902
0.121
0.051
8.48
0.925
0.957
0.907
0.079
1/466
2.30
EaD?
0.855
0.972
0.993
0.120
0.451
0,052
-1.47
0.928
0.963
0.998
0.075
3,367
0.033
1.42
DER
0.902
0.135
0.464
0.053
3.552
In table 4 we assess all models on the task of out-oo- input detection. Two OOD test-datasets considered: LSUN-churde (LSN-C) and LSUN-bed (LSN-B) (Yu et al.] (0015). which consist of images of churches and bedrooms. The latter is most similar to NYU Depth- V2 and more challenging to detect. OOD images are center-cropped and re-scaled to the in-domain data. With regards to KITTI, which consists of outdoor images of roads and displays a large range of depth in each image, the OOD data is far closer to the camera than the in-domain data as a result of crop-and -rop-and-scale prepossessing. Examples of this a provided in appendix D.4
Results show several trends. EnD consistently outperforms the original ensemble using measures of knowledge uncertainty (I, IC and V(u)). However, when considering measures of total uncertainty (H[E), V(yl). the ensemble tends to yield superior performance on NYU. This is likely due to the over-estimation of the support of the ensemble's 's distribution by EnD?. In contrast, EnD and MDDFnn perform worse than a single model, likely due to either failing to match the ensemble due to distributional mismatch (EnD), or having limited capacity to model the individual behaviour of each ensemble member (MD-EnD). In contrast, EnD models the 'bulk" behaviour of the ensemble, and does not suffer from either issue, which highlights its importance.
At the same time, while NWPN yields the best OOD-detection performance on KITTI, it fails to be robust on NYU. It is necessary to point out that we found training RPNs with RKL challenging in this setting, as it is non-trivial to define what OOD is, especially for depth estimation. In this paper, we consider a different dataset to be OOD. An ablation study with varying OOD weight (appendix (D) shows a trade-off between predictive quality and OOD detection quality. Appropriate training yields the best OOD detection performance (KITTI,, and if the balance is incorrect, the performance is poor (NYU). We speculate that this is because depth estimation is a task sensitive to local features, while discriminating between datasets requires global features, so the two tasks interfere. This highlights the value of EnD?, which does not require OOD training data or additional hyperparameters, and yields good predictive and OOD detection performance.
Interestingly, variance- based measures outperform information -hhoorrticcccecccccco measures of total uncer- tainty, and sometimes knowledge uncertainty on NYU, but fail on KITTI. This is a result of their sensitivity to scale. The models predict very low depth values for OOD data, which means that they have lower entropy and variance. In contrast, MI and EPKL, which are scale-invariant, are not affected. The issue of scale sensitivity is also the reason why Deep Evidential Regression (DER) completely fails on KITTI. As discussed in section @ the evidence regularizer forces the model to yield low uncertainty measures in regions of low error. Therefore. when it observes data too close to the camera, DER tends to detect as in-domain. Additional OOD detection results presented in appendix D.4
8

---

Under review as a conference paper at ICLR 2021
Table OOD detection AUC-ROC (1) comparison
Method
000
NYUN2 vs LSUN OOD Detection K
KTTTI vs LSUN GOD Detection
V[v]
V[μ
*|E]
V(y]
K
Viu!
Single
0.69
0.60
0.02
0.02
NWPN
0.801
0.801
0.199
0.199
0.799
0.999
0.989
1,0
1.0
0.646
0.033
EnD
LSN B
0.724
0.733
0.817
0.806
0.770
0.015
0.017
0.887
0.868
0.090
DER
0.722
0.752
0.717
0.638
0.728
0.028
0.030
0,029
0.005
0.035
MDD-FaD
0.630
0.044
ENSM
0.723
0.672
0.745
0.032
0.822
0.097
Single
0.845
NWPN
0.993
0,993
0.002
0.003
0.992
0.994
0.994
1.0
1.0
0.998
FuD
0.703
017003
0.004
0.044
108993
0.964
0.188
DER
0,877
0.877
0.791
0.675
0.878
0.044
0.355
0.035
0.009
0.040
MD-EaD
13,998
0.119
01422
0.012
0,506
0.031
RNSM
0.887
0.606
0,036
Total
ENSH
Stentttic
Figure 3: Comparison of uncertainty measures between ensembles and EnD². For two input images, we demonstrate the difference between prediction and ground truth (Error), measures of Total Variance (Total) and EPKL (Knowledge) obtained from ensembles and our model. The left and right images corresponds to models trained on the KITTI an NYU datasets, respectively.
Last, figure shows the error and estimates total and knowledge uncertainty of the ensemble and an EnD² model for the same input image. Both the ensemble and model effectively decompose uncertainty. Total uncertainty is correlated with an error and is large at object boundaries and distant points while knowledge uncertainty concentrates both error and uncertainty measures, which are very similar to that of the original ensemble. This demonstrates that EnD² can emulate not only the predictive performance of the ensemble but also the behavior of the ensemble's measures of uncertainty. Further comparisons are provided in appendix D
the interior of unusual objects. EnD² yields
CONCLUSION
This work proposed Regression Prior Networks, yielding a set of general, efficient, and interpretable uncertainty estimation approaches for regression. A Regression Prior Network (RPN) predicts the parameters of Normal-Wishal distribution, enabling to efficiently represent ensembles of regres- sion models, allowing interpretable measures of uncertainty be obtained at low computational cost. In this work closed-form measures of total, data and knowledge uncertainty are obtained for Normal-Wishart RPNs. Two RPN training approaches are proposed. First, the reverse-KL. divergence between the model and a target Normal-Wishar distribution is described, allowing the behaviour of an RPN to be explicitly controlled but requiring an OOD training dataset. Second, Ensemble Distribution Distillation (EnD²) is used, where an ensemble of regression models is distilled into an RPN such that retains the improved predictive performance and uncertainty estimates of the original ensemble. This approach is particularly useful when it is challenging to define an appropriate out-of-domain training dataset, such as depth-estimation. The properties of RPNs were evaluated on selected UCI datasets and two large-scale monocular depth-estimation tasks. Here Ensemble Distribution Distilled RPNs, which do not need OOD training data, were shown to outperform other hingle-mded and distillation approaches in terms of predictive performance, and all models in overall -OD-detection quality. This demonstrates its value as a computationally cheap. general-purposs uncertainty estimation approach for regressions tasks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ibraheem Alhashim and Peter Wonka. High quality monocular depth estimation via transfer learning. arXiv e-prints, 11w/1/1..19441ax11.11.8 1941,, 2018.
org/abs/ abs/
1812. 11941
Babak Alipanahi, Andrew Delong, Matthew T. Weirauch, and Brendan J. Frey. Predicting the sequence specificities of DNA- and RNA- proteins by deep learning. Nature Biotechnology. 33(8):331833, July 2015, ISSN 1087-0156. doi: 10.1038/nbt.3300 BtLhttp: /ttp:///xx.///
/xx doi
2rg/10.1038/ /nt... 3300
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33, 2020.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in Al safety. 1606.06565.
owt. 06565) 2016. arXiv:
REINALDO B CRELLANO-VALLES JAVIER E CONTRERAS-REYS and Mare G Genton. Shan- non entropy and mutual information for multivariate skew-elliptical distributions. Scandinavian Journal of Statistics, 40(1):42-62, 2013.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. International Conference on Learning Representations, 2020. URL htttt::://opnneevieee htt.:///eee..eeeeooooroodddd/ddd Bid-BJxISgKKDDD
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Nicholas Carlini and David A. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. CoRR, 2017. URL htt.:tp:://axxivoorr/bbb///////a
07263
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proc. 2/th
ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '15 pp. 1721-1730, New York, NY, USA, 2015. ACM. ISBN 978- 400444553336-22 doi: 10.1145/2783258. 2788613. httt:://///o acm. ott/ 1ee.111422222 /ttp://.................1/////22
2788613
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2006.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using multi-scale deep network. In Proceedings of the 27th International Conference Neural Information Processing Systems Volume 2, NIPS 14, pp. 2366-2374, Cambridge, MA, USA, 2014. MIT Press.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive. arXiv preprint arXiv:1912.0275: 2019.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proc. 33rd International Conference Machine Learning (ICML-16), 2016.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. 2013.
Ross Girshick. Fast R-CNN. In Proc. 2015 IEEE International Conference Computer Vision (ICCV), pp. 1440-1448, 2015.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Maya Gupta and Santosh Srivastava. Parametric bayesian estimation of differential entropy and relative entropy. Entroyy 12(4):818-8A) 2010.
10

---

Under review as a conference paper at ICLR 2021
Awni Y. Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng. Deep speech: Scaling up end-to-end speech recognition, 2014. URL http: org hwtp://wwggga/ab/eeee/ee
5567 arXiv:1412.5567.
Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition. Signal Processing Magazine, 2012.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http:httt/aat
org/ org//bs// 1503. 02531
Andreas Kirsch, Joost Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning, 2019.
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles, In Proc. Conference on Neural Information Processing Systems (NIPS), 2017.
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. arXiv preprint SCKTv:9902.04477 2019.
Andrey Malinin. Uncertainty Estimation Deep Learning with application to Spoken Language Assessmens. PhD thesis, University of Cambridge, 2019.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pp. 7047-7058, 2018.
Andrey Malinin and Mark JF Gales. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness. 2019.
Andrey Malinin, Bruno Mlodozeniec, and Mark JF Gales. Ensemble distribution distillation. In International Conference on Learning Representations, 2020. URL htttps:://ppeeeeeiie
Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent Neural Network Based Language Model. In Proc. INTERSPEECH, 2010.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Represen- tations in Vector Space. http:://axxiv org/ abs 1301. 3781 2013a. arXiv:1301.378
Tomas Mikolov et Linguistic Regularities in Continuous Space Word Representations. In Proc. NAACL-HILT, 2013b.
Kevin P. Murphy. Machine Learning. The MIT Press, 2012.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
Yaniy Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 2019.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 3179-3189. Curran Associates, Inc., 2018. URL httpp:/ http::///apprs.nis..ooo wttp::/ssiiis..c.ppppeeeeeeeeeee 7580-evidential htt.re/eeeeettaa---eeeeeeeaaaona htt.::/eee........oo.ooo.oaaaaaa pdf
11

---

Under review as a conference paper at ICLR 2021
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. International Conference on Learning Representations (ICLR), 2015.
L. Smith and Y. Gal. Understanding Measures of Uncertainty for Adversarial Example Detection. In UAI, 2018.
Linh Tran, Bastiaan S. Veeling, Kevin Roth, Jakub Swiętkowski, Joshua V. Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans, Sebastian Nowozin, and Rodolphe Jenatton. Hydra: Preserving ensemble diversity for model distillation, 2020. htRtsttp::/ htpppss// wttp:/vwwwwweeeorrureaeee id=ByeaXeBFvi
Ruben Villegas, Jimei Yang. Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to Generate Long-term Future via Hierarchical Prediction. In Proc. International Conference on Machine Learning (ICML), 2017.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach efficient
ensemble and lifelong learning. arXiv preprint arXiv:2002.06711. 2020.
Xixin Wu, Kate M Knill, Mark JF Gales, and Andrey Malinin. Ensemble approaches for uncertainty in spoken language assessment. Proc. Interspeech 2020, pp. 3860-3864, 2020.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction of
large- scale image dataset using deep learning with humans in the loop, 2015. h //a parxiv //arxiv..orr/bbs/1150000 003663365 03365 arXiv:1506.03330
12

---

Under review as a conference paper at ICLR 2021
A DERIVATIONS FOR NORMAL- -IISAART PRIOR NETWORKS
The current appendix provides mathematical details of the Normal- Wishart distribution and deriva- tions of the reverse-KL KL divergence loss, ensemble distribution distillation and all uncertainty measures.
A.1 NORMAL -IISARTT DISTRIBUTION
The Normal- Wishart distribution is conjugate prior over the mean μ and precision A of normal distribution, defined as follows
P(1,, AJot = (W L, k,,,) = (N,m,, K)),:
(18)
where 5 -mm, K, /) the parameters predicted by neural network, W is the density of the Normal and W is the density of the Wishart distribution. Here, m and L are the prior mean and inverse of the positive-definite prior scatter matrix, while x and v are the strengths of belief in each prior, respectively. The parameters and v are conceptually similar to precision of the Dirichlet distribution agg. The Normal -issart is a compound distribution which decomposes into a product of conditional normal distribution over the mean and an Wishart distribution over the precision:
(m KA) = K |
2 (u m)) mm)
K (((m-)))A(a
exp
-
2
(19)
A K-1
exp(- ())
W(
A.L>O
2 2
K ( L
where (-)()) is the multivariate gamma function and K is the dimensionality of y. From (Murphy 2012) the posterior predictive of this model is the multivariate T-distribution:
K +
p 0) = ( A) my
(20)
<(v K +1)
The T distribution is heavy-tailed generalization of the multivariate normal distribution defined as:
T(ylu, E, r( (( 2
(v -0 (1+ 11)
1 0:
(21)
r r())=**
where is the number of degrees of freedom. However, the mean is only defined when and the variance is defined only when > 2.
A.2 REVERSE KL-DIVERGENCE TRAINING OBJECTIVE
Now let us consider in greater detail the reverse KL--divergence training objective
Cly, x, 0 0, = (BplnAAaa,0) In p(y μr, A)] KL[p(H, A A) + Z (22)
where Ωn (mo, Lo, Ko, Po] are prior parameters that we set manually discussed in section 2 It necessary show why the reverse KL-divergence objective will yield the correct level of data uncertainty. Lets consider taking the expectation of the first term in (IT) with respect to the true distribution targets pt. (y(xz) (z). Trivially, we can show that by exchanging the order of expectation, that we optimizing the expected cross-entropy between samples from the Normal-Wishart and the true distribution:
(p.r((ll)) [P(n. pyyle,, A)]] = Ep(...Aaa,)) [p..((/a)] In p(ylu, A)]]
(23)
This will yield an upper bound on the cross entropy between the predictive posterior and the true distribution. However, if we were to consider the forward KL-divergence between Normal-Wishar distributions, would not obtain such expression and not correctly estimate data uncertainty. Interestingly, the reverse KL-divergence training objective has the same form as an ELBO the predictive term and reverse KL-divergence to the prior.
Having established an important property of the RKL objective, we not derive it's closed form expression. Note, that in these derivations, make extended use of properties for taking expectations
13

---

Under review as a conference paper at ICLR 2021
of traces and log- degereeermnnatss matrices with respect to the Wishart distribution detailed in Guupt & Srivastava 2010). For the first term in 22 we use the following property of the multivariate normal:
a ~ ( >
Blle Aæ) = Tr(AE) + m Am;
(24)
which allows us to get:
E pp -- Inp(ylp,, A)] =
2 BOLnAA p²² (ATAAI A(y - K In |A]]
E WwW.m.WWWAA - A 11) K (n - In ||]
A
2
(25)
E PAAA 1.))
11 m) A(y -m) Kr 1 - |n
2
"
1 mn |
2 y m) L m + 2x
-
ln
The second term in 12 may expressed as follows via the chain-rule of relative Cntropy(ovver &
Thomas 2006):
KL p A ( A[Do)] =
(26)
(((())))
KL (L(p( wWwpppp//la.HHOO// Ro)l + (Lpp(A()))I((A(2o)))
The first term in can be computed as:
Ep KL A, 2) PHHAA
Ew(A KL[N RA)][N(y)mm)) koAi]]
(27)
Ko (m m vL(m - mo) +
K
Kig
No
In
1);
2
K
K
while the second term is:
KL (AP(AA12)IP((((SA))) KLIW x)) Lo, vo)]
= v (tr((E L) K)
1o In L.
+I
Px (2))
+
1/0
2/ (5)
(28)
2
Tx (x)
2
A.3 UNCERTAINTY MEASURES
Given Kormal-Wishar Prior Network which displays the desired of behaviours detailed in [ it is possible to compute closed-form expression for all measures of uncertainty previously discussed for Dirichlet Prior Networks (Malinin 2019). The current section details the derivations of uncertainty measures introduced in section 2 for the Normal-Wishee distribution. We make extensive use of (Gupta & Srivastava 2010) for taking expectation log-determinants and traces matrices.
A.3.1 DUTERENTIAL ENTROPY OF PREDICTIVE POSTERIOR
As discussed in section 2 the predictive posterior of Prior Network which parameterizes Normal- Wishart distribution is multivariate T distribution:
+1
(e (p(y/14 A)] Ty m, R(V K K+
L -
(29)
The differential entropy of the predictive posterior will be measure of total uncertainty. The differential entropy of a standard multivariate student's T distribution with an identity scatter matrix Σ = I is given by:
H[T ( μ1 []]] In
r(
V/ K
(v V + K
2 5);
(30)
(VT)
which is result obtained from (ARELLANO VALLE et al. 2013). Using the property differential entropy (Cover& & Thomas 2006), that if æ ~ p(z) and = / + Ax. then:
H(p(v)) H[p(x)] LA).
(31)
14

---

Under review as a conference paper at ICLR 2021
We can show that the differential entropy of standard general multivariate student's T distribution is given by:
H |, = In |م] In
I( (UT)
+
V
K
(v
+
K
2(2)
(32)
Using this expression, we can show that the differential entropy of the predictive posterior of Normal- -ishart Prior Network is given by:
1: K+1
H
= H Hyy,,,
K(V - K + +1 1 P
L my-K-y K
V
#+1
+
K -1(=
I(
(33)
2
2
T(- K
((v - K + 1)7)
K
K K+1
In
|
+
In
2
2
- - +1)
K(V
A.3.2 MUTUAL INFORMATION
The mutual information between the target y and the parameters of the output distribution {μ. A} is measures of knowledge uncertainry, and it the difference between the (differential) entropy of the predictive posterior and the expected differential entropy of each normal distribution sampled from the Normal Wishart:
Tly, {1e, A]] = Ep (p) [, Ep(n.Ala.0) H)p(y μ. A)]]
(34)
Knowledge Uncertainty
Total Uncertainty
Expected Data Uncertainty
The first term, the differential entropy the predictive posterior was derived above in derive the expected differential entropy as follows:
We we
Enw(HLA HHN(y) nr A)]]
BNW(H.AI3) K In(2T)) In |A||
(35)
K |L| vxx |1]
Thus, the final expression for mutual information is:
V ++1
V
pu++1
wb( v K k))))
r ((tt1)
Tly [1,
2
2
2
K+1 I1 1 (
((v K 1)7)
(36)
K
+ 1
K In(Te) vx vxl "1=
In
K
Note that this expression is no longer function of L, which was important in representation data uncertainty.
A.3.3 EXPECTED PAIRWISE KI KL-DIVERGENCE
An alternative measures of knowledge uncertainty which can be considered is the expected pairwise kl-divergence (EPKL), which upper bounds mutual information (Malinin 2019). In this section we derive it's closed form expression for the Normal-Wishan distribution.
Kly,, (H, = Eppcoo.Ao) Ex A.) (x ())ll Ao)I + Tr(AnA K].
1
AA
(37)
2 Ep(ro:Ao) EpcrrnA)) (H1 - pOo 1o - (20 + In
Ao
15

---

Under review as a conference paper at ICLR 2021
Here p(uo Ao) = p(111 A1) p(p,, Ax; 0) In (37) the first term is:
Ep( Bp(()) [11] - No) Ao( - [o)]
Pol Bp(A.) [Im- Ao) 140) (Tr(AA-A7) _Ai')]
1
(38)
Bpluo.Ao) (m mo) Aommo no) x(v K -1
Tr Tr(Aob-1)
vK
x(v K 1)
The second term in 37 is zero, and the third term is:
1
vK
(o( Bp(n Tr(AuAT') = Ao( which in sum gives us:
Tr(Au
L L-1)
(39)
K
- -
1vK(n +
K
K
Kly fM, AH = (v- K -
(40)
2 2k
Note that this also not function of L. just like mutual information. Rather, it is only a function of the pseudo-counts and v.
A.3.4 LAW OF TOTAL VARIATION
Finally, in order to be able compare with ensembles, we can also derive variance-based based measures of total, data and knowledge uncertainty via the Law of total variance, as follows:
V ppH,.Alan,0) [u]
Vplwin.o)ly
Apr,,A (A [A_
(41)
Knowledge Unsertainty
Total Uncertainty
Expected Data Uncertainty
This has a similar decomposition as mutual information. In this section we derive its closed form expression. We can compute the expected variance by using the probabilistic change of variables:
Ep(n.Ala, [A ||= = ENW(HAA) A
[A-']
EWIA) [A- = Ew 1)(A-1]
(42)
V L L-1;
and the variance of expected as:
1 [ [A-1]
V Vp(u.A|z.0)111 = ENW(HA)) (10 m m)
(43)
<(V - K 1) 1-1
Thus, the total variance is then expressed as:
1 +
Vp(vl=,0))1/ k(l
(44)
Note that this yields a measure which only considers first and second moments. In addition, order to obtain a scalar estimate of uncertainty, it is necessary to consider the log-determinant of each measure.
B EXPERIMENT ON SYNTHETIC DATA
The training data consists of 2048 inputs x uniformly sampled from [-10, 10] with targets y N(sin x + ). We use a relu network with 2 hidden layers containing 30 units each to predict the parameters of either Gaussian Normal- -iishrt distribution on this data. In all cases, Adam optimizer with learning rate 10-2 and weight decay 10-4 for 800 epochs with batch size 128. Gaussian models in ensemble are trained via negative log-likelihood starting from different random initialization. To train Regression Prior Network with reverse-KL divergence 512 points were uniformly sampled from [-25, -20 U [20. 25) training-ood data. We use objective (8) with coefficient = 0.5. The prior belief is Ki = 10-2 and in-domain is 10?. For EnD? training we set
T = 1 and add gaussian noise to inputs with standard deviation 3.
16

---

Under review as a conference paper at ICLR 2021
C UCI EXPERIMENTS
The current appendix provides additional details of experiments on the UCI regression datasets. Note that we leave out the Yacht Hydrodynamies datasets, as is the smallest with the fewest features. The remaining datasets are described in the table below:
Table 5: Description of UCI datasets.
Dataset
size
number of features
Boston housing
506
13
Concrete Compressive Strength
1030
8
Energy efficiency
768
9
Combined Cycle Power
9568
4
Red Wine Quality
1599
11
YearPredictionMSD
515345
90
C.I TRAINING
Following Lakshminarayanan et al., 2017) in all experiments except MSD we use 1 layer relu neural network with 50 hidden units, for MSD use 100 hidden units. We optimize weights with Adam for 100 epochs with batch size 32. All hyper-parameters. including learning rate, weight decay, RKL prior belief train data Ko, RKL OOD coefficient y, EnD² initial temperature T and noise level e are set based on grid search, where we use an equal computational budget in all models to ensure fair comparison. Additionally, we use 10 folds cross-validation and report standard deviation based on
C.2 CREATING OUT-OF-DOMAIN DATA
Here we detail how OOD training data for reverse KL-divergence trained Prior Networks is created and how evaluation OOD data is created.
As reverse Klv-diveeeeece training of Prior Networks requires out-of-domain training examples, we use a factor analysis model to generate samples for this in the same way as was done in (Malinin) 2019). Specifically, we train a linear generative model that approximates inputs with & ~ NNu,, WW + +)) where [W, M, V/ are model parameters. In-domain training data is used train this model. The -ut-of-domain training examples are then sampled from N (μ, 3WWT + 3VD)
To estimate the quality of out-of-domain detection, additionally create evaluation out-of-domain data from external UCI datasets: "Relative location of CT slices on axial axis Data Set" for MSD and "Condition Based Maintenance of Naval Propulsion Plants Data Set" for other datasets. We drop all constant columns them and leave first columns and first N rows, where K" a number of features and N is number test examples in corresponding dataset. For each comparison, the out-of-domain datasets are normalized by the per-column mean and variance obtained on in-domain training data, in order make out-of-domain detection task more difficult.
C.3 PREDICTION REJECTION RATIO
Prediction Rejection Ratio (PRR) (Malinin et al. 2020 measures what part of the best possible crter-detetionn performance our algorithm covers. In Figure prrediction curves are shown. They are built by iteratively rejecting test examples according to different orders and computing MSE enror each step. Uncertainty curve corresponds to the order of decreasing uncertainty of some model. Oracle curve corresponds to the best possible order. Prediction Rejection Ration is ratio of the area between Uncertainty and Random curves A R (orange in Figure 4) and the area between Oracle and Random curves ARpracte (blue in Figure
ARuncertainty
PRR
ARarade
17

---

Under review as a conference paper at ICLR 2021
Drade Inrertiitty Ra md0m
Dradle
Intretsiitty
Ra 1d0m
103
Error (MSE)
100
20
60
60
20
20
Percentage rejected examples
Percentage of rejected examples
Shaded area is ARuneetaiinty
(b) Shaded area is ANtoratte
Figure 4: Prediction-rejection curves.
C.4 RESULTS FOR ALL DATASETS
The current section provide full set of predictive performance, error detection and OOD detection results on UCI datasets in tables 8 respectively. Results in table show that all models achieve comparable performance and that EnD? tends come close the ensemble. Table 7 shows the error detection performance of all models in terms of prediction-rejection ratio (PRR). The results clearly show that measures of total uncertainty are useful in detecting errors, though it is more challenging on some datasets. At the same time, measures of knowledge uncertainty do significantly worse. Finally, table shows the OOD detection performance in terms of % AUC-ROC. Here measures of knowledge uncertainty do far better. Notably, on the larger datasets, EnD comes closer to the performance of the ensemble.
Table 6: Prediction performance metrics models on six UCI datasets.
RMSE (-)
NLL (U)
Single
ENSM
Ean
NWPN
Single
ENSM
Ean
NWPN
boston
3,54 d: 0.32
3.52 0.33
3.64 4: 0.33
3.53 :: 0.31
2.58 d: 0.08
2.53 E: 0.07
2.5 0.08
2.47 :: 0.04
1.83 1: 0.07
1.79 0.07
1998 1: 0.05
1.83 1 0.07
1.38
0.04
1,32 1 0.04
1.72 0.04
1665 1 0.08
3
300
wine
0.65
0.01
0,63
0,63 0.01
0.63 : 0.01
1.24
0.16
0.95
0.03
0.91 0.02
0.3 1: 0.02
power
4,07
:: 0.07
4.00
: 0.07
4:06
1: 0.07
4.09
0 0.07
2.82
1: 0.02
2.79
: 0.82
2.79
1: 0.01
2:81 1: 0.01
MSD
9.08
= 0.00
8.92 0.00
8.94 1 0.00
9.07 - 0.00
3.51
0.00
3.39
0.90
3,39
0.00
344 : 0.00
Table 7: PRR scores on all six UCI datasets.
Data
Model
V/y
I
K
V|μ
ENSM
0.50 +: 0.07
.0.15 + 0.7
0.41 0.08
boston
E
0.61
0007
-0.0 0.15
4002 + 0.15
0.06
:: 0.09
+
RNSM
0.90 0.01
40.0 1: 0.02
0.34
EnD
0.83
0.02
0.83 1: 0.00
0.77 1 0.03
-0.7 1: 0.03
0.60 0.05
NWPN
0.85
0.02
0.85 4: 0.02
0.32
0.32 =
0.84 0.02
ENSM
0.48 +: 0.05
0D1 # 0:07
0.27 0.06
0.06
0:06
NWPN
0.54 0
0.54 1 0,03
0.35 = 0.04
0.35 = 0.04
0.51 0.03
ENSM
0.32 1: 0.02
0.10 0.03
0.25 0.04
wine
2
0.30 0.02
0:03
0.06 0.03
0.19
BNSM
0.23 0.01
+ 0.2
DOWNY
2
0.23
0.01
0.23 1: 0.01
.0.02 + 0.03
.002 + 0.03
0.15 0.02
NWPN
0.20
0:02
(1.00 +: 0.02
-0.03 4: 0.02
2003 + 0.02
0.16 0.01
ENSM
0.64 0.0
0.07 = 0.0
0.39 + 0.0
E
NWPN
0.64 =: 0.0
0.64 0.0
.0.20 di 0.0
40.20 4: 0.9
061 + 0.0
18

---

Under review as a conference paper at ICLR 2021
Table 8: OOD Detection (ROC-AUC) of models UCI datasets.
Model
M[4
I
ENSM
0.75 0.02
0.68 0,02
0.86 1: 0.02
EnD
0.75 + 0.01
0.75 0.0
0.63 4 0.01
0.63 0.01
0.76 +: 0.01
NWPN
0.61 1 0.02
0.65 1: 0.02
0.71 1: 0.01
0.71 0.01
0.89 1: 0.01
ENSM
0.16 0.01
0.51 0.05
0.77 + 0.03
Fat
061 + 0.0
0:57 0.02
+ 0.03
0.3 +
+: 0.01
0.02
ENSM
EnD
0.48 1 0.02
0.18 0.02
0.45 1: 0.01
0:45 0.01
0.47 1: 0.01
0.84 0.01
=: 0.01
0.01
ENSM
0,88 0.03
0.58 0.01
0.56 :: 0.02
EnD
0.59 0.03
0.65 4: 0.01
0.01
0.65 1: 0.03
NWPN
0.22 + 0.02
0.42 - 0.02
0.64 = 0.01
0.64 0.01
0.53 + 0.00
ENSM
0.02
0.64 0.02
0.62 1: 0.02
power
EnD
0.37 + 0.03
0.66 + 0.01
0.66
+ 0.02
0.02
ENSM
0.66
0.55
0.62 0.0
msd
EaD
0.66 + 0.0
0.66 = 0.0
0.50 =: 0.0
0.50 =: 0.0
0.65 1
NWPN
0,68 0.0
0.68 1:
0.73 0.0
0.73 4:
0.75 0.9
D DEPTH ESTIMATION EXPERIMENTS
D.I METRICS DEFINITION
In this section we give a description metrics Table which are commonly used prior work on monocular depth estimation (Eigen et al. 2014 Fu et al. 2018 Alhashim & Wonka 2018).
Let y be target depth map for particular image, while ÿ is predicted depth map, and let D, prediction for i-th pixel, while / is the set of all pixels. Then, the metrics for an individual image defined as follows:
ÿ: yi
Thresholds (61, 82, 8z): of G. s.t. max
8h
1.25, 1, 1,
y:
1
yi
Absolute Relative Difference (rel):
7
yi
(45)
iEr
RMSE in log space: (logıo)
1l 9: log 3:)
Metrics for the whole dataset are obtained by averaging individual metrics. They allow to assess different properties of the model: & demonstrate confidence intervals, shows the ratio between prediction error and target, and log measures the error in log-space which is less sensitive to outliers.
D.2 CALIBRATION
KDE estimates for histograms of NL..s are provided in the figure below. We can see that both EnD? DER, and NWPN models yield far more consistent NLL than their Gaussian counterparts, while the latter has a long tail outliers. This is probably connected with the fact that the Student T distribution has heavier tails than Gaussian, which allocates greater probability mass farther from the mean, allowing the model to be less confident about its predictions, especially outliers. We also that DER likelihoods are shifted slightly to the right, which highlights the bias introduced by its evidence regularizer.
D.3 ADDITIONAL DEPTH ESTIMATION EXPERIMENTS
In RPN training with RKL objective, we observed the optimization trajectory to be very unstable and initializationnsensiiiee To combat this, we linearly increase 7 from to some predefined value during
19

---

Under review as a conference paper at ICLR 2021
0.6
0.6
Single
EnD
0.5
ENSM
0.5
DER
MD-EnD
NWPN
0.4
EnD
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
5
10
15
20
5
10
15
(a) NLL on Nyuv2
(b) NLL on Nyuv2
1.2
Single
1.2
EnD2
ENSM
DER
MD-EnD
1.0
11
NWPN
0.8
EnD
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
2
4
6
10
2
4
6
10
(c) NLL on KITTI
(d) NLL on KITTI
Figure NLL histograms on Nyu and KITTI datasets.
the first five epochs, which allows our models concentrate initially on predictive performance, and then gradually capture the properties of "in-domain" samples.
Additionally, we performed ablation study across different coefficients y, with the results provided in tables (nd On NYU dataset, we see that models with lower improve performance at the cost of decreased quality of OOD detection. This may be an indication that the task of accurate prediction may not align well with the model's s ability to detect unfamiliar samples. Based on this, we decided to use the coefficient 7 = 0.05 as achieving the best trade-off, and then fine-tuned the respective model for 10 additional epochs until convergence.
Table 9: RKL ablation on NYU with training OOD KITTI.
Method
Prodictive Performance rel(L)
OOD Detection ROC-AUC(T)(H(E)
(1)
62(1)
6s(t)
rmse(L)
10g
NLL(L)
Single
0.852
0.971
0.993
0.122
0.456
0.053
5.74
0.69
NWPN
0,852
0.971
0.993
0.122
0.450
0.052
-1.46
0.691
7 0.005
0.987
NWPN 0.01
0.805
0.952
0.986
0.148
0.515
0.061
-1.57
0.81
NWPN 7 0.025
0.665
0.905
0.975
0. 0.197
0.666
0.085
.0.84
0.866
NWPN 0.05
0.556
0.849
0.956
0.240
0.821
0.106
0.876
D.4 ADDITIONAL OOD DETECTION EXPERIMENTS
We also provide additional OOD detection results, where models trained on NYU detect KITTI OOD data, and vice versa. Note, we do not evaluate NWPN this scenario, as there two datasets represent the training data. The results generally follow the same trends those outline in the main paper.
20

---

Under review as a conference paper at ICLR 2021
Table 10: RKL. ablation KITTI with training OOD NYU.
Mothod
Prodictive Performance rel(L)
OOD Detection
61(1)
62(1)
6s(t)
mss(J)
log:o(4)
NLL(J)
ROC-AUC(T))22
Single
0.924
0.987
0.997
0.078
3:537
0.035
2.98
0.845
NWPN, 0.005
0.997
3.528
0.035
NWPN.
0.924
0.987
0.997
0.079
3.392
0.035
1.55
1.0
NWPN. 7 0.01
0.919
0.986
0.997
0.080
3,435
0.035
1.0
NWPN. 0.025
0.922
0.986
0.997
0.079
3.513
0.036
1.63
1.0
NWPN, 0.05
0.920
0.986
0.997
0.077
3.526
0.035
1.52
1.0
Table 11: OOD Detection (ROC-AUC) of NYU models on KITTI and vire-versa.
Method
NYUv2 vs KITTI OOD Detect
KITTI vs NYUv2 OOD Detect
H[
V[y]
I
KC
V|u
H[E
V(y]
I
K.
[[4
Single
0.892
0.892
0.013
0.013
ENSM
0.927
0.779
0.929
0.02
0.806
0.076
EnD
0.774
0.774
0.001
0.001
EnD
0.896
0.905
0.946
0.938
0.935
0.004
0.004
0.750
0.724
0.009
DER
0.937
0.937
0.882
0.811
0.937
0.038
0.04
0.04
0.008
0.047
MD-EnD
0.707
0.223
0.403
0.002
0.733
0.026
Additionally, in the following two figures we provide examples of the in-domain and OOD data appear when it is pre-processed. From the figures can clearly see that relative to NYU Depth, all images are either at comparable depth or little farther. However, relative to KITTI, OOD images much closer to the camera, both naturally, and exacerbated via the crop and scale operation.
(a) NYU Depth V2 ID
(b) LSUN-BED OOD (c) LSUN-Church OOD
(d) KITTI OOD
Figure 6: Examples of test inputs for Nyu model. Images are in order: NYU, LSUN-bed, LSUN- church, KITTI. OOD images are center-cropped and re-scaled to the in-domain data, preserving the aspect ratio.
21

---

Under review as a conference paper at ICLR 2021
(a) KITTI ID
(b) LSUN-BED OOD
(c) LSUN-Church OOD
(d) NYU NYUDepth V2 OOD
Figure 7: Examples of test inputs for KITTI model. Images are in order: KITTI, LSUN-bed, LSUN- church, NYU. OOD images center-cropped and re-scaled to the in-domain data, preserving the aspect ratio.
22

---

Under review as a conference paper at ICLR 2021
Knowledge
ENSM
ENSM
EnDi
Stataties
Staratics
Inout
Einor
Total
Knowledge
Inout
Eroor
Total
Knowledge
ENSA
ENSM
EnDx
Scetitics
Stetistics
Total
Knowledge
Knowledge
Dije
ENSM
ENSM
t Uie
Enio
Statsites
Statstics
Inout
Enror
Total
Knowledge
Input
Enro
Total
Knowledge
ENSM
ENSM
EnD
Statstics
Statatics
Figure 8: Uncurated comparison of ENSM EnD behaviour Nyuv2 dataset (best viewed in color).
23

---

Under review as a conference paper at ICLR 2021
Predictions
Predictions
Knowledge
ENSM
ENSM
EnDi
.10
Statstics
Staratics
Ingul
Prodictions
Total
Knowledge
Inout
Predictians
Total
Knowledge
ENSM
ENSM
EnDt
Scetistcs
Stetistic
Predictians
Total
Knowledge
Predictions
Total
Knowledge
ENSM
ERSM
Enio
Statsiles
Stanstics
Inout
Predictions
Total
Knowledge
Inout
Predictions
Total
Knowledge
ENSS
ENSM
EnDt
Statatics
Statatics
Figure 9: Uncurated comparison of ENSM EnD models trained Nyuv2 behaviour Kitti and LSUN-bel bed datasets (best viewed in color).
24

---

Under review as a conference paper at ICLR 2021
Input
Knowledge
Input
EX134
ENSM
EmD
Statisties
Statittis
Input
Enror
Total
Knowledge
Input
Error
Total
Knowledge
ENSSM
ENSM
EnDY
Stllistcs
Stalistics
Enror
Total
Knowledge
Error
Total
Knowledge
ENSM
ENSSA
EnD
Statishies
Statistics
Input
Knowledge
Input
ENSM
ENSM
EnDo
Stalisiics
Stalistics
Figure 10: Uncurated comparison of ENSM EnD? behaviour KITTI dataset (best viewed in color).
25

---

Under review as a conference paper at ICLR 2021
Input
Prediction
ENSM
ENSM
EnD
Stattstics
Stalistiss
Input
Prediction
Total
Knowledge
Input
Prediction
Total
Knowledge
ENSM
ENSM
EnD
Statistics
Statisifes
Prediction
Total
Knowledge
Prediction
Total
Knowledge
ENSM Cee
ENSS
EnD
Stallssiiss
Statisties
Input
Prediction
Total
Knowledge
Input
Prediction
Total
Knowledge
ENSM
ENSM
EmDo
Statittis
Statsfies
Figure 11: Uncurated comparison of ENSM EnD models trained KITTI behaviour Nyu and LSUN-bed datasets (best viewed in color).
26


Reviews:
[{"review_id": "awTxN3m2fOi", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends Prior networks models, previously introduced for classification, to regression problems.  Prior networks are neural networks whose main target is to \"modelling uncertainty in classification tasks by emulating an ensemble using a single model\".  Standard Prior networks models output the parameters of a Dirichlet probability distribution. This Dirichlet probability distribution then defines a distribution over categorical probability distributions over the different classes. This hierarchical approach allows to better capture uncertainty. The presented approach extends this framework to regression tasks. So, instead of returning the parameters of a Dirichlet distribution, it returns the parameters of a Normal-Wishart distribution, which then defines a probability distribution over Normal distributions, and, in turn, each Normal distribution defines a probability distribution over the value of the target variable.  \n\n\nPros:\n* The presented approach is sound and addresses a relevant problem, which is modelling uncertainty for regression problems. \n* A method for distilling an ensemble model into a single model while maintaining accuracy is also proposed. \n* The proposed approach does not incur in computational and memory overheads like standard deep ensembles. \n* This work properly approaches technical difficulties (such as employing numerical stable precision parametrizations of the Normal-Wishart distribution) that arise in this kind of problems.\n\nCons: \n* The presented approach does not introduce any novel idea or insight. It's a relatively simple extension of a previously published method. \n* The empirical results do not show a clear advantage of the presented approach wrt previously published proposals. \n* The advantage of having a small computational and memory overhead is not properly evaluated with other proposals which also have a small  computational and memory overhead [1] (although this proposal has not been defined for regression problems, the adaptation to regression is as simple as the adaptation of the DeepEnsembles models employed in this work). \n\n\nI can not recommend the acceptation of this paper because I find the originality of the work quite limited. Although the extension of prior networks to regression task is mot really straightforward because of technical issues related to the problem of learning the parameters of a Normal-Wishart distribution. The general strategy to do that exactly matches the previous steps employed when introducing prior networks.  In consequence, this work does not provide any new relevant insight into the problem of modelling uncertainty and learning models with well-calibrated predictions. \n\n\nMinor comments:\n- Eq (14): T parameter is not defined. Temperature? \t\n- Typo at the end of Page 5: [-25,20] --> [-25,-20]\n- ENSM is defined after Table 1. \n- Fix the following reference:\nAndrey Malinin and Mark JF Gales. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness. 2019. \n\nPost-rebuttal:  I thank  the authors' effort for the improvement of the manuscript following the comments of the different reviewers. I think the overall quality of the paper has really improved. But, after many thoughts, I still think there is a limited novelty in this paper. I have increased my score to 5. But I can not recommend this paper for publication. \n\n  adding baseline models to the paper and missing citations. I do think this improves the overall paper by a lot. As mentioned already in my paper, I do believe this is a nice idea and executed well, even though novelty might be limited. I am keeping my score and recommending an accept.\n\n[1]  Wen, Y., Tran, D., & Ba, J. (2020). BatchEnsemble: an Alternative Approach to Efficient Ensemble and Lifelong Learning. arXiv preprint arXiv:2002.06715.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_GrpdkEvnoU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Prior Networks (Malinin & Gales, 2018) use Dirichlet prior over categorical predictive distributions to distill ensembles for classification tasks. This paper extends Prior Networks to the regression setting by using a Normal-Wishart prior in order to attempt to match the predictive diversity. The authors define the model and loss terms including analytical derivation and evaluate their proposed approach with synthetic data, UCI datasets and monocular depth estimation. \n\n_Strengths_:\n- The paper is well-written and clearly structured.\n- Most design choices are justified.\n- Simple idea (in a good way!) which seemed to work well, shown by the evaluation.\n\n_Weaknesses_:\n- Most of the work seems to be heavily based on Prior Networks (Malinin & Gales, 2018). Even Section 2.1 seems to be exactly like the Subsection in the paper about Prior Networks. This paper mainly focuses on an extension to the regression task. Therefore, the contribution / novelty of this paper is incremental. However, I still think the authors did a good job to present a general distillation method for regression task. Therefore, I would consider the novelty a minor weakness.\n- I am on the fence about specifying the OOD dataset for learning with the loss in Eq. 8. I believe it is difficult to decide what kind of model to use for generating the OOD dataset, thus, the model choice can lead to large differences in performance. This is not really discussed. Further, the models trained have more data available for training, I believe it is not quite fair to compare against models which only have been trained on in-domain-data.\n- There are no comparisons to other approaches for distillation of regression tasks. I understand, that this paper wants to show a viable general approach for regression distillation, however, this work is not the first one to do so and therefore should consider existing work.\n\n_Overall assessment_: For me, this paper is borderline. The weaknesses, especially the OOD dataset used for training and the lack of comparisons in the evaluation are concerns. However, I like the idea and the execution so therefore, I would recommend a weak accept (6).\n\n_Detailed comments and questions_:\n- OOD data: I have seen that you have an ablation for the degree of regularization on the OOD dataset. However, what about different OOD data? Why choose KITTY and not a different dataset? Were there any large difference in performance?\n- Table 3: I notice that NLL performance of distilled models are better than the actual ensemble, how can this be?\n- OOD detection for monocular depth estimation: Did you also trained the comparing models with the OOD data, e.g. DD?\n- Comparing models: Have you consider comparing your model to other ones, e.g. [1, 2]? This could improve your paper and approach to show that it also consider existing work on regression distillation.\n\n_Post-rebuttal_:\nI really appreciate the authors adding baseline models to the paper and missing citations. I do think this improves the overall paper by a lot. As mentioned already in my paper, I do believe this is a nice idea and executed well, even though novelty might be limited. I am keeping my score and recommending an accept.\n\n[1] Chen, G., Choi, W., Yu, X., Han, T. and Chandraker, M., 2017. Learning efficient object detection models with knowledge distillation. In Advances in Neural Information Processing Systems (pp. 742-751).\n[2] Saputra, M.R.U., de Gusmao, P.P., Almalioglu, Y., Markham, A. and Trigoni, N., 2019. Distilling knowledge from a deep pose regressor network. In Proceedings of the IEEE International Conference on Computer Vision (pp. 263-272).", "extra_scores": {}, "extra_text": {}}, {"review_id": "KMA7obppU9F", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the Paper:\n\n        This paper introduces regression prior networks. These are models that aim at capture predictive uncertainty, both epistemic and aleatoric, in the context of regression problems. Regression prior networks can also be used to compress an ensemble of predictors into a single model while keeping the benefits of the ensemble. That is, better predictive performance and uncertainty estimates. The method is validated on several problems from the UCI repository and compared with ensemble methods.\n\nSpecific details:\n\n        I believe that this is a nice paper that illustrates an appealing method for uncertainty estimation in the context of neural networks. My main concern, however, is that it builds heavily on previous work. In particular, prior networks have already been proposed for classification and they have also been used to distill (compress) an ensemble. There is hence not much novelty here, only the extension to regression problems since, previously, only classification problems have been addressed. The use of prior networks for ensemble distillation is also not new. All this questions the novelty of the proposed approach.\n\n        The extension to regression seems to follow very closely the work already carried out for classification. The only difference is that a Normal Wishart distribution is used instead of a Dirichlet distribution.\n\n        The experiments carried out are extensive and consider different tasks involving prediction accuracy and out of distribution data detection. My main concern, however, is that no comparison is carried out with alternative methods to estimate prediction uncertainty such as those of Bayesian neural networks using variational inference or dropout. The authors should comment on the advantages of their method with respect to these techniques.\n\n        The method proposed is also complicated and has several training parameters. The authors give specific values for them, but it is not clear the motivation for them or the sensitivity to their values.\n\n        The paper is clearly written but heavily relies on previous work, making the reading difficult for someone who is not familiar with it. The paper is not self-contained.\n\n        Summing up I believe that this could be an interesting contribution for the conference, suffering from a reduced amount of novelty.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6l-gyq0bIrO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses interpretable uncertainty quantification for data driven models. In particular, the authors focus on a sub-class of methods known as Prior Networks and attempt to extend these methods to regression tasks as existing approaches address classification only. The author contribution is thus clearly stated and positioned w.r.t. prior arts and tackle a non-trivial issue.\n\nIn the classification setting, the Dirichlet distribution is pretty much the universal model for the parameters of multinomial distributions. For regression, i.e. continuous r.v., there is no such universal solution and the authors chose to focus on outputs that have a normal distribution. The parameters of this latter are assumed to be normal-Wishart. Although, the proposed method is de facto non-applicable to other types of distributions, it can be argued that this already covers a majority of situations. \n\nThe paper is rather well organized and seems technically sound. This said, a few mathematical details are missing and, most importantly, the experiments are not very convincing. These concerns, also with other minor remarks are detailed below, section by section.\n\nsec 2.2\n\nMaybe give the explicit definition of Z to clarify that is does not depend on network parameters.\nThe presence of the OOD loss term in (8) is a bit artificial as it boils down to regularizing because of the choice of beta. Is this choice systematic ?\nIn (9), how is p(y | mu, Lambda) computed ? Is it a T distribution ?\n\n2.3\n\n(12) lacks clarity : dataset is equal to an empirical distribution... Do you mean p hat is a sum of Dirac ?\nWhat does phi represent ?\n\n3\nThe acronym ENSM is not explained. I believe this corresponds to the deep ensemble. \nThe Prior Networks achieve a form of disambiguation but the quality of it is a bit disappointing compared to ENSM. In particular, data uncertainty raises quickly for out-of-domain inputs. \n\n4\n\nThe presentation of the experimental protocol in 4 lacks clarity thereby impairing the interpretation of the results. The definition of the unconventional performance criteria [(Malinin et al. 2020] must be recalled (at least in an appendix). \nIn addition, as honestly mentioned by the authors, these datasets may not offer sufficiently rich problems to provide interesting comparisons. Besides, the way that OOD data is generated does not seem to necessarily produce inputs that are not covered by the in-domain distribution. Perhaps, the authors could use a \"bad GAN\" to obtain such data points, I mean a GAN where the generator and the discriminator would co-operate instead of being adversaries. If it converges, the generator would produce synthetic inputs that are easy to discriminate, thus far from true inputs. \n\n5\nWhile the dataset used in this section is more challenging, the experiment description is confusing. Again, performance criteria are not sufficiently explained and the general message becomes cryptic. Table 3 is overly complicated, I think RMSE is fairly enough to depict regression performances. Moreover, the definition of some columns are missing.\nIn Table 4, the performances of the methods seem quite unstable. For example, NWPN works fairly well for a given dataset configuration for one knowledge uncertainty criterion but fails miserably using another criterion on the same data.\nOn Fig 3, from what dataset are these image coming from ? Why are these or that object presumably \"unknown\" to the model ?\nI think the whole section deserves some re-writing.\n\nFinal remark : there are a few English mistakes that should be wiped out. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a useful contribution to the growing literature on uncertainty estimation with deep learning. The review process has significantly helped with strengthening this paper, specifically with the concerns about novelty and sufficient comparisons to existing work. I hope you will continue to improve this work for submission to a future venue.
Decision: Reject