PIVEN: A Deep Neural Network for Prediction Intervals with Specific Value Prediction
Authors: Eli Simhayev
Gilad Katz
Lior Rokach
Venue:  (2021)
Keywords: Prediction Intervals
Uncertainty Estimation
Regression

Abstract: Improving the robustness of neural nets in regression tasks is key to their application in multiple domains. Deep learning-based approaches aim to achieve this goal either by improving their prediction of specific values (i.e., point prediction), or by producing prediction intervals (PIs) that quantify uncertainty. We present PIVEN, a deep neural network  for producing both a PI and a prediction of specific values. Unlike previous studies, PIVEN makes no assumptions regarding data distribution inside the PI, making its point prediction more effective for various real-world problems. Benchmark experiments show that our approach produces tighter uncertainty bounds than the current state-of-the-art approach for producing PIs, while maintaining comparable performance to the state-of-the-art approach for specific value-prediction. Additional evaluation on large image datasets further support our conclusions.

Under review as a conference paper at ICLR 2021
PIVEN: A DEEP NEURAL NETWORK FOR PREDIC- TION INTERVALS WITH SPECIFIC VALUE PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Improving the robustness of neural nets in regression tasks is key their application in multiple domains. Deep learning- based approaches aim to achieve this goal either by improving their prediction of specific values (i.e., point prediction), or by producing prediction intervals (PIs) that quantify uncertainty. We present PIVEN, deep neural network for producing both Pl and prediction of specific values. Unlike previous studies, PIVEN makes no assumptions regarding data distribution inside the PI, making point prediction more effective for various real-world problems. Benchmark experiments show that our approach produces tighter uncertainty bounds than the current state-of-the-ar approach for producing Pls. while maintaining comparable performance to the state-of-the-an approach for specific palue-prediction. Additional evaluation on large image datasets further support conclusions.
INTRODUCTION
Deep neural networks (DNNs) have been achieving state-of-the-art results in a large variety of complex problems. These include automated decision making and recommendation systems in the medical domain (Razzak et al. 20188) autonomous control of drones (Kaulmann et al. (018)) and self driving cars (Bojarski et al. 2016). In many of these domains, is crucial not only that the prediction made by the DNN is accurate, but rather that its uncertainty is quantified. Quantifying uncertainty has many benefits, including risk reduction and more reliable planning (Khosravi et al., 2010).
In regression, uncertainty is quantified using prediction intervals (PIs), which offer upper and lower bounds on the value of data point for a given probability (e.g., 95%).. Existing non-Bayesian PI generation methods can be roughly divided into two groups: a) performing multiple runs of the regression problem, as in dropout (Gal & Ghahramani 2016) or ensemble-based methods Lakshminarayanan al. 2017), then deriving post-hoc the Pl from prediction variance, and; b) dedicated architectures for the Pl generation (Pearce et al. 2018 Tagasovska Lopez- Paz, 2019).
While effective. both approaches have shortcomings. On the hand, the former group is not optimized for PIs generation, having to convert a set of sampled values into a distribution. This lack of Pl optimization makes using these approaches difficult in domains such as financial risk mitigation or scheduling. For example, providing a PI for the number of days a machine can function without malfunctioning (e.g., 30-45 days with 99% certainty) is more valuable than prediction for the specific time of failure. On the other hand, the latter group Plad-----dicctted architectures -provides accurate upper and lower bounds for the prediction, but lacks in the accuracy of its value predictions. Consequently, these approaches select the middle of the interval as their value prediction, which is (as we later demonstrate) sub-optimal strategy that makes assumptions regarding the value distribution within the interval. The shortcomings of Pl-dedicated architectures with regard to value prediction are supported both by Pearce et al. 2018) and by our experiments Section 5
We propose PIVEN (prediction intervals with specific value prediction). a novel approach for simultaneous PI generation and value prediction using DNNs. Our approach combines the benefits of both above-mentioned groups by producing both PI and value prediction, while ensuring that the latter is within the former. We follow the experimental procedure recent works, and compare approach to current best-performing methods: Quality- -Drienn PI (QD) by Pearre et al. (2018) (a dedicated PI generation method), and Deep Ensembles (DE) by DeepMind (Lakshminaray et al. 2017) Our results show that PIVEN outperforms QD by producing narrower Pls, while

---

Under review as a conference paper at ICLR 2021
simultaneously achieving comparable results to DE in terms of value prediction. Additional analysis on large image datasets and synthetic data shows that PIVEN performs well on skewed value distributions and can be effectively combined with pre-trained DNN architectures.
2 RELATED WORK
Modeling uncertainty in deep learning has been active of research in recent years PPaace
et al. 2018 Qiu ct al., 2019; Gal & Ghahramani 2016; Lakshminaray yanan et al., 2017 Keren
al. 2018 Kendall & Gal 2017: Geifman al. 2018 Ovadia al. 2019). Studies in uncertainty
modeling and regression can be generally divided into two groups: Pl-based and non- non-Pl-based
Non-PI approaches utilize both Bayesian (MacKay) 1992) and non-Bayesian approaches. The former methods define prior distribution on the weights and biases of a neural net (NN), while inferring a posterior distribution from the training data. Non-Bayesian methods (Gal & Ghahramani 2016
Lakshminarayanan et al.) 2017: Qiu et al. 2019h do not use initial prior distributions. In (Gal
& Ghahramani, 2016). Monte Carlo sampling was used to estimate the predictive uncertainty of NNs through the use of dropout over multiple runs. A later study (Lakshmiaarayaaa et al., 2017) employed combination of ensemble learning and adversarial training to quantify model uncertainty. In expansion of previously propoosed approach (Nix & Weigend, 1994), each NN optimized to learn the mean and variance of the data, assuming a Gaussian distribution. Recently, Qiu et al. (2019) proposed a post-hoc procedure using Gaussian processes to measure uncertainty.
Pl-based approaches are designed to produce PI for each sample. Keren et al. (2018) propose a post-processing approach that considers the regression problem one of classification, and uses the output of the final softmax layer to produce Pls. Tagasovska & Lopez- Paz (2019) propose the use of loss function designed to learn all conditional quantiles of given target variable. LUBE (Khosravi] et al. 2010)) consists of loss function optimized for the creation of Pls, but has the caveat of not being able use stochastic gradient descent (SGD) for its optimization. recent study (Pearce et al., 2018) inspired by LUBE, proposed a loss function that is both optimized for the generation of PIs and can be optimized using SGD.
Each of the two groups presented above tends to under-perform when applied to tasks for which its loss function was not optimized: Non-PI approaches produce more accurate value predictions, but not optimized to produce PI and therefore produce bounds that are less tight. Pl-based methods produce tight bounds, but tend to underperform when producing value predictions. Recent studies (Kivaranovic al. 2019 Romano al., (209)) attempted produce both value predictions and Pls by using conformal prediction with quantile regression. While effective, these methods use a complex splitting strategy, where one part of the data is used produce value predictions and PIs, while the the other part is to further adjust the Pls. Recently, SSaemm et al. (2020) proposed a method for combining the two, together with post-hoc optimization. Contrary to these approaches, PIVEN produces PIs with value predictions in an end-to-end manner by relying on a novel loss function.
PROBLEM FORMULATION
In this work we consider a neural network regressor that processes input a E X' with an associated label y E IR, where X' can be any feature space (e.g., tabular data, age prediction from images). Let (Iz, y4) X' IR be data point along with its target value. Let U, and L; be the upper and lower bounds of PIs corresponding to the ith sample. Our goal is to construct (L,, U,, ys) so that Pr(1, < y < U.) 2 a. We refer to a the confidence level of the PI.
We define two quantitative measures for the evaluation of PIs, as defined in Khosravi et al. 2010). First, we define coverage the ratio of dataset samples that fall within their respective Pls. We measure coverage using the prediction interval coverage probability (PICP) metric:
PICP == k
where n denotes the number of samples and k; = 1 if yE E (L,, U,), otherwise k; = 0. Next, we define quality metrie for the generated Pls with the goal of producing tight bound possible
2

---

Under review as a conference paper at ICLR 2021
while maintaining adequate coverage. We define the mean prediction interval width (MPIW) as,
MPIW
U.-L
(2)
1=1
When combined, these metrics enable comprehensively evaluate the quality of generated Pls.
4
METHOD
4.1
SYSTEM ARCHITECTURE
The proposed architecture is presented in Figure 1 It consists of three components:
Backbone block. The main body block, consisting of varying number of DNN layers sub-blocks. The goal of this component is to transform the input into a latent representation that is then provided input the other components. It important to note that PIVEN supports any architecture type (e.g., dense, convolutions) that can be applied to a regression problem. Moreover, pre-trained architectures can also be used very easily, with PIVEN being added on top of the architecture for an additional short training. For example, we use pre-trained VGG-16 and DenseNet architectures in our experiments.
Upper & lower-bound heads. L(x) and (z) produce the lower and upper bounds of the PI respectively, such that Pr(L(x) S y(x) S U()) 2 a where y(x) is the value prediction and 1 - a is the predefined confidence level.
Auxiliary head. The auxiliary prediction head, v(x), enables us to produce a value prediction. v(x)) does not produce the value prediction directly, but rather produces the relative weight that should be given to each of the two bounds. We define the value prediction using,
y == u. . U + v1-
where E (0, 1). By expressing function of U and L, we (a) bound them together and improve their performance (see Section 4.4) and; (b) ensure that the value prediction falls within the PI.
U(x)
Uxx)
Input
Backbone (e.g. VGG)
y(x)
v(x)
Lxx)
L(x)
Pryy e (1, (1, 2 (1-4)
Figure 1: The PIVEN schematic architecture
The goal of the loss function of our approach, denoted as LPIVEN. is to balance two conflicting goals: generating tight PIs and producing accurate value predictions. LPIVEN achieves this goal by synchronizing two loss functions that are designed assess each the two goals described above.
4.2 NETWORK OPTIMIZATION
Our goal is to generate narrow PIs, measured by MPIW, while maintaining the desired level of coverage PICP = a. However, PIs that fail to capture their respective data point should be encouraged to shrink further. We follow the derivation presented in Pearce al. (2018) and define captured M P/W (MPTW capt) as the M //W of only those points for which ys S U.
MPIW capt ==
Σ 1=1
UU: L2) k;
(4)
3

---

Under review as a conference paper at ICLR 2021
where Ei= hgs Hence, seek to minimize MPIW capt subject PICP 2 Oz 0 = arg min PPW PIWcept))) s.t PICPa 2 1
where is the parameters of the neural net. To enforce the coverage constraint, we utilize a variant of the well-known Interior Point Method (Potra & Wright 2000), resulting in unconstrained loss:
Lp = MPIW capt,0 + vn AL PICP) (x) ma)(0,x)²
where is a quadratic penalty function, n is the batch size which was included because a larger sample size would increase confidence in the value of PICP (thus increasing the loss) and À is hyperparameter controlling the relative importance of width vs. coverage. We use a default value of 15 in all our experiments, and perform further analyze of this parameter in Section 6.3
In practice, optimizing the loss with a discrete version of k (see eq. 4 fails to converge, because the gradient is always positive for all possible values. We therefore define continuous version of k, denoted as Kzoft o (y L)) @ (s (U - y)), where a is the sigmoid function, and > 0 softening factor. The final version of Lpr uses the continuous and discrete versions of k in its calculations of the PTCP and MP/W capt metrics, respectively. The discrete k enables us to assign a score of zero to points outside the interval, while k produces continuous values that enable gradient calculations.
Neural networks optimized by the abovementioned objective are able to generate well-calibrated Pls. However, they have two significant drawbacks-overfine
and lack of value prediction capabilities.
Overfitting. The M M/W capt term in Lpr focuses only on the fraction e of the training set where the data points successfully captured by the PI. As result, the network is likely to overfit to subset of the data on which it was able to perform well in the first place. Our reasoning is supported by experiments in sections and
Lack of value prediction capabilities. In its current form, LPs is not able to perform value prediction, i.e., returning accurate prediction for the regression problem. To overcome this limitation, one can return the middle of the PI, as done in Pearce et al. (2018) and Tagasovska & Lopez-Paz (2019). This approach performs well only if the dataset value distribution is uniform or Gaussian (i.e., where choosing the mean minimizes prediction loss). For skewed distributions, as is often the case in the real world, this approach leads sub-optimal results, as we show in Section 5.4
We propose a novel loss function that combines the generation of both PIs and value predictions. To optimize the output of (2x) (the auxiliary head), we minimize the standard regression loss,
C. Σ a(v: U. (1 - vx) Li, vs)
(5)
where is a regression objective against the ground-truth. Our final loss function is convex combination of LpI. and the auxiliary loss L. Thus, the overall training objective is:
LPIVEN 3Cpr (1 B)C,
where 3 is hyperparameter that balances the two goals of our approach: producing narrow PIs and accurate value predictions. In all our experiments, we chose assign equal priorities to both goals by setting 3 = 0.5. We performed no hyperparameter optimization.
4.3 USING ENSEMBLES TO ESTIMATE MODEL. UNCERTAINTY
Each ensemble model produces a Pl and a value prediction. These outputs need to be combined into single PI and value prediction, thus capturing both the alcatoric and parametric uncertainties.
While PIs produced by a single PIVEN architecture could be used to capture alcatoric uncer- tainty, ensemble of PIVEN architectures work, we consider the aggregation proposed by Lakshminarayanan et al.] (2017), to follow fair evaluation as Pearce et al. (2018): Given an ensemble of m NNs trained with LPIVEN. let Ü. L represent the Pl, and 0, represent the ensemble's auxiliary and value prediction. We calculate
capture the uncertainty of the PI itself. In this
4

---

Under review as a conference paper at ICLR 2021
the Pl uncertainty and use the ensemble to generate the PIs and value predictions follows:
U. 1 T
m Σ
Σ U1
(7) OpT o..
(U U:)²
(9)
m j=1
m
j=1
Ü. U, 2a/2 ou;
(8) y:
Vij Uij + (1 (1-1i)) Lij (10)
m
j=1
where (Uny Liy) and Biy represents the PI and the auxiliary prediction for data point for NN j. A similar procedure followed for L,, subtracting 2a/2 OL,, where 2a/2 is the score for confidence level 1 a.
found the above to perform satisfactorily in our experiments. However, PIVEN could potentially be further improved by incorporating aggregation methods such Split Normal Mixture (SNM) (Wallis 2014). We consider this venue as future work.
4.4 DISCUSSION OF CONTRIBUTIONS
PIVEN differs from previous studies two important aspects. First, our approach the first to propose an integrated architecture capable of producing both PIs and well-calibrated value predictions an end-to-end manner, without using post-hoc steps. Moreover, since the auxiliary head produces predictions for all training set samples, it prevents PIVEN from overfitting to data points which were contained in their respective PIs, which is a possible problem for studies such as Pearce et al. (2018). This ability improves PIVEN's performance, shown in Section 5
PIVEN's second differentiating aspect is its method for producing the value prediction. While previous studies either provided the middle of the PI Pearce et al. 2018 Tagasovska & Lopez-Paz
2019) or the mean-variance (Lakshminarayanan et al.) 2017) 7) as their value predictions, PIVEN's auxiliary head can produce any value within the PI, while also being distribution free (i.e., without making any assumptions regarding data distribution within the PI). By expressing the value prediction function of the upper and lower bounds, the prediction always falls within the PI. Our use of non- Gaussanns observation model enables us to produce predictions that are not in the middle of the interval and create representations that are more characteristic of many real-world cases, where the points within the PI is not necessarily uniformly distributed. Our experiments, presented in Sections 5.4 (55
ensure that the three heads are synchronized and that
5.5 support our conclusions.
5 EVALUATION
5.1 DATASETS
UCI Datasets. We conduct our experiments on set of benchmark datasets used by several state-of- the-art studies (Gal & Ghahramani, 2016, Hernández-Looatt & Adams 2015 Lakshminarayanaa
ct al. 2017 7, Pearce et al. 2018). This benchmark includes ten datasets (Asuncion & Newman (007).
IMDB age estimation dataset. The IMDB- WIKI dataset (Rothe et al. 2018) is currently the largest age-labeled facial dataset available. Our dataset consists of 460,723 images from 20,284 celebrities, and the regression goal is to predict the age of the person in the image. It is important to note that this dataset is known to contain noise (i.e., alcatoric uncertainty), thus making it highly relevant to this study (despite usually being used for pre-training).
RSNA pediatric bone age dataset. This dataset is a popular medical imaging dataset consisting of X-ray images of children's hands (Halabi al., 2019). The regression task is predicting one's 's age from one's 's's bone image. The dataset contains 12,611 training images and 200 test set images.
While the first group of datasets enables to compare PIVEN's performance recent state-of-the thrtt--art studies in the field, the two latter datasets enable us to demonstrate that our approach is both scalable and effective multiple types of input.
5

---

Under review as a conference paper at ICLR 2021
5.2 BASELINES
We compare our performance to two top-perforning NN-based baselines from recent years:
Quality driven PI (QD) (Pearce al., 2018). This approach produces PIs that minimize smooth combination of the PICP/MPIW metrics without considering the value prediction task in its objective function. Its reported results make this approach state-of-the-art in terms of PI width and coverage.
Deep Ensembles (DE) Lakshminarayanan et al. 2017). This work combines individual condi- tional Gaussian distribution with adversarial training, and uses the models' variance to compute prediction intervals. Because DE outputs distributions instead of Pls, we first convert it to Pls, and then compute PICP and MPIW. Its reported results make this method one of the top performers with respect to the RMSE metric (i.e., value prediction).
By comparing PIVEN to these two baselines, we are able to evaluate its ability simultancously satisfy the two main requirements for regression problems in domains with high certainty.
5.3 EXPERIMENTAL. SETUP
Throughout our experiments, we evaluate our two baselines using their reported deep architectures and hyperparmeters. For full experimental details, please see Appendix A We ran our experiments using a GPU server with two NVIDIA Tesla P100. Our code implemented using TensorFoow and Keras (Abadi al. 2016 Chollet et al. 2015), and made available online]]
UCI datasets. We use the experimental setup proposed by Hernández Lobato & Adams (2013), which was also used by our baselines. Results averaged 20 random 90%/10% splits of the data, except for the "Year Prediction MSD" and "Protein", which were split once and five times respectively. Our network architecture is identical to that of our two baselines: one hidden layer with ReLU activation function (Nair & Hinton| (2010), the Adam optimizer (Kingma & Ba 2014), and ensemble size M 5. Input and target variables are normalized to zero mean and unit variance.
IMDB age estimation dataset. We use the DenseNet architecture (Huang et al. 2017) as the backbone block, then add two dense layers. We apply the data preprocessing used in Yangg et al.. 2018). We report the results for 5-fold cross validation.
RSNA bone age dataset. We use the VGG-16 architecture (Simonyan & Zisserman 2014) as the backbone block, with weights pre-trained on ImageNet. We add two convolutional layers followed by a dense layer, and perform additional training. This dataset has 200 predefined test images.
5.4 UCI DATASETS EVALUATION RESULTS AND ANALYSIS
We use two evaluation metrics: MPIW and RMSE, with results presented in Table I The desired coverage, measured by PICP, was set to 95%. In terms of PI-quality, PIVEN outperforms QD in nine out of ten datasets (although method reached the required PICP in the "Boston" and "Conerete" datasets, and therefore not in bold for DE), while achieving equal performance in the remaining dataset, DE trails behind PIVEN and QD in most datasets, which is to be expected since this approach does not attempt to optimize MPIW. For the RMSE metric, it is clear that PIVEN and DE are the top performers, with the former achieving the best results in five datasets, and the latter in four. The QD baseline trails behind the other methods in all datasets but one. QD's performance is not surprising given that the focus of this approach PI generation rather than value prediction. We also perform additional experiments for DE with ensemble size M = 1. See full results in T the appendix.
The results of our experiments clearly show that PIVEN capable of providing accurate value predictions for regression problems (i.e., achieving competitive results with the top-performing DE baseline) while achieving state-of-the-art results uncertainty modeling by the use of Pls. Please note that higher a values (i.e., lower PICP) enable our model to significantly outperform the baselines in other metrics. Please see our analysis in Section 6.2
we describe rationale in expressing the value prediction as function of the upper and lower bounds of the interval. To further explore the benefits our auxiliary head v, evaluate two variants of PIVEN. In the first variant, denoted POO (point-only optimization),
Analysis. In Section 4.4

---

Under review as a conference paper at ICLR 2021
decouple the value prediction from the Pl. The loss function of this variant is LpI + E(v, Vtrue) where l is set to be MSE loss. In the second variant, denoted MOI (middle of interval), the value prediction produced by the model is always the middle of the PI (in other words. v is set to 0.5). While both return the middle of the Pl as point prediction, MOI differs from QD by an additional component in its loss function.
The results of our analysis are presented in Table 2 which contains the results of the MPIW and RMSE metries (the PICP values are identical for all variants and are therefore omitted-see full results in Appendix [B. The full PIVEN significantly outperforms the two other variants. This leads us to conclude that both novel aspects of our approach- the simultaneous optimization of PI-width and RMSE, and the ability to select any value on the PI as the value prediction contribute to PIVEN's performance. Finally, is important to note that while inferior to PIVEN, both the POO and MOI variants outperform the QD baseline in terms of MPIW, while being equal or better for RMSE. These results, and in particular the comparison of POO to PIVEN with respect to their RMSE values, provide a clear motivation for deploying the full version of PIVEN.
Table 1: Regression results for benchmark UCI datasets comparing PICP, MPIW, and RMSE. Top performance defined as in Pearce et al. (2018): every approach with PICP > 0.95 was defined as best for PICP. For MPIW, lowest value was best. If PICP 2 0.95 for neither, the largest PICP was best, and MPIW was only assessed if the one with larger PICP also had smallest MPIW.
PICP QD
MPTW QD
RMSE
Danasets
DE
PIVEN
DE
PIVEN
DE
QD
PIVEN
Boston
0.87 + 0.01
0.93 + 0.01
0.93
0.01
0.87 0.03
1,15 = 0.02
1.09 : 0.01
2.87 0.19
3.39 + 0.26
3.13 = 0.21
Concrete
0.92 1: 0.01
0.93 1 0.01
0.93
2: 0.01
1.1
1 0.02
1.08 = 0.01
1.02 2 0.01
5.21 1 0.09
5.88
1: 0.10
5.43 = 0.13
Energy
0.99 + 0.00
0.97
0.01
0.97 --7 0.00
0.49
0.01
0.45 0.01
0.42 0.01
1.68 + 0.96
2.28 + 0.04
1,68 0.03
Kin8om
0.97 +: 0.00
0.96 +: 0.00
0.96 0.00
1,14 + 0.01
1.18
0.00
1,10
0.00
0.08 + 0.00
0.08 +: 0.00
0.07 0.00
Naval
0,98 1: 0.00
0.97 1: 0.00
0.98
: 0.00
0,31 : 0.01
0.27
- 0.00
0.24
= 0.00
0.00 :: 0.00
0.00 1: 0.00
0.00
0.00
Power
0.96
00
0.96 :: 0.00
0.96 0.00
0.91 0.00
0.86
0.00
0.86
0.00
3.99 : 0.04
4.14 1: 0.04
4.08 0.04
Protein
0.96 + 0.00
0.95 0.00
0.95
0.00
2.68 0.01
2.27
0.01
2.36 0.01
4.36 0.02
4.99 + 0.02
4.35 0.02
Wine
0.90 1: 0.01
0,91
: 0.01
0.91
: 0.01
2.50 : 0.02
2.24
= 0.02
2.22
0.01
0.62 : 0.01
0.67 1: 0.01
0.63 0.01
Yacht
0,98 1 0.01
0.95 0.01
0,95 0,01
0.33 0.02
0,18
0.00
0.17
0.00
1.38 1 0.07
1,10 0.06
0.98 0,07
MSD
0.95 NA
0.95 NA
2.91
2.45
NA
2,42 NA
8,95 NA
9.30 NA
8,93
Table 2: Analysis comparing MPIW and RMSE. Results were analyzed as in Table
MPIW MO1
RMSE MOI
Datasets
POO
PIVEN
POO
PIVEN
Boston
1.09 0.02
1.15 1 0.02
1.09
0.01
3.21 1 0.24
3.39
1: 0.27
3,13 1 0,21
Concrete
1.02
1.07
1.02
5.55
5.73
0.10
5
Energy
0.42 0.01
0.45 1: 1.011
0.42
: 0.01
2.16
1 0.04
2.27
1: 0.04
1.65
: 0.03
Kinknm
1.13
1.17 + 0.00
1.10 0.00
0.08
+ 0.00
0.08
+ 0.00
0.07
0.00
Naval
0.24 0.00
0.30 = 0.02
0.24
: 0.00
0.00 +: 0.00
0.00
+: 0.00
0.00
= 0.00
Power
0.86 0.00
0.86 2 0.00
0.86
: 0.00
4.13 1 0.04
4.15 1: 0.04
4.08 : 0.04
Protcin
2.25
2.26
4.78
4.35
Wine
2.24 0.01
2.23 1 0.01
2.22 1: 0.01
0.64 1 0.01
0.67 1: 0.01
0.63 1. 0.01
Yacht
0.18 0.00
0.19 + 0.01
0.17 + 0.00
0.99 + 0.07
1,15 + 0.08
0.98 0.87
MSD
9,10
9.25 NA
5.5 LARGE-SCALE IMAGE DATASETS
We now evaluate PIVEN on two large and noisy image datasets: bone age and age estimation. We add an additional baseline - denoted as NN of a dense layer on top of an existing architecture (DenseNet/VGG. see Section 5.3). that outputs value prediction using the MSE metric. In doing so, we follow the approach used in Qiu et al. (2019) for similar evaluation.
Our results are presented in Table 3 We use mean absolute error (MAE), which was the datasets' chosen metric. For the IMDB age prediction dataset, results show that PIVEN outperforms both baselines across all metrics. It is particularly noteworthy that our approach achieves both higher coverage and tighter PIs compared to QD. We attribute the significant improvement in MPIW 17% to the fact that this dataset has relatively high degrees of noise (Yang et al. 2018). In the bone age dataset, PIVEN outperforms both baselines in terms of MAE Our approach fares slightly worse compared to QD the MPIW metric, but that is likely due to its higher coverage (i.e., PICP).
Our results support our hypothesis that for large and gigh--imensional data (and in particular those with high degrees of noise), PIVEN is likely to outperform previous work due to its ability combine
7

---

Under review as a conference paper at ICLR 2021
value predictions with PI generation. PIVEN produces tighter Pls and place the value prediction more accurately within the PI. A detailed analysis of the training process in terms of training/validation loss, MAE, PICP and MPIW is presented in Appendix E and further supports conclusions. Please note that a major reason for DE's high performance is the fact that population age distribution is Gaussian, and is therefore fully compatible with this baseline's assumptions. PIVEN is able to obtain comparable results to DE without relying on any distributional assumptions.
Table 3: Results on the RSNA bone age and IMDB age estimation datasets.
Datasets
Method
PICP
MAE
NN
NA
PIVEN
0.93
2.09
18.13
Bone age
QD
0.9
1.99
20.24
DE
0.93
NN
NA
7.08 0.03
IMDB age
PIVEN
0.95
0,01
2.87 0.04
7.03 0.04
QD
0.92
0.01
3.47
0.03
10.23 :: 0.12
DE
0.95 0.01
2.61 0.05
6.66 0.06
Additionally, we evaluated PIVEN's performance under dataset shift, using the Flight Delay dataset (Hensman al. 2013). Our results presented in Appendix C
DISCUSSION
6.1 VALUE PREDICTION IN NON-UNIFORM PI DISTRIBUTIONS
As noted in Pearce et al. (0188) the QD baseline's 's approach of producing value predictions inside the PI "breaks the distribution free assumption". This means that for value distributions that neither uniform nor Gaussian, QD's approach of returning the middle of the interval as the value is likely sub-optimal. To test this hypothesis, we evaluate PIVEN and QD on the Sine and skewed-normal distributions. The results, presented in Figure 2 (for skewed-normal) and Appendix A.T (for Sine), clearly illustrate PIVEN's superior ability to adapt to multiple value distributions.
PIVEN
QD
1.75
4C30 fourd
1.72
1.50
1.50
100
0.75
850
0.50
22
0.25
0.00
0.00
2.0
1.5
2.0
-0.5
0.0
0.5
10
15
2.0
2.0
-35
2.0
0.5
0.0
0.5
1.0
2.0
Figure 2: Value prediction comparison, where y is sampled from skewed-normal distribution f(z;a) 20(z)4(a .r) where o(x) and 4aa I) denote the N/ 1) density and distribution function respectively; the parameter denotes the skewness parameter.
6.2 PI PERFORMANCE AS FUNCTION OF COVERAGE
In Section 4.4 we argue that PIVEN's auxiliary head has the secondary effect of enabling it to train on the entire training set rather than only on the data points captured within their Pls. We argue that this property makes PIVEN more robust compared to current SOTA benchmarks. We explore the effects of capturing a smaller percentage of data points (i.e., smaller coverage) on the performance of both PIVEN and QD. Given that QD ignores such data points, it will optimize its performance for a smaller training set. A summary of the MPIW results is presented in Table 4 and full results in Appendix D As the coverage decreases, PIVEN's relative performance to QD continually improves. Interestingly, PIVEN's s RMSE performance remains unchanged (Figure B). The results clearly support our hypothesis regarding PIVEN's training robustness to points outside the PI.
8

---

Under review as a conference paper at ICLR 2021
YearPredictionMSD
alpha
Datasets
0.05
0.15
0.20
0.25
0..00
RVM
Boston
7%
6%
7%
8%
8%
HANSE
4.3
Concrete
7%
7%
8%
7%
12%
13%
Energy
6%
12%
8%
17%
19%
27%
9.0
Kin8om
5%
9%
19%
13%
15%
Naval
16%
14%
17%
23%
0.10
0.25
0.30
Power
0%
0%
1%
1%
1%
1%
yacht
Protcin
1%
1%
25
3%
2%
2%
Q3
Wine
0%
2%
2%
4%
4%
RVEN
Yacht
14%
20%
20%
21%
3005E
4.0
MSD
1%
-3%
-3%
1%5
-5%
-2%
Mean
4.5%
6.0%
6,3%
9.1%
9.1%
11.2%
2.0
0.10
0..1
0.20
0.25
0.55
Table 4: MPIW improvement in percentages (PIVEN relative to QD) as the coverage decreases (i.e, alpha increases).
appha
Figure 3: RMSE function of on of the UCI datasets. See Appendix D for full results.
6.3 HYPERPARAMETERS INFLUENCE
We now analyze the effects of selecting various values to the two parameters that govern the behavior of our approach: B and A. We analyze the behavior of our approach using the Sine function, whose challenging any PI-producing approach.
value fluctuations
The goal of 33 is to balance the two goals of our approach: value predictions. For large values of this parameter, we expect PIVEN put greater emphasis on optimizing the PI at the expense of the value prediction. For small values of B, we expect the opposite. Figure (Left) confirms our expectations. It is clear that 3 0.99 produces the most accurate Pls, but its value predictions is the least accurate. The opposite applies for 3 = 0.1, which produces the most accurate value predictions but whose PIs are the least accurate. This analysis also shows that B = 0.5 strikes better balance than the other settings, as it optimizes for both tasks simultaneously.
producing narrow PIs and accurate
The goal of À is to balance the goal of capturing as many samples as possible within our PIs (PICP) with the conflicting goal of producing tight a PI as possible for each sample (MPIW). We expect that for small values of X, PIVEN will attempt to produce tight PIs with less emphasis on coverage. For large À values, we expect the opposite. Results in Figure our expectations.
Influence
Influence on the PI
Value
2=0:
2-05
-20 -15
-10
-d.5
00
c5
L0
15
25
-2.0
-15
-1.0
-05
02
05
-20
-15
-100
-05
0.
0.5
13
15
Figure 4: (Left) An analysis of the effect of varying 3 values on PIVEN's performance in terms of PI width and value prediction. (Right) The effect of various À values on the tightness of the
CONCLUSIONS
We present PIVEN, a novel architecture for combining the generation of prediction intervals together with specific value predictions in an end-to-end manner. By optimizing for these two goals simulta- neously we are able to produce tighter intervals while at the same time achieving greater quality in value predictions. PIVEN's performance is enhanced by two factors. First, its value-prediction and Pl-prediction are connected, making it better at modeling non-uniform distributions in the PI. Secondly, PIVEN does not ignore but rather trains data points that not captured by the PI.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX) Symposium Operating Systems Design and Implementation ({OSDI) 16), pp. 265-283, 2016.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firnrr, Beat Flepp, Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End end learning for self-driving cars. arXiv preprinl arw:i:::044000000 2016.
François Chollet et al. Keras. https: /atpss///kras...0 2015.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Yonatan Geifman, Guy Uziel, and Ran EI- Yamiv. Bias-reduced uncertainty estimation for deep neural classifiers. arXiv preprint arXiv:1805.08200 2018.
Safwan Halabi, Luciano M Prevedello, Jayashree Kalpathy-Cramer, Artem B Mamonov, Alexander Bilbily, Mark Cicero, Ian Pan, Lucas Araújo Pereira, Rafael Teixeira Sousa, Nitamar Abdala, et al. The rsna pediatric bone age machine learning challenge. Radiology, 290(2):498-506 2019.
Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian deep ensembles via the neural
tangent kernel. Advances in Neural Information Processing Systems, 33, 2020.
James Hensman, Nicolo Fusi, and Neil D Lawrence. Gaussian processes for big data. arXiv preprint a3X9v:/309.660 6835, 2013.
José Miguel Hernández- Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International Conference Machine Learning, pp. 1861-1869, 2015.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 502.0115 2015.
Elia Kaufmann, Antonio Loquercio. Rene Ranftl, Alexey Dosovitskiy, Vladlen Koltun, and Davide Scaramuzza. Deep drone racing: Learning agile flight in dynamic environments. arXiv preprint 1806001806.88588, 2018.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? In Advances in neural information processing systems, pp. 5574-5584, 2017.
Gil Keren, Nicholas Cummins, and Björn Schuller. Calibrated prediction intervals for neural network regressors. IEEE Access, 6:54033-544 54041. 2018.
Abbas Khosravi, Saeid Nahavandi, Doug Creighton, and Amir F Atiya. Lower upper bound estimation method for construction of neural network-based prediction intervals. IEEE transactions on neural nefworks, 22(3):337-34 2010.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv.1412.6980, 2014.
Danijel Kivaranovic, Kory D Johnson, and Hannes Leeb. Adaptive, distribution- free prediction intervals for deep neural networks. arXiv preprint arXiv:1905, 1904./0644. 2019.
10

---

Under review as a conference paper at ICLR 2021
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in neural information processing systems. pp. 6402-6413, 2017.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computa- tion, 4(3):448-472, 1992.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814,
2010.
David Nix and Andreas Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 ieee international conference on neural networks (ICNN"94), volume 1, pp. 55-60. IEEE, 1994.
Khusnul Novianingsih and Rieske Hadianti. Modeling flight departure delay distributions. In 2014 International Conference on Computer, Control, Informatics and Its Applications (IC3INA), pp. 30-34. IEEE, 2014.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, pp- 13991-14003 2019.
Tim Pearce, Mohamed Zaki, Alexandra Brintrup. and Andy Neely. High-quality prediction intervals for deep learning: distribution-fiet ensembled approach. arXiv preprint arXiv: 1802.07 167, 2018.
Florian Potra and Stephen Wright. Interior-point methods. Journal of Computational and Applied Mathematics, 124(1-2):281 2000.
Xin Qiu, Elliot Meyerson, and Risto Miikkulainen. Quantifying point-predicctinn uncertainty in neural networks via residual estimation with an i/o kernel. arXiv preprint arXiv:I906 190..00890008888 2019.
Muhammad Imran Razzak, Saeeda Naz, and Ahmad Zaib. Deep learning for medical image process- ing: Overview, challenges and the future. In Classification in BioApps, pp. 323-350. Springer, 2018.
Yaniv Romano, Evan Patterson. and Emmanuel Candes. Conformalized quantile regression. In Advances in Neural Information Processing Systems, pp. 3538-3548, 2019.
Rasmus Rothe, Radu Timofte, and Lue Van Gool. Deep expectation of real and apparent age from single image without facial landmarks. International Journal of Computer Vision, 126(2-4): 144-157, 2018
Tárik S Salem, Helge Langseth, and Heri Ramampiaro. Prediction intervals: Split normal mixture from quality-driven deep ensembles. In Conference on Uncertainty in Artificial Intelligence, pp. 1179-1187. PMLR, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14099 1556, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 2014.
Natasa Tagasovska and David Lopez-Paz. Single- model uncertainties for deep learning. In Advances in Neural Information Processing Systems, pp. 6414-6425. 2019.
Kenneth F Wallis. The two-piece normal, binormal, or double gaussian distribution: its origin and rediscoveries. Statistical Science, pp. 106-112, 2014.
Tsun-Yi Yang, Yi-Hsuan Huang, Yen- Yu Lin, Pi-Cheng Hsiu, and Yung- Yu Chuang. Ssr-net: A compact soft stagewise regression network for age estimation. In IJCAI, volume 5, pp. 7, 2018.
Yunxuan Zhang, Li Liu, Cheng Li, et al. Quantifying facial age by posterior of age comparisons. arXiv preprint arxiv: / 7708 0968899777... 2017.
11

---

Under review as a conference paper at ICLR 2021
A EXPERIMENTAL SETUP
In this section we provide full details of our dataset preprocessing and experiments presented in the main study. Our code is available online
A.1 SYNTHETIC DATASETS
For the qualitative training method comparison, QD vs. PIVEN (Section 6. all NNs used ReLU activations and 100 nodes in one hidden layer. Both methods trained until convergence using the same initialization, loss and default hyperarameters, as described in Pearce et al.) (2018): 0.5, À 15.0, 160.0, and the random seed was set to 1. The generated data consisted of 100 points sampled uniformly from the interval [-2 2]. We used a = 100 as the skewness parameter. In Figures 5 and we present comparison on the Sine and skewed-normal distributions.
0.05, 8
PIVEN
QD
1.0
2.0
1.5
05
0.0
00
-4.5
-05
-15
-1.
-10
-25
-15
-1.0
-0.5
0.2
0.5
1.0
15
2.
-20
-2.0
-15
-1.0
-55
0.0
05
100
15
20
Figure 5: Comparison of value prediction for QD PIVEN method, where y was generated by
f()) = 1.5 (2).
PIVEN
QD
1.75
150
1.5
125
1.24
1.0
0.75
0.50
0.50
0.25
0.00
-10
0.2
15
-15
-.0
0.5
0.0
1.0
Figure Value prediction comparison, where y sampled from akewed-norma distribution ((;;a)= = 26(x)) .)) where Φ(x) and Φ x) denote the No0, 1) density and distribution function respectively; the parameter a denotes the skewness parameter.
A.2 DATASET PREPROCESSING AND FULL EXPERIMENTAL SETUP
In addition to the ten benchmark datasets used by all recent studies in the field, we evaluated PIVEN on two large image datasets. Due to the size of the datasets and the nature of the domain, preprocessing was required. We provide the full details of the process below.
UCI datasets. For the UCT datasets, we used the experimental setup proposed by Hernández-Lohato & Adams (2015). which was also used in all the two baselines described in this study. All datasets were averaged on 20 random splits of the data, except for the "Year Prediction MSD" and "protein" datasets. Since "Year Prediction MSD" has predefined fixed splits by the provider, only one run was conducted. For "protein", 5 splits were used, as was done in previous work. We used identical network architectures to those described in previous works: one dense layer with ReLU (Nair & Hinton] 2010), containing 50 neurons for each network. In the "Year Prediction MSD" and "protein" datasets where NNs had 100 neurons. Regarding train/test split and hyperparameters, we employ the same setup as Pearce et al. 2018): train/test folds were randomly split 90%/10%, input and
12

---

Under review as a conference paper at ICLR 2021
target variables were normalized to mean and unit variance. The softening factor was constant for all datasets, s = 160.0. For the majority of the datasets we used À = 15.0, except for "naval", "protein", "wine" and "yacht" where À was set 4.0, 40.0, 30.0 and 3.0 respectively. The value of the parameter & was set to 0.5. The Adam optimizer (Kingma & Ba 2014) was used with exponential decay, where learning rate and decay rate were tuned. Batch size of T00 was used for all the datasets, except for "Year Prediction MSD" where batch size was set to 1000. Five neural nets were used in each ensemble, using parameter re-sampling. The objective used to optimized u was Mean Square Error (MSE) for all datasets. We also tune X, initializing variance, and number of training epochs using early stopping. To ensure that our comparison with the state-of-the-ae baselines is accurate, we first set the parameters of our neural nets so that they produce the results reported in Pearce et al. (2018). We then use the same parameter configurations our experiments of PIVEN.
For the IMDB dataset, we used the DenseNet architecture (Huang) al. 7017) as a feature extractor. On top of this architecture we added two dense layers with dropout. The sizes of the two dense layers were 128 and 32 neurons respectively, with a dropout factor of 0.2, and ReLU activation (Narr & Hinton 2010). In the last layer, the biases of the PIs were initially set to [5.0.-5.0 for the upper and lower bounds respectively. We used the data preprocessing similar to that of previous work (Yang et al. 2018 Zhang et al.] (017): all face images were aligned using facial landmarks such as eyes and the nose. After alignment, the face region of each image was cropped and resized to a 64 x 64 resolution. In addition, common data augmentation methods, including zooming, shifting, shearing, and flipping were randomly activated. The Adam optimization method (Kingma & Ba (014)) was used for optimizing the network parameters over 90 epochs, with a batch size of 128. The learning rate was set 0.002 initially and reduced by a factor 0.1 every 30 epochs. Regarding loss hyperparameters, we used the standard configuration proposed in Pearce et al. (2018): confidence interval set to 0.95, soften factor set to 160.0 and À 15.0. For PIVEN we used the same setting, with 8 0.1. Since there was predefined test set for this dataset, we employed a 5-fold cross validation: each split, we used 20% as the test set. Additionally, 20% of the train set was designated the validation set. Best model obtained by minimizing the validation loss. In QD and PIVEN, we normalized ages to zero mean and unit variance.
3 IMDB age estimation datase
RSNA pediatric bone age dataset For the RSNA dataset, we used the well-known VGG-16 ar- chitecture (Simonyan & Zisserman 2014) as a base model, with weights pre-trained on ImageNet. On top of this architecture, with two CNN layers of 64 and 16 neurons each, two average pooling layers, dropout (Srivastaval al. 2014) with a 0.25 probability, and a fully connected layer with 1024 neurons. The activation function for the CNN layers was ReLU (Nair & Hinton 2010), and we used ELU for the fully connected layer. For the PIs last layer, we used biases of [2.0- 2.0). for the upper and lower bound initialization, respectively. We used standard data augmentation consisting of horizontal flips, vertical and horizontal shifts, and rotations. In addition, we normalized targets to zero mean and unit variance. To reduce computational costs, we downscaled input images to 384 x 384 pixels. The network was optimized using Adam optimizer (Kingma & Bal 2014). with an initial learning rate of 0.01 which was reduced when the validation loss has stopped improving over 10 epochs. We trained the network for 50 epochs using batch size of 100. For our loss hyperparameters, we used the standard configuration like proposed in Pearce al. 2018): confidence interval set to 0.95, soften factor set to 160.0 and x = 15.0. For PIVEN, we used the same setting, with 3 = 0.5.
added batch normalization (loffe & Szegedy (013). attention mechanism
https:: hwt.:://wwwwkkk www..agw..kaag..oooo...o..kk wtt..///aaeeerraaaaaoooooo http:////ww.kkeogaaoo///////e//a
age
13

---

Under review as a conference paper at ICLR 2021
B ANALYSIS FULL RESULTS
We now present the full results of our ablation studies, including PICP, for the ablation variants:
Table 5: Ablation analysis, comparing PICP, MPIW and RMSE. The guidelines for selecting best results are as those used Table
PICP
MPIW MOI
RMSE MOI
Datasets
POO
MO1
PIVEN
POO
PIVEN
POO
PIVEN
Boston
0.01
0.93
1,09
1.15
0.01
3.21
3.39 1:
3.13
Concrete
0.93 1: 0.01
0,93
0.01
0.93
0.01
1.02 0.01
1.07 1: 0.01
1.02
0.01
5.58 4: 0.11
5.73 1: 0.10
5.43
0.13
Energy
0.97 + 10.01
0.97
0.00
0.97 + 0.00
0,42 + 0.01
0.45 + 0.01
0.42 + 0,01
2.16 1166+0..00 0.04
2.27 + 0.04
1.65 + 0.03
KinSom
0.00
0.00
0.96 0.00
1.13
1.17
1.10
0.08
0.00
Naval
0.98 0.00
0,98 0.00
0.98 0.00
0.24 = 0.00
0.30 + 0.02
0.24 + 0.00
0.00 + 0.00
0.00 + 0.00
0.00 + 0.00
Power
0.96 0.00
0.96 0,00
0.96 0.00
0,86 0.00
0,86 :: 0.00
0.86 : 0.00
4.13 0.04
4.15 0.94
4,08 0.04
Protein
0.00
0.00
0.95
2.25
2.27
2.26
4.78
4.99
4.35
Wine
0,91 = 0.01
0,91
:: 0.01
0.91 2 0.01
2.24 = 0.01
2.23 : 0.01
2.22 1 0.01
0.64 : 0.01
0.67 1: 0.01
0.63 = 0.01
Yacht
0.95 1 0.01
0.95
0.01
0.95 0,01
0.18 0.00
0.19 0.01
0.17 0.00
0.99 0.07
1.15 4: 0.08
0.98 0.07
MSD
0,95 NA
0.95 + NA
2.43
NA
2.42
9.10 +
9.25 NA
8.93
Additionally, we tested the Deep Ensembles baseline with an ensemble size of one. We expect the baseline to under-perfong in this setting, and this expectation is supported by the results presented in
Table 6
Table 6: Results on regression benchmark UCI datasets comparing PICP, MPIW, and RMSE. The guidelines for selecting best results are as those used in Table Note: the results for DE are presented for ensemble size of one, denotes as DE-One.
PICP QD
MPTW
RMSE
Datasets
DE-One
PIVEN
DE-One
PIVEN
DE-One
QD
PIVEN
Boston
0.76 :: 0.01
0.93 1: 0.01
0,93 1: 0.01
0.75
0.02
1.15 4. 0.02
1.09 0.001
3.11 0.21
3.39 :: 0.26
3.13
0.21
Concrete
0,87 :: 0.01
0,93 1: 0.01
0.93 1 0.01
0.93 0.02
1,08 0.01
1,02 0.001
5.52
0.13
5,88 0.10
5.43
0.13
Energy
0.01
0.97
0.00
0.02
0.45 0.91
0.42
2.28
KinSom
0.9 = 0.00
0.96 +: 0.00
0.96 : 0.00
1.06 :: 0.01
1.18 0.00
1.10 + 0.00
0.08
+ 0.00
0.08 = 0.00
0.07
0.00
Naval
0.94 = 0.00
0.97 1: 0,00
0.98 : 0.00
0.25 0.01
0.27 0.00
0.24 1: 0,00
0,00
0
0.00
0,00
0,00
0.00
Power
0.96
0.00
0.80
0.86
0,86
4.14
408
Protein
0.95 0.00
0.95 + 0.00
0.95 0.00
2.64 + 0.01
2.27 + 0.01
2.26 0.01
4.43 0.02
4.99 + 0.02
4.35
0.02
Wine
0.86
0.01
0,91 1: 0.01
0.91
0.01
2.34 0.02
2,24 0.02
2.22 : 0.01
0,64 0.01
0.67 0.01
0.63 0.01
Yacht
0.95 0.01
0.95 0.01
0,95
0.01
0,26 0.02
0.18 0.00
0.17 0.00
0.07
1,10
0
MSD
0.95 2 NA
0.95 NA
0.95 1: NA
2.86 2 NA
2.45 = NA
2.42 = NA
9.03 1 NA
9.30 NA
8,93 NA
C EVALUATION UNDER DATASET SHIFT
Dataset shift is a challenging problem in which the dataset composition and/or distribution changes over time. This scenario poses significant segnfficnttchallenngee to the application of machine learning algorithms, because these models tend to assume similar distributions between their train and test sets. Dataset shift scenarios are of particular interest to uncertainty modeling (Ovadia et al., 2019). as it enables researchers to evaluate algorithms' robustness and adaptability. We now evaluate PIVEN's ability to adjust to this challenging scenario.
For our evaluation, we chose the Flight Delays dataset (Hensman et al.) 2013), which is known to contain dataset shift. We train PIVEN and our baselines on the first 700K data points and test on the next 100K test points at 5 different starting points: 700K, 2M (million), 3M, 4M and 5M. This experimental setting fully replicates (both in terms of dataset splits and the neural architectures used in the experiments) the one presented in He al. (2020). The dataset is ordered chronologically throughout the year 2008, and it is evident that the dataset shift increases over time. Our experimental results - using the PICP, MPIW and RMSE metrics are presented in Figure 7 As in our other experiments, present the results for confidence level of 95% (i.e. 0.05).
The results show that PIVEN outperforms the baselines in MPIW metric (a result consistent with our other experiments) while achieving the desried coverage rate. Interestingly, our approach is also competitive in terms of the RMSE metric, although the value fluctuations make difficult to reach a clear conclusion. Neither approach reached the desired PICP level in the first test set, but all were able to do so in consequent experiments. DE was able to achieve slightly higher PICP rates than the other two methods, but its Pl width were larger than the other two methods. In contrast, PIVEN manages to
14

---

Under review as a conference paper at ICLR 2021
achieve the desired PICP levels while producing smaller intervals. It should be noted that the dataset exhibits seasonal effect between 2M and 3M samples, which causes variance in performance. This phenomenon is described in He et al. (2020). We hypothesize that the reason PIVEN outperforms DE in terms of Pl width is due to the fact that the former aims to optimize MPIW, whereas DE aims optimize for negative log-likelihood (Lakshminarayanan
al. (017). Additionally, the Flight Delays dataset is known to have non- Con-Gausiian distribution [Novianingsih & Hadianti 2014) which invalidates the basic assumption applied by DE.
PICP
MPIW
RMSE
1.8
1.7
PIVEN
0.98
QD
1.6
0.96
DE
1.5
5-
1.4
0.92
0.90
2
1.0
700k
2m
3m
4m
5m
700k
2m
3m
4m
5m
700k
3m
4m
5m
Figure 7: Compassion between PIVEN and the baselines in terms of PICP (left), MPIW (center) and RMSE (right) for 0.05.
D FULL RESULTS PI PERFORMANCE AS A FUNCTION OF COVERAGE
We argue that QD only focuses on the fraction of the training set which was successfully captured by the PI. In this experiment study the effect of changes in the coverage levels (represented by a) on the width of the produced both by PIVEN and QD. Additionally, we examine the affect of on the value prediction. In Figures and 10 present our results measured by PICP, MPIW and RMSE - on all UCI datasets. As expected, PIVEN consistently outperforms QD in the above metrics.
PI Coverage. Both methods able achieve the coverage desired by a, which decreases increases, as expected.
1.00
0.53
0535
0569
6951
6.7%
0.72
3.34
2.95
0505
0589
0.00
0.5
230
633
0:00
335
130
002
310
855
630
0.2
3.30
625
0.10
0.15
630
033
0.30
035
031
0.12
3.0
035
033
135
038
012
3.0
035
030
proteis
yacht
YearPredicionMMD
0.00
055
0.59
3.20
0.03
0315
0.00
0539
2.00
0783
$.50
0.03
0.00
0735
3.000
8155
0175
8.75
015
015
0.00
Figure 8: comparing PICP between QD and PIVEN over different values of alpha.
15

---

Under review as a conference paper at ICLR 2021
PI Width. QD's inability to consider points outside the PI clearly degrades its performance. PIVEN, on the other hand, is able to reduce MPIW consistently over all datasets.
boaton
1.10
kin8nm
1.30
3.3
man
0.00
0.43
0.35
0.80
1.3
0.25
0.70
077
0.2
533
0.15
020
022
935 0.30
133
0.0
0.25
123
0.25
0.30
8.85
0.10
832
0.00
035
120
025 CC
power-plant
protein
yncht
YearPredictonM50
1911
9131
2.2
mm
0.19
1-4
1,4
60ha
69%0
89h
Figure 9: comparing MPIW between QD and PIVEN over different values of alpha.
Value prediction accuracy. Due to the way QD performs its value prediction (i.e., middle of the PI), the value prediction is not able to improve, and in fact degrades. Contrarily, PIVEN's robustness enables it to generally maintain (with a slight decrease) its performance levels.
boston
6.3
6.5
concrate
2.4
energy
0.100
kinSam
0.00
naval
0p
6.4
0085
6.2
6.-
5.0
0065
-0.22
5.5
544
1.0
0.00
0002
-0.3
0.00
033
313
0.00
033
333
000
830
0.35
6.00
823
0.30
0.0
330
011
0.33
8.23
0.30
0.75
0.30
8.33
0.20
0.2
8.33
035
0.00
0.35
1.27
0.25
630
power-plant
protein
wine
yacht
YearPradictioNM5
0.0
-Bo
mix
10.0
016
045
064
060
2.2
0.15
636
835
035
3.50
833
6.55
0.20
0.5
6.3
6.22
0.15
0,20
8.5
222
0.25
Figure 10: comparing RMSE between QD and PIVEN over different values of alpha.
E IMDB AGE ESTIMATION TRAINING PROCESS AND ROBUSTNESS TO OUTLIERS
E.1 TRAINING PROCESS
In the following figures we present comparisons of the training progression for PIVEN, QD and NN the MAE, PICP and MPIW evaluation metrics. We used 80% of images the training set while the remaining 20% were used as the validation set (we did not define a test set as we were only interested in analyzing the progression of the training). For the MAE metric, presented in Figure 11] we observe that the values for QD did not improve. This is to be expected since QD does not consider this goal in its training process (i.e., loss function). This result further strengthens our argument that selecting the middle of the interval is often a sub-optimal strategy for value prediction. For the remaining two approaches-.... NN and PIVEN- we note that NN suffers from overfitting, given that the validation error is greater than training error after convergence. This phenomena does not happen in PIVEN which indicates robustness, a result which further supports our conclusions regarding the method's 's robustness.
For the MPIW metric (Figures PIVEN presents better performance both for the validation and train sets compared to QD. Moreover, we observe a smaller gap between the error produced by PIVEN for the two sets validation and training which indicates that PIVEN enjoys greater robustness and an ability to not overfit to subset of the data. Our analysis also shows that for the PICP metric (Figure 13 PIVEN achieves higher levels of coverage.
16

---

Under review as a conference paper at ICLR 2021
Mean absolute error
12
Training
12
10-
Validation
10
Mean absolute
8
6
Training
4
Validation
20
60
40
80
Number of epochs
Number of epochs
Mean absolute error
12
Training
12
10
Validation
10
absolute
8
QD
6
PIVEN
4
NN
2
80
Number of epochs MAE PIVEN
Number of epochs
(d) MAE validation
Figure 11: Comparison of MAE metric in the training process. We observe that the values for QD (b) do not improve, which is expected since QD does not consider value prediction in its loss function. Moreover, we note that NN suffers from overfitting, given that the validation error greater than the training error after convergence. This phenomena do not affect PIVEN, thus providing an indication of its robustness.
4.5
4.5
Training
4.0
Validation
MPIW
3.5
3.0
2.5
Training Validation
2.5
2.0
2.0
20
40
80
20
60
Number epochs
Number of epochs
(a) MPTW QD
(b) MPIW PIVEN
4.5
4.0
MPIW
3.5
2.5
QD
PTVEN
0
40
60
Number of epochs (6) MPTW validation
Figure 12: Comparison of the MPIW metric between QD and PIVEN in the training process. As can be seen, PIVEN significantly improves over QD, and has a smaller gap between training and validation errors.
17

---

Under review as a conference paper at ICLR 2021
1.000
1.000
0.975
Training
0.975
Validation
0.925
0.925
RC
0.900
0.900
0.875
0.875
0.850
0,850
Training
0.825
0.825
Validation
0.800
0.800
20
40
60
80
20
40
60
80
Number epochs
Number epochs
(a) PICP PICPQD
(b) PICP PIVEN
1.000
0.950
0,925
0.900
0.875
QD
0.825
PIVEN
0.800
20
40
60
80
Number of epochs
validation
Figure 13: Comparison of PICP metric between QD and PIVEN in the training process. PIVEN achieves higher coverage when two methods converges.
E.2 ROBUSTNESS OUTLIERS
Since PIVEN is capable of learning from the entire dataset while QD learns only from data points which were captured by the PI, it reasonable to expect that the former will outperform the latter when coping with outliers. In the IMDB age estimation dataset, high or very low age as outliers. Our analysis shows that for this subset of cases, there a large gap in performance between PIVEN and QD. In Figure 14 we provide several images of very young/old individuals and the results returned by the two methods. We can observe that PIVEN copes with these outliers significantly better.
can consider images with very
18

---

Under review as a conference paper at ICLR 2021
True age: 75.0Y
True age: 75.0Y
Predicted age QD: 37.9Y Uncertainty: 12.9Y-62.9Y
Predicted age PIVEN: 78.2Y
Uncertainty: 72.0Y 72.0Y-82.8Y
True age: 14.0Y
True age: 14.0Y
Predicted age QD: 36.4Y
Predicted age PIVEN: 17.6Y
Uncertainty: 15.2Y-57.67
Uncertainty: 9.1Y-26.9Y
H I
C
C
True age: 68.0Y
True age: 68.0Y
Predicted age QD: 37.9Y
Predicted age PIVEN: 75.3Y
Uncertainty: 12.0Y-63.8Y
Uncertainty: 84.8Y-83.3Y
Figure 14: The predictions produced for outliers (i.e., very young/old individuals) by both PIVEN and QD for the IMDB age estimation dataset. The results for QD are on the left, results for PIVEN on the right.
19


Reviews:
[{"review_id": "xlcx_WvrTay", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Quality and Clarity**\n\nWhile the overall message of the paper is clear, the explanation of the method in Section 4 is a bit hard to follow. Specifically it is a bit hard to keep a track of the meaning of the multiple terms in the loss function and the associated hyper parameters (see Queries and Suggestions below).\n\n**Originality and Significance**\n\nThe authors seek to jointly address the problem of making accurate predictions and generating tight prediction intervals by designing a new loss function that combines these two goals. I believe this work can be quite impactful as there are multiple areas where good quality prediction intervals and accurate value prediction are equally necessary.\n\n**Strengths**\n\nThe proposed approach appears principled and seems to give good empirical results in the experiments considered.\n\n**Weaknesses**\n\nThere seem to be multiple hyperparameters ($\\lambda$,$\\beta$) whose choice might affect the performance of the approach. Moreover as this seems to be the first time such a loss function is used in this context there is no evidence on good choices of hyper parameters (other than the values used in the experiments herein). I would highly appreciate an ablation study or some comments on good values of hyper parameters which can guide readers intending to apply the proposed approach to a new dataset.\n\n**Queries and Suggestions**\n\n1. I'm curious about the performance of the ensemble of PIVEN architectures approach. It does not seem to have been used in any of the experiments which is fine since a single model seems to be doing well. But have you tried it on any dataset? What are the settings where the improvement with an ensemble would be significant enough to compensate for the added cost?\n\n2. Why are the MPIW results for DE on Boston and Concrete not in bold since DE seems to be the best in terms of MPIW on the two datasets?\n\n3. Why is only a single dense layer (and not an ensemble) used when applying DE to large scale image datasets? If the backbone is pre-trained then training an ensemble of dense layers should not be too expensive. The current comparison does not seem fair since the NN baseline only has a MSE loss while the PIVEN approach uses multiple regularizers (for good MPIW, PICP etc.)\n\n4. What is the difference between the MOI variant of PIVEN and the QD baseline?\n\nIn addition to responding to the above queries, I would recommend improving the explanation (one suggestion is to write the entire loss i.e. $\\mathcal{L}_{\\text{PIVEN}}$ at the beginning of Section 4 right after Figure 1 and then explain the meaning of each term in the loss, as opposed to the current approach where the terms are introduced first and then the loss is given) and commenting on good choices of the hyper parameters or including an ablation study for the same.\n\n**Comments after Author Response**\n\nI thank the authors for their response. Queries 1,2, and 4 have been adequately addressed. Regarding Query 3, I appreciate the addition of the Deep Ensemble results though I find that the text of Section 5.5 has not been changed to reflect the same. Specifically the paper still says \"For the IMDB age prediction dataset, results show that PIVEN outperforms both baselines across all metrics\". This is now incorrect since there is a third baseline, DE, which appears to outperform/match PIVEN for this dataset. However the explanation that this is because the population age is approximately Gaussian makes sense to me and so keeping in mind the good performance of PIVEN on the other datasets, and the improved explanation and added ablation study for hyper parameters, I recommend accepting the paper as long as the relevant corrections are made in Section 5.5.", "extra_scores": {}, "extra_text": {}}, {"review_id": "g7lB8dF7uk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe submission considers the continuous real-valued regression problems and how to obtain accurate point predictions (specific value prediction in the text) and prediction intervals (the uncertainty of the predictions, given by [lower bound LB, upper bound UB]). The paper proposes a loss function which is the weighted combination of (i) the coverage constraint [proportion of outputs that fall between LB and UB] so that the coverage meets a required level, (ii) the prediction interval width [for those that satisfy (i), their prediction intervals should be tight], and (iii) the prediction loss [which makes sure the predictions match the outputs]. The LB, UB and predictions are parameterised using three separate heads: one for LB, one for UB and one for the prediction weight (which makes sure the prediction stays in between LB and UB. The proposed loss function and parameterisation architecture are evaluated on UCI regression datasets and two age prediction problems using image inputs. \n\nAssessment:\nWhilst I think the submission tackles an important problem that could be of interest to the ICLR community, the novelty and experimental evaluation are limited and thus I do not recommend acceptance. Below are some of my concerns/questions and I appreciate the response from the authors.\n\n1. The key contribution is the paper is the combination of the prediction interval loss and the prediction loss. Each of these losses are not new, for example: the prediction interval loss has been considered by Pearce et al (2018). The UCI regression also do not show strong evidence that the proposed method is better than QD, in contrast to the claim of “state-of-the-art results in uncertainty modeling by the use of PIs”.\n\n2. The use of a separate head for v is also not strongly supported by the UCI results, compared to directly parameterising the prediction (POO) or MOI. In fact, PIVEN is probably better because of the hyperparameter search over \\beta.\n\n3. The contribution of the paper includes the output head architecture and the loss function, so I’d not call this a deep neural network. \n\n4. In several places, the paper alludes to skewed output distributions and how PIVEN can handle this better than alternatives. However, it is not clear from the experiments that this is the case. PIVEN is very poor at predicting the Sine example in the appendix -- could you clarify this? Could the \"skewed distribution\" argument be formalised theoretically? like the PIVEN objective is inspired from another likelihood which can support skewed outputs compared to the usual Gaussian likelihood/L2 loss?\n\n5. it would be good to compare to the post-hoc calibration procedure for regression by Kuleshov et al (2018)\n\n6. Does the ensemble of PIs work in practice?\n\nminor: abstract: read-world -> real-world", "extra_scores": {}, "extra_text": {}}, {"review_id": "VGGStxXFgrF", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new objective function for training regression networks with prediction intervals. The goal is to provide tight confidence bounds to accompany predictions, which is of course important for practical deployments of ML systems where uncertainty quantification is critical. Previous work has largely assumed that uncertainty is symmetrical, or even Gaussian distributed, which is often not the case in practice. The innovation of this work is to simultaneously predict the bounds for a given confidence level and a point prediction within those bounds, which is not necessarily the mean. It is accomplished by predicting 3 values: an upper bound, a lower bound, and a mixing parameter that allows making the point estimate as a weighted sum of the bounds. Experimental results are provided on 3 datasets, with 2 baseline methods compared against.\n\nThe strong points of this paper are:\n+ The problem being addressed is important.\n+ The approach is relatively simple.\n+ The paper is clear and well-written.\n+ The metrics introduced for evaluation make sense.\n+ The results are strong and improve over the baselines.\n+ The choice of how to produce the point prediction is well motivated and backed up by ablation experiments.\n\nWeak points:\n- Since the method produces bounds rather than a distribution, it's necessary to know at training time what confidence levels you are interested in. It also therefore can't be evaluated on standard negative log likelihood metrics, or produce something akin to accuracy vs rejection-fraction plots. This is potentially a big problem for practical deployment, since the tradeoffs of alpha vs RMSE can't be evaluated without retraining, and the same model can't be used for multiple downstream tasks with different confidence requirements.\n- The claims about using PIVEN in an ensemble context don't seem to be experimentally validated anywhere.\n- The NN baseline for large-scale image datasets seems unnecessarily weak. Why not predict both mean and variance (basically DE, but a single model)? That way you could at least compute PICP and MPIW.\n- In table 1 it's not at all clear what the bold numbers are. They don't seem to correspond to the best result in each row.\n- In table 2, why is PICP not provided? It seems like the value should not be identical between the different experiment arms.\n- It would have been nice to see an evaluation under dataset shift, similar to the \"Flight Delays\" experiment in https://arxiv.org/pdf/2007.05864.pdf\n\nOverall, this seems like a simple and valuable technique that addresses an important problem, but its applicability is somewhat limited by the fact that it produces bounds rather than a distribution.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DDxv5gayt63", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method for predicting a prediction interval. The authors compare this method to deep ensembles and show improved performance on a set of benchmarks.\n\n\nThe are several things I like about this paper:\n- The paper is written quite clear \n- Uncertainty calibration is an important topic.\n- A lot of benchmarks are considered\n- The method is easy to implement.\n\nThese are the things  that could be improved:\n\n1. The authors say \"\"First,  our approach is the first to propose an integrated architecture capable of producing both PIs and exact value predictions.\"\n\nThis is not correct.  A simple approach such as Lakshminarayanan et al., 2017 will produces the same, even without using ensembles.  In that work,  the predictions parameterize a Gaussian with state dependent mean and variance. From this quantiles and thus prediction intervals can be derived on top of an \"exact value prediction\" which would be the mean.\n\n2.  One of the papers main weakness is the lack of distinction between epistemic and aleatoric uncertainty:\n\n\nSection 2:\nThe way  Section 3 is formulated it speaks about aleatoric uncertainty:    P(L_1 < y_i < U_i) is the probability that y_i falls into th ebound defined by L_i and U_i and thus is \"data uncertainty\" (aleatoric) and not due to the lack of knowledge given by the model (epistemic).  it is thus similar to a Softmax in classification or  a neural network that predicts both mean and variance (as in  (Lakshminarayanan et al., 2017)). Indeed as Section 4 show,  the model does not contain any parametric uncertainty. \n\n\n3. Results and method comparisons:\n\nSection 4: \nI am not sure why RMSE is a useful metric when proposing a method for uncertainty calibration.  Secondly, as mentioned above the comparison to deep ensembles is not fair, as deep ensembles will also model epistemic uncertainty.  I would just use an ensemble of size 1 for a more accurate comparison (thus excluding epistemic uncertainty). Thirdly, the two other metrics need to be explained (at least write them once without acronyms.)\n\nOverall,  the improvements appear  to be only marginal improvements.\n\n4. Related work:\nHow is this work different from  [1] and why wasn't it compared to this work?\n\n[1] Salem, Tárik S., Helge Langseth, and Heri Ramampiaro. \"Prediction Intervals: Split Normal Mixture from Quality-Driven Deep Ensembles.\" Conference on Uncertainty in Artificial Intelligence. PMLR, 2020.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents PIVEN, a deep neural network that produces a prediction interval in addition to a specific point prediction.  PIVEN is distribution free and does not assume symmetric intervals.  

All the reviewers agree that the paper investigates an important problem and the paper is well-written. The reviewers also identified a couple of weak points, namely:
- Novelty: The key idea seems to a combination of prediction loss (common) and prediction interval loss which has been proposed by Pearce et al. 2018.
- Claims that PIVEN outperforms existing methods (QD and DE) empirically as some of the improvements seem marginal. 

Given these concerns, I think the current version falls a bit short of the acceptance threshold unfortunately. I encourage the authors to revise the draft and resubmit to a different venue.

Decision: Reject