Distribution-Driven Disjoint Prediction Intervals for Deep Learning
Authors: Jaehak Cho
Jae Myung Kim
Seungyub Han
Jungwoo Lee
Venue: ICLR 2022 Submitted (2022)
Keywords: Uncertainty
Prediction Interval
Regression Uncertainty

Abstract: This paper redefines prediction intervals (PIs) as the form of a union of disjoint intervals. PIs represent predictive uncertainty in the regression problem. Since previous PI methods assumed a single continuous PI (one lower and upper bound), it suffers from performance degradation in the uncertainty estimation when the conditional density function has multiple modes. This paper demonstrates that multimodality should be considered in regression uncertainty estimation. To address the issue, we propose a novel method that generates a union of disjoint PIs. Throughout UCI benchmark experiments, our method improves over current state-of-the-art uncertainty quantification methods, reducing an average PI width by over 27$\%$. Through qualitative experiments, we visualized that the multi-mode often exists in real-world datasets and why our method produces high-quality PIs compared to the previous PI. 

Under review as a conference paper at ICLR 2022
DISTRIBUTION-DRIEER DISJOINT UNCERTAINTY Es- TIMATION FOR DEEP LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper redefines prediction intervals (PIs) as the form a union of disjoint in- tervals. PIs represent predictive uncertainty in the regression problem. Since pre- vious PI methods assumed a single continuous PI (one lower and upper bound), it suffers from performance degradation in the uncertainty estimation when the conditional density function has multiple modes. This paper demonstrates that multimodality should be considered in regression uncertainty estimation. To ad- dress the issue, we propose a novel method that generates union of disjoint Pls. Throughout UCI benchmark experiments, our method improves over current state-of-ther -art uncertainty quantification methods, reducing average PI width by over 27%. Through qualitative experiments, we visualized that the multi-mode often exists in real-world datasets and why our method produces high-quality PIs compared to the previous PI.
INTRODUCTION
Deep neural networks (NNs) show remarkable performance in predicting target for regression problems. However, the prediction is not enough to make it trustworthy; minimization of objective functions the NN leads to network outputs which approximate the conditional averages of the target data with no information about sampling errors and prediction accuracy. Moreover, if the target is multivalue, NN output can be far from the actual target in the regression problems. Incorporating the predictive uncertainty into the deterministic approximation generated by NNs improves the re- liability and credibility of the predictions. This issue being discussed in various domains such as autonomous driving (Feng et al. 2018), object detection (He et al.] (2019), solar energy forecasting (Glvván et al. 2017), electricity demands and price estimation (Shrivastava & Panigrahi 2015), and sensor anomaly detection (Pang et al. 2017).
Prediction interval (PI) represents and quantifies predictive uncertainty the regression problem. Pearce et al. (018); Tagasovska & Lopez- Paz (2019); Salem et al. (2020) have recently provided competitive performance by generating a PI to estimate predictive uncertainty. PI describes predic- tive uncertainty for each sample in the form of two values (lower and upper bound) between which potential observation falls with a certain probability (e.g.. 95% or 99%). PI can provide the amount of uncertainty for each sample by the width of PI. It also provides the possible range of prediction by bounds. It is a self-evident principle that high-quality PI should be as narrow as possible while containing some specified proportion of data points (hereafter referred to the HQ principle). The quality of a Pl is often evaluated by the metric derived from the HQ principle (Khosravi et al. 2010
Galván al. 2017 Pearce et al. 2018 Tagasovska & Lopez- Paz 2019 Salem al. 2020))
Previous methods estimate the regression uncertainty with a single continuous PI, but it may suffer from performance degradation in the regression having multimodality. A toy example in Figure D is a one-dimensional regression example that has two modes. We observe that a single continuous P/ (gray shade) provides unnecessarily large PIs to fill in the gap between the two modes compared to disjoint Pls (blue shade). This means that a single continuous PI provides low-quality Pls in terms of the HQ principle. Including intervals that are unlikely to contain future observations makes PIs less reliable. Note that this issue becomes severe the distance between modes increases. We qual- itatively confirmed that multimodality often exists in real-world regression datasets through approx- imating conditional probability density function. We also confirmed that late-of-the-ars methods

---

Under review as a conference paper at ICLR 2022
10
ground dota/train) (SQR)
-3
-10
95% Pl [Ours)
Figure 1: Estimating uncertainty in regression having multimodality. Prediction interval for y with 95% coverage probability drawn for SQR and DDD (our method). SQR estimates continuous prediction intervals, while DDD provides disjoint prediction intervals.
generate low-quality PIs on real-world samples with multimodality. This is covered more detail in Section 5.4
Considering multimodality has been successful at handling the underlying stochastic structure in various fields (Ameijeiras-Alonsa al. 2019 Lerch et al.] 2020). Concerning multimodality, var- ious works such as clustering, multi-object detection (Yoo et al. 2019), missing data reconstruc- tion (Smieja al. 2018), multiple-choice learning (Lee et al. 2017), and multi-output prediction (uzzman Riizaan-Riveraaaa et al. (2014) have been conducted. However, recent regression uncertainty estima- tion studies do not consider multimodality in depth.
In this work, we redefine PI as a union of disjoint PIs due to the limitation of single continuous PI in multimodality (Section 3). Since prior PI methods and loss functions do not apply to the union of disjoint Pls, we propose new differentiable objective function and NN architecture that produce the union of disjoint PIs (Section 4). Additionally, we use the ensemble method to boost the performance for both in- and out-of-distribution regions (Section 5.2 As result, our method improves over current state-of-the-an art methods, reducing an average width by 27% throughout eleven real-world datasets (Section 5.3). In addition, method can provide the coverage probability of each disjoint PI (c.g., 20% chance of being between and 3, 75% chance of being between 5 and 9). This means that our method gives information about how reliable each interval is (Section 5.5
RELATED WORK
There are two approaches for estimating the predictive uncertainty for regression problems: Bayesian and non-Bayesian. In the Bayesian approach, NN parameters are considered as dis- tribution, and the uncertainty is calculated by marginalizing the parameters (Graves) 2011; Blundell
et al. 2015; Hernández- Lobato Adams 2015 Gal al., 2017; Khan et al. 2018, Wu et al.
2018 Yao et al. 2219 Izmailov al.) (020).. Though theoretically grounded, approximation
needed since calculating the posterior distribution of NN parameters is computationally intractable. It also requires high computational demand in the inference time. The non-Bayesian approach, on the other hand, defines the output of NN as parameters to describe the predictive uncertainty. is usually less computational than the Bayesian approach. However, since the NN parameters are fixed, non-Bayesian methods have limitation in expressing the model uncertainty. Therefore, the deep ensemble with random initialization is additionally used deal with model uncertainty. Several papers in the non- non-Bayeen Bayesian branch have recently provided competitive performance (Laksh. minarayanan et al. 2017 Pearce et al. 2018 Tagasovska & Lopez- 2019 Salem et al.) 2020).
Our paper focuses on the Non-Bayesian approach, especially for the regression problem. There- fore, we would take closer look at the non-Bayesian methods by dividing them into PT and non-PI methods.
As PI methods for non-Bayesian methods, Khosravi et al. (010)) propose the Lower Upper Bound Estimation (LUBE) method that produces PI for the first time. Followed by that, Pearce et al. (0118) propose a quality-driven (QD) loss function that is compatible with gradient descent opti- mization. They also propose ensemble method for PI with multiple predicted lower and upper bounds to estimate the model uncertainty. [Salem et al. (2020) retrofit the QD loss function and pro- pose ensemble method by fitting the split normal mixture distribution (Wallis. 2014) to the
2

---

Under review as a conference paper at ICLR 2022
Pl and averaging the distribution, where they name it as SNM-QD+. It increases the robustness of the training process compared to the QD method. However, SNM-QD+ has difficulties searching hyperparameters because the loss function contains various hyperparameters to achieve the advan- tages. (Taaasvvska & Lopez-Parl Paz (2019) propose the simultaneous quantile regression (SQR) and the orthonormal certificates (OC) estimate data noise and model uncertainty, respectively. However, this strategy generates PI only by the SQR without an ensemble method, and model uncertainty from OC not included in the PI. Therefore, the PI of SQR does not consider the model uncer- tainty. Aforementioned loss functions and methods can only generate single continuous P/ but not a union of disjoint Pls.
As non-PI methods. Mean-Variane Variance Estimation (MVE) (Nxx & Weigend 1994) uses a NN with two output nodes that are considered as a mean and a standard deviation of the conditional probability distribution. Since NN parameters are fixed, it cannot deal with the model uncertainty. [Lakshmi-] narayanan al. 2017) demonstrate the deep ensemble multiple MVE with random initialization improves the performance, especially in nut-of-distribution regions. (so-called MVEens). Fort et al. (2019) shows that ensemble with random initialization may sample different modes in function space and therefore perform well in exploring model uncertainty.
3
UNION OF DISJOINT PREDICTION INTERVALS
3.1 PROBLEM SETUP
Consider dataset {æ,, y1 } where æ; an input and yı is target. For each data point {æ,, v. the disjoint set of PIs that covers the desired given proportion y E [0, I] is defined as follows:
Jm
P4
UL4 Un)) 5=1
(1)
where Pr(ys PL.) 2 and Lij S Uij < La(+t) for all j
Lyj and Ul is lower and upper bound of jth PI related with ith data point. ( is the number of disjoint intervals when PI is expressed with the smallest number of disjoint intervals. That is, ( is unique for given interval. Note f(O may have a different value for each data point. The previous methods assume a single continuous P/ that is ((6 1 for all i.
3.2 PERFORMANCE METRIC: PICP AND MPIW
To measure the quality of PI methods based the HQ principle, let Prediction Interval Coverage Probability (PICP) and Mean Prediction Interval Width (MPIW) be defined as,
f1, 0,
if ys E PT; otherwise
PICP N where
C and
(2)
N jm
MPIW = N MV (U. Lij)
(3)
PICP measures the ratio of the target that is captured within PIs while M MPIW measures total length of PIs over the entire samples. According to the HQ principle, Pls should minimize M P/W subject to PICP 2 y (e.g. 7 = 0.95 or 0.99). This metric is widely used to compare performance of PI-related methods (Khosravi et al. 2010: Pearce et al., 2018 Tagasovska & Lopez-Paz 2019
Salem al. 2020).
4 DISTRIBUTION-DRIVENN-
METHOD
We propose a learning-basee method that generates distribution-driven disjoint (DDD) Pls, and we call our method as DDD method. The DDD method produces high-quality PIs without the assump- tion that ((0 1. To generate multiple disjoint PIs with learning-based method, need to
3

---

Under review as a conference paper at ICLR 2022
formulate differentiable loss function that reflects the HQ principle. However, this is a challenging problem because с from (2) is non con-differentiable. Pearce et al. (2018); Salem et al. (2020) proposed QD and QD+ loss function in the form of constraint optimization by approximating in differ- entiable way. Tagasovska & &
Lopez-Paz (2099) employed pinball loss which reflects HQ principle
and differentiable. However, these loss functions have limitation in that they work for single continuous PI which has J( =
To derive a new differentiable loss function (so-called DDD loss) for multiple disjoint intervals, we first approximate the conditional distribution given input, p(g (z), with Gaussian mixture model. Then, we derive the DDD loss by using the cumulative density function of the Gaussian mixture (why call our method distribution-driven disjoint). Our DDD method trains NN by minimizing the DDD loss.
Another major problem is that the optimal number of disjoint prediction intervals J may differ
for each (()]))))) making it hard implement (Taj. Uاz)ي= ,(9)
output of NN. To deal with the problem, after producing K intervals regardless of conditional density, the union process removes the overlapping part. We propose a novel architecture that can implement this in a differentiable way. Additionally, we employed simple ensemble method to improve performance for both in- and out-of distribution observations.
4.1 DERIVATION OF THE DDD LOSS
In this section we derive the DDD loss, L.DDD, that reflects the HQ principle: PIs should minimize MPIW subject to PICP 2 y. LDDD combine LMPIW and LPICP with Lagrangian, X, controlling the importance of width vs. coverage. Each term is defined as:
LDDD = LMPIW + ALrie
(4)
1 N N [=1
N K
LPICP max(0,
a),, Σ
LMPIW = Σ ΣΣ
1
(0 - Lnj)
(5)
N j=1
where é = -
(6)
The coverage probability &; is an approximation of c2 in Equation The function F; is cumu- lative density function of the distribution p(y, (r.)) Then, the coverage probability é is calculated from the cumulative density function F,. LPICP is defined as the mean square eтror with the target proportion y with the max(0, x) operation. The mar(0, x) operation is used to minimize the loss only when the average coverage probability, TV Σ, és is lower than the given proportion 7. The loss function LMPIW is naively derived from definition of M P/W in equation (3)
For Lpdp to be differentiable for Ley and Üsg, è must be differentiable for Lij and Ú.₁- Therefore, F; should be differentiable. We approximate the p(y: (z.) with the mixture density network with Dopccation-Maxmizatte
algorithm (Greff et al.; 2017). This is not distributional assumption because any distribution can be approximated by the Gaussian mixture model when the number of the mixture is sufficient (Bishop 1994 Sung 2004). When parameters of the Gaussian mixture distribution are (Mi, J, } an approximated probability distribution p(y (x:)) is defined
f(EA o.
Σ (x:) N(w o((
where K is the number of mixture components. K can be considered large enough increasing K does not reduce the negative likelihood loss of the mixture density network. Setting K=5 worked well for our experiment in section For brevity, we refer to μ4 (x:) as fak. and same abbreviation applied to dg and mi-
4

---

Under review as a conference paper at ICLR 2022
Algorithm Pseudocode for the DDD Network
1: Definition
2: M4 c. n. Input of NN where size(p.) = a (i,)( sizc(T.. = K. The means, standard deviations, and weights of the Gaussian mixture p(y.
3:
(+)
Output of NN where si size(e, size(c(*)) K, c 0, e 2
4:
Train
5:
L = μ 1---((1 e
and ú = p e
6:
i =[arrgsort())), 0 Ü (argsort(E)
ascending order in k-related dimension
7:
for + 1...... do
8:
for kt + 1... K do
9:
UR Lik + ReLU(Üie - iik)
10:
LU(+1) = Un + ReLU( - Uww if lo #
11:
end for end for
12:
13:
Minimize LDDD = Σ EK
E,n,(Ush (Ü.e - L1s) + A max(0, 7 * +m 4))
where è = = (F(((()) - F((E(n))
4.2 DDD NETWORK ARCHTECTURE
This section describes a neural network architecture that generates instributionddriee--iiifi disjoin PIs (so-called DDD network) as pseudocode Algorithm [ The architecture classified into four blocks according the major objectives and explained in detail. All blocks achieve each objective differentiable form.
Input Generation (line 2) Input of the DDD network [ut oi, π.. because μ4i o,, and π, are used to compute the loss function. The input æ; is converted into [pg o [T] by the mixture density network. For this, is necessary to train the mixture density network before training the DDD net- work. The Neural Exppecation-Maxxiizzaiin algorithm (Greff et al. (017) is used for the training.
Multiple Interval Generation (line 3-5) This block generates K multiple intervals. The last
nodes of NN are squared ensure that the outputs of NN ((c!-) c (+t)) are positive. The primary aspect is that this block restricts each interval to include each peak jAsk of the Gaussian mixture model (line 5). This aspect contributes to producing well-calibrated PIs because it is self-evident that well- calibrated PIs should contain more than one peak. If the output of NN is defined as lower bounds and upper bounds without the restriction, experimentally confirmed that often converges to low-quality PI.
Union: overlap prevention (line 6-12) This block implements the union process a differen- tiable way using sorting and mar(0, 2) functions (the sorting process does not interfere with back- propagation). Through this process, K intervals can be converted to (() blocks. As result, the DDD method provides the union of disjoint Pls.
Minimize LODD (line 13). By training the DDD network to minimize LDDD, the union of dis- joint PIs becomes high-quality. À is hyperparameter that determines how much trade-off between LPICP and LMW. If PICP obtained through training is less than ?, increase This increments PTCP, along with PIW.
4.3 ENSEMBLE METHOD
Optimizing the several NN models with random initialization and then aggregating them is known to be effective for the uncertainty estimation the out-of-distribution region (Lakshminarayanae al. 2017). Since then, recent studies (Pearce et al. 2018: (aalem et al. 2020) have promoted the ensemble of Pls.
We train M different mixture density newtork models for p(y av;) with random initialization fol- lowed by averaging them, fene(r;) x Em. (x))(xx). where (m) is the probability density
5

---

Under review as a conference paper at ICLR 2022
function in equation 7). Then, acheive cumulative density function of the ensembled model from Fe (3;) M E.. ((m)(x;)) Finally, we train the DDD network proposed in 4.2 to achieve estimating the predictive uncertainty. When M models are ensembled, not K but K M7 intervals are generated in the multiple interval generation block.
5
EXPERIMENT
5.1 IMPLEMENTATION DETAIL
Dataset We use UCI regression benchmarking datasets (Dua & Graff, 2017). While previous works use nine datasets (Boston, Energy, Kin8nm, Naval, Power, Protein, Wine, and Yacht) for evaluation, we add two more datasets that are the Cotinnnoo--Tlmmmittrrriinn and Bike-Sharing. For each dataset, the experiment is repeated 20 times with different random seeds except for the Protein dataset, where it is repeated five times. Each dataset split into train, valid, and test set with the ratio of 0.81, 0.09, and 0. with random shuffling for each experiment. The target is standardized to zero mean and unit variance based on the entire data in each dataset. More specifics about data pre-processing are provided in the Appendix A
Methods We compare DDD (ours) with MVE (Lakshminarayanall et al. 2017 and QD+ (Salem al. 2020) where the ensemble method of each of them are named DDD eDDDens. MVE and SNM- QD+, respectively. SQR (Tagasovska & Lopez-Paz (019) participates in the comparison without an ensemble since Tagasovska & Lopez-Paz/ (2019) did not propose an ensemble method for its PI generation. Considering QD+ 4Salem et al. 2020) is retrofit version of QD (Pearce et al.) 2018), QD is not included the comparison. We build the PI of the MVEena method based on its mean and standard deviation like Pearce et al. (2018) did.
Hy perparameeee While larger K (the number of mixtures) is preferable in theory (Sung 2004), it needs more data and increased model complexity in practice. In our method, K 5 sufficient to apply in the UCI datasets. We experimented with the same hyperparameters for all methods as follows: All neural networks have two hidden layers with 50 units except for the Protein dataset, where 100 units are used, with each layer having the rectified linear activation function. We set y = 0.95 for the ratio of PIs to be covered. The ensemble size is set to M = Adam optimizer
(Kingma & Bal (2014) is used for the optimization, and we did not use the dropout (Srivastava et al., 2014) and the weight decay.
For fair comparison, hyperparameters should be set to derive near-optimal performance for each method. The grid search used for the hyperparameter search on the learning rate for MVE and the mixture density network of the DDD method. For the DDD network, we search only on the lambda contained in the loss function (the same learning rate was used as the first phase). However, in the case of SNM-QD+, it is difffiuult to optimize because the loss function includes Therefore, we take the quantitative results from Salem et al. (2020) for
multiple Syper-parameterd the nine existing UCI datasets. For the additional two datasets, run 300 hyperparameter searches with the code provided by the paper's 's authors.
We set À that DDD has PICP similar to other methods. It is easy compare the performance between methods from the perspective of the HQ principle if the PTCP is similar. This is because if PICP is similar, only M MPW needs to be compared. Note that, as À increases, PICP and MPIW increase, and vice versa.
5.2 EFFECT OF THE ENSEMBLE METHOD
Figure presents qualitative results to confirm the effects of ensemble in in-distribution and out- of-distribution. It describes uncertainty estimation of three single DDD models and the ensemble of the three. The blue dotted line means the amount of uncertainty (yticklabels on the right side). Data points are generated through y (0.02.z - 0.02ez) + (1 (z + 0.02e2) where el Bernoulli()) () and 62 ~ NO,032) Training data points are not included in x < -4 and x > 4 to demonstrate the effect of DDD ens those diito---iitribution regions. The models used in Figure 2 are the result of adjusting the À value so that PICP for the validation set is 0.95 + 0.01. Since the data points generated to have homoscedastic noise, the amount of uncertainty in in-distribution

---

Under review as a conference paper at ICLR 2022
single
single
single
ersemble
-30
-3.5
0.0
-78 -1.5 00
1.0
-10
-15 03
35
10
-30 -35
0.
35
70
Figure 2: Qualitative comparison of DDD and DDD DDD is trained with the training dataset (red dots). Gray shade and blue shade represent the estimated prediction intervals of DDD and DDDens ens, respectively. The Blue dotted line represents the width of prediction intervals for each input. DDD shows better calibration both n--distribution and dfstsiiibbbttinnn.
is almost constant. In out-of-distribution, the amount of uncertainty increase toward both ends. In other words, the blue dotted line should have U shape if high-quality. We can observe that the ensemble method produces more tight PIs for the in-distribution region (compared to the left side of #1) and quantifies better the uncertainty for the oui-t--diitriuutiin region (compared to both sides of #1 and the left side of #2 and #3). Put simply, the ensemble model has clearer U shape compared to other single models.
We quantitatively compared DDDens with a single DDD model the UCI benchmark datasets. According to the HQ principle, when PICP is greater than or equal to, and MPIW is smaller, better performance is achieved. For all datasets, we confirmed that DDD ens achieves higher PICP than DDD while having lower to equal M P/W that implies outperformance DDD The comparison result is presented in appendix
Combining all the experimental results, we can conclude that our ensemble method boosts the per- formance for both in- and outtof-distribution..
5.3 BENCHMARKING EXPERIMENT
Table compares DDD MVE SNM-QD+ and SQR on the 11 UCI datasets. For each datasets, we compare DDD with the best performance method among MVE SNM-QD+ and SQR and measure how much MPIW is reduced. Improvement means how much DDDens reduces MPIW. DDD and the best method for each dataset are marked in bold. is better method if the method has higher equal PTCP and smaller PPW. However, if both PICP and M P/W are higher or smaller the same time, comparison is not clear. Therefore, we set the criterion that the best performance method of each dataset is determined to have the smallest M MPW among those with a PICP of 0.94 or higher.
DDD has higher equal PICP and smaller MP/W compared to the best performance meth- ods in 10 out 11 datasets. This confirms that DDD provides well-calibrated PIs for predictive uncertainty estimation compared to the state-of-the-art methods. We can observe minor improve- ments like 1% or 3%, and dramatic improvements like 72% or 93%. It is reasonable to infer that datasets with significant improvement have multimodality, and those with low improvement have unimodality. The reason why DDDens shows better performance and multimodality of the datasets discussed Section 5.4 through qualitative experiment.
5.4 QUALITATIVE EXPERIMENT
We draw P(y ax;) through the mixture density network for the following reasons: 1. to find out whether the conditional probability density of samples multimodal. to ascertain the reason for the outperformance of DDDens- Each plot in Figure is the sample of the Protein, Parkinson, Kin8, and Energy dataset. The light blue line denotes the approximated the conditional density p(y) xx.). Note that, although approximated by Gaussian mixture model, p(y, (z.) can express unimodal distribution (See Kin8 samples and Power-Plant sample #1).
In the Protein and Parkinson samples, observe that our mixture model describes multimodal distribution. DDD provides multiple disjoint prediction intervals covering the target and does not contain less plausible regions than SQR and MVE. In other words, even with much shorter PI,
7

---

Under review as a conference paper at ICLR 2022
Table Quantitative comparison among three different methods.
Method
bike
boston
concrete
energy
kin8
naval
SQR
0.92:10.01
0.84=:006
0.881:0.06
0.84110.04
0.92:10.01
0,95:10.02
MVEgns
0.96=0001
0.95=0002
0.95=0003
0.97=00.22
0.98+0.01
1.00++0.01
PICP
SNM-QI+
0.96+0.00
0.95+0.01
0.94+0.01
0.99+00.00
0.97+00.00
1.00+0.00
DDD
0.96.10.11
0.95-10.03
0.94
0.98:10.02
0.97
1.00-10.00
SQR
1.38:0.10
0.22
0.92-10.16
0.11+:0.02
0.98
0.14:10.02
MVEgas
1.62::0.07
1.43:=0.19
1.88=0.18
0.14
1.11=00.02
0.09:10.01
MPIW
SNM-QD+
1.82+0.11
1.58=00.6
0.99-0004
0.29++0.01
1.07==0.01
0.09+00.00
eDD
1.20
1.21:10.12
0.96
0.16110.01
1.03:10.02
0.06:10.01
IMPROVEMENT
26%
15%
3%
NA
4%
33%
Method
parkinson
power
protein
wine
yacht
SQR
0.90::0.02
0.94
0.941:0.00
0.91
0.82++0.09
MVE
0.9910.00
0.961
0.97+0.00
0.95
0.99::0.02
PICP
SNM-QD+
0.97:10.01
0.95=0.00
0.95
0.94
0.9410.01
DDD
0.9910001
0.96
0.95
0.94.10.02
0.94
SQR
1.79:10.13
0.09
2.25+0005
2.51=10.51
0.08
MVE
1.43:10100
0.84=
2.46+10.07
3.33
0.004
MPIW
SNM-QD+
1.6810.10
0.00
2.02
2.62:10.06
0.12:10.00
DDD 0.40+=0.05
0.79=0002
1.57+0.02
0.18+0005
0.09
IMPROVEMENT
72%
1%
26%
93%
25%
Protein Sample #1
Protein Sample #2
Protein Sample #3
Protein Sample #4
(S(RR)
Pi (DGD)
Parkinson Sample
Sample
Sample
Kina Sample #1
Kinä Sample #2
Kina Sample #3
Kinä Sample #4
Power Sample #1
Power Sample #2
Power Sample #3
Power Sample #4
Figure 3: Qualitative comparison between DDD, SQR, and MVE. Blue dotted line represents target value. The solid red lines, blue lines, and black lines the PIs of DDD, SQR, and MVE repectively. DDD, SQR, and MVE were configured to have the same PICP for fair comparision.
DDD cover future outcomes. This explains the outperformance of DDD having 26% and 72% improvement. In the Kin8 samples, p(y.(x;) is very close to unimodal. In the Power sample #2, #3, and #4, (z:) are bimodal, but the distance between modes is short that DDD has
8

---

Under review as a conference paper at ICLR 2022
union of PIs that looks like single continuous PI. The length of each PI generated by the three methods does not differ significantly in Kin8 and Power samples. This explains the reason why there is small improvement in Kin8 and Power datasets.
There may be some misspecification because p3ys (xx) approximated by the mixture density network not ground-truth. Even considering the misspecification, the experimental results sufficient to explain that multimodality exists the real world data and the reason for the outperformance of our method.
5.5
PIS WITH COVERAGE PROBABILITY
This section covers additional advantages of our method. One advantage of DDD is that it contains coverage probability for each Pl, giving critical in- formation about how reliable each is. Figure 4 represents the uncertainty estimation in sam- ple from the Parkinson dataset. The y-axis of plots denotes the coverage probability for each PI. Our method provides three Pls (red line) with close to 0%. 43%, and 54% probability, respectively. We can neglect the leftmost PI and rely more on the other two Pls. Another advantage is that we can presume the number of the mode of each sample. The information that two distinet PIs are not neg- ligible implies that the sample has two modes. On the other hand, the previous single-PI Pl methods do not provide such information.
Telemonitoring
1.0
0.4-
0.2
Figure 4: PIs (DDD) with coverage probabil- ity. Left scale means probability.
CONCLUSION
This paper proposes novel method that generates disjoint PIs and demonstrates the need for disjoint PIs to estimate the predictive uncertainty of the regression problems. We confirm that the ensem- ble method improves performance for estimating uncertainty in both in- and out-of distribution. Through the UCI benchmarking datasets, we have shown quantitatively that the DDD method out- performs state-of-the-ard methods. The DDD method improves over current state-of-the-ard meth- ods, reducing an average PI width by over 27%. Additionally, by approximating the conditional density function, we showed that multimodality often exists in real-world data, and we qualitatively explained why DDD generates well-calibrated PIs compared to previous methods. Our method also provides coverage probability corresponding each PI that simple but informative summariza- tion of the conditional probability of the target. We can conclude that our method can provide cell-calibrated and informative PIs for predictive uncertainty estimation various fields.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jose Ameijeiras-Alonse Rosa M Crujeiras, and Alberto Rodriguez-Casal. Mode testing, critical bandwidth and excess mass. Test, 28(3):900 919, 2019.
Christopher M Bishop. Mixture density networks. 1994.
Charles Blundell, Julien Comebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613-1622. PMLR, 2015.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive ics. wui. edu/ml
Di Feng, Lars Rosenbaum, and Klaus Dietmayer. Towards safe autonomous driving: Capture un- certainty in the deep neural network for lidar 3d vehicle detection. In 2018 2/st International Conference Intelligent Transportation Systems (ITSC), pp. 3266-3273. IEEE, 2018.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per-
spective. arXiv preprint arXiv: 1912.02777 2019.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1 1183-1192 192. PMLR, 2017.
Inés M Galván, José M Valls, Alejandro Cervantes, and Ricardo Aler. Multi-objective evolutionary optimization of prediction intervals for solar energy forecasting with neural networks. Information Sciences, 418:363-382, 2017.
Alex Graves. Practical variational inference for neural networks. In Advances in neural information processing systems, pp. 2348-2356 Citeseer, 2011.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. arXiv preprint arXiv: 1708049708 2017.
Abner Guzman- Rivera, Pushmeet Kohli, Dhruv Batra, and Rob Rutenbar. Efficiently enforcing diversity in multi-output structured prediction. In Artificial Intelligence and Statistics, pp. 284- 292. PMLR, 2014.
Yihui He, Chenchen Zhu, Jianren Wang, Marios Savvides, and Xiangyu Zhang. Bounding box regression with uncertainty for accurate object detection. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pallern Recognition, pp. 2888-2897, 2019.
José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learn- ing of bayesian neural networks. In International Conference on Machine Learning, pp. 1861- 1869. PMLR, 2015.
Pavel Izmailov, Wesley Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and An- drew Gordon Wilson. Subspace inference for bayesian deep learning. In Uncertainty in Artificial Intelligence, pp. 1169-1179. PMLR, 2020.
Mohammad Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable bayesian deep learning by weight-perturbation in adam. In International Conference on Machine Learning. pp. 2611 1-2600. PMLR, 2018.
Abbas Khosravi, Saeid Nahavandi, Doug Creighton, and Amir F Atiya. Lower upper bound estima- tion method for construction of neural setwrk-bbssed prediction intervals. IEEE transactions on neural networks, 22(3):337-346, 2010.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.690, 2014.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference Newral Information Processing Systems, pp. 6405-6416. 2017.
10

---

Under review as a conference paper at ICLR 2022
Kimin Lee, Changho Hwang, KyoungSoo Park, and Jinwoo Shin. Confident multiple choice learn- ing. In International Conference on Machine Learning, pp. 2014-20222 2023. PMLR, 2017.
Florian Lerch, Alfred Ultsch, and Jörn Lötsch. Distribution optimization: An evolutionary algorithm to separate gaussian mixtures. Scientific reports, 10(1):1-10, 2020.
David Nix and Andreas Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 ieee international conference on neural networks (ICNN"94), volume 1, pp. 55 60. IEEE, 1994.
Jingyue Pang, Datong Liu, Yu Peng, and Xiyuan Peng. Anomaly detection based on uncertainty fusion for univariate monitoring series. Measurement, 95:280-299 292, 2017.
Tim Pearce, Alexandra Brintrup, Mohamed Zaki, and Andy Neely. High-quality prediction inter- vals for deep learning: A distribution-frre ensembled approach. In International Conference on Machine Learning. pp. 4075-4084. PMLR, 2018.
Tárik Salem, Helge Langseth, and Heri Ramampiaro. Prediction intervals: Split normal mixture from quality-driven deep ensembles. In Conference on Unceriainty in Artificial Intelligence, pp- 1179-1187. PMLR, 2020.
Nitin Anand Shrivastava and Bijaya Ketan Panigrahi. Prediction interval estimations for electricity demands and prices: a multi-objective approach. IET Generation, Transmission & Distribution, 95)):4945502 2015.
Marek Smieja, Lukasz Struski, Jacek Tabor, Bartosz Zieliński, and Przemyslaw Spurek. Processing of missing data by neural networks. arXiv preprint arXiv:1805.074000 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-198, 2014.
Hsi Guang Sung. Gaussian mixture regression and classification. PhD thesis, 2004.
Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning. Advances in Neural Information Processing Systems, 32:6417-6428, 2019.
Kenneth F Wallis. The two-piece normal, binormal, or double gaussian distribution: origin and rediscoveries. Stafistical Science, pp. 106-112, 2014.
Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E Turner, Jose Miguel Kernandez-Lobate and Alexander L Gaunt. Deterministic variational inference for robust bayesian neural networks. arXiv preprint arXiv: wtx.v:1100003958 2018
Jiayu Yao, Weiwei Pan, Soumya Ghosh, and Finale Doshi-Velez. Quality of uncertainty quantifica- tion for bayesian neural network inference. arXiv preprint arXiv: 19X6v:1906.099 1906.09686, 2019.
Jaeyoung Yoo, Geonseok Seo, and Nojun Kwak. Mixture-model boder-baedd bounding box density esti- mation for object detection. 2019.

---

Under review as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIALS
A DATA PRE-PROCESSING
We use the same dataset and pre-processing implemented in SNM-QD+ Salem et (2020) We
add two more datasets that are the Parkinson- Parkinson-Telemonnee Telemonitoring and Bike- Bike-Sharing from UCI datasets:
Parkinson Telemonitoring" "subject id" is excluded
Seoul Bike Sharing Demand "date", "seasons", "holiday", and "functioning day" are excluded.
B QUANTITATIVE COMPARISON BETWEEN DDD AND DDD ens
Table 2 compares DDD ens with a single DDD model on the UCI benchmark datasets. DDD ens has higher PICP and lower or equal MPIW over DDD throughout all datasets except for the Yacht. In the Yacht dataset, DDD has similar MP/W and much higher PICP, so it can be considered that DDD is high-quality. This result implies that the ensemble method improves performance in terms of the HQ principle.
Table 2: Quantitative comparison between DDD and DDD ens-
Method
bike
boston
concrete
energy
kin8
naval
DDD
0.94+0.01
0.92==0.03
0.90=10.05
0,89 0.98:10.02
0.94=0001
0.97+00.00
PICP
DDD
0.96=10.01
0.95=0.03
0.94=0.03
0.97=0.01
1.00=
DDD
1.23:10.07 1.30=0.26
1.00#0.18
0.12::0.02 0.16
1.07=0.05
0.06+00.02
MPIW
DDD
1.00
1.21 0.012
0.00
1.03:10.02
0.06:10.01
Method
purkinson
power
protein
wine
yacht
DDD
0.95110.01 0.95
0.94:0.00
0.94110.02
0.8620.07
PICP
DDD
0.99++0.01
0.96
0.95++0.00
0.94+-0.02
0.94+0.04
DDD
0.43:10.13 0.80:10.02
1.64:0.00
0.26110.22
0.08
MPIW
DDD
0.40+-0.05
0.79+0.02
1.57++0.02
0.18++0.05
0.09
'The code from the author is available http://ww..taaaaaaaaaa.....aaaaa
12

---

Under review as a conference paper at ICLR 2022
C MORE MULTIMODAL SAMPLES
We draw p(vil a;)) through the mixture density network in the same way as Section 5.4 Sharing dataset shows clear multimodality and DDD produce high-quality PIs compared to MVE and SQR. The Wine dataset has a discretized outcome because it is for the regression problem of guessing the grade of wine. DDD generates very short intervals for each grade position compared to others. For datasets with discretized outcomes like the Wine samples, it is much more helpful to obtain the coverage probability for each PI than the total width of PIs (e.g., grade with probability of 20%, grade 2 with a probability of 45%, and grade 3 with probability of 30%).
The Bike
Bike Sample #1
Bike Sample
Bike Sample 03
Bike Semple #4
1992)
1505)
PID200
y
Wine Sample #1
Wine Samale #2
Wine Sample #3
Wine Sample #4
Figure 5: Qualitative comparison between DDD, SQR, and MVE. Blue dotted line represents target value. The solid red lines, blue lines, and black lines are the PIs of DDD, SQR, and MVE repectively.
13


Reviews:
[{"review_id": "txWDLqe9SeS", "reviewer": "Reviewer_RmVv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of this paper is to construct better prediction intervals addressing the multimodality of predictive distribution and show better performance compared to the previous. It can be helpful to assess the uncertainty in a more rigid manner. \n\nHowever, I have a doubt that the title ‘Distribution-driven disjoint uncertainty estimation for deep learning” is adequate. The paper uses deep architectures to generate the distribution-driven-disjoint PIs after the preprocessing of mixture density networks. The mixture of density networks is an architecture, and there is no solution or explanation when we use the other deep architectures for other tasks. This implies that the algorithm is not general to address the PIs.\n\nIn detail, I have some issues.  \n\nAt first, I  feel that the figures in the paper are not kind to readers. In Figure 2, the prediction intervals are not shown well exception of the length of intervals, and in Figure 4, there is no explanation about solid curves after 0 on the x-axis.      \n\nAlso, I have some issues with the proposed algorithms. \n1) The use of $F_i$ seems reasonable. If you can provide more concrete evidence for the use of $F_i$, then the paper can be improved. More general criteria can be required for the generalization. \n2) The $K$ in the mixture density network is a hyperparameter, and the empirical evidence for using $K=5$ is validated. Can you use the CV or other learning algorithm to choose the $K$?\n3) I am interested in the computation time for Alg. 1. Is it efficient? \n4) There is no detail concerning $M$ of ensembles.            \n\nAdditionally, I cannot find the more significant use of disjoint PIs in various tasks. I was hoping you could provide more practical tasks related to PIs, such as active learning or calibration problems.   \n", "summary_of_the_paper": "This paper provides the algorithm for the construction of prediction intervals composed of disjoint intervals. The authors proposed the motivation of disjoint intervals well, and the motivating example is impressive. The algorithms are also well accommodated with the statistical or learning-based prediction intervals, which contribute to the assessment of the uncertainty of prediction in general.\n\n", "summary_of_the_review": "The paper success in the construction of disjoint PIs. However, there is a lack of thorough insightful motivation, remain of some algorithmic issues. Presentation is not better.     ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "a94qmZQZvta", "reviewer": "Reviewer_3Zfh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper makes a strong and somewhat overlooked point that existing methods for the evaluation of PIs provide low fidelity in the presence of multimodality. The recent introduction of flexible methods for heteroscedastic noise modelling, such as SQR, creates a clear gap in the literature regarding their evaluation. The paper under consideration here attempts to fill this gap. However, from my point of view, it has 2 critical shortcomings that prevent it from representing a significant contribution.\n\n1. The two key contributions of the paper are difficult to disentangle. As a result, I do not think the experiments validate the main claims of the paper.\n\t* According to the authors, existing methods provide single predictive intervals and thus can not be evaluated using the tighter multi-interval method provided in equation 1. I do not agree with this point. It is not clear to me why a potentially multimodal predictive distribution, like SQR, can not generate disjoint PIs. A simple procedure to do this would be to place the centre of these PIs at the top-K modes of the predictive distribution. This is not very different from what the authors already do with their GMM, which also has full support in $\\mathcal{R}$.\n\t* As a result of the above issue, the authors evaluate their method using a disjoint set of PIs while evaluating baselines using a single PI. It seems clear that the PICP / MPIW tradeoff obtained by the multimodal generation+evaluation approach produces better results. However, there is no evidence that the proposed PI generation method is any better than competing approaches, which could potentially also produce multimodal distributions. For instance, in figure 3 and figure 5, SQR seems to always cover the right range in output space but its PI is always spread too thin due to multimodality. \n\n\n2. The motivation for the proposed PI generation method is unclear to me.\n\t* The authors propose to use a mixture density network combined with an auxiliary NN that outputs PI lower and upper bounds from the GMM parameters. It is not clear to me why they need the auxiliary network considering that disjoint $\\gamma$ % PIs can be obtained in closed form from the mixture density network directly. Is this last step intended to relax the Gaussianity assumption of the mixture density network?\n\nI am willing to raise my score to an accept if the authors clarify the motivation for their PI generation approach and evaluate competing multimodal noise modelling approaches using disjoint PIs.\n\n**Other comments:**\n\nThe paper has many grammar errors. I would recommend the use of an online grammar checker. \n", "summary_of_the_paper": "This paper notes that existing approaches to predictive interval (PI) generation and evaluation assume unimodal predictive distributions. This results in unnecessarily loose PIs in the presence of multimodality in the predictive distribution or multimodality in the targets. The authors first propose to extend the definition of PI to be a union of disjoint intervals, allowing for more fidelity in evaluation. They then propose a method to explicitly generate multimodal PIs. This method is based on feeding the output of a mixture density network (conditional GMM) through a secondary NN that outputs a set of lower and upper bounds for PIs. The authors compare their method against alternative heteroscedastic noise models in terms of PICP and MPIW on 11 UCI datasets.", "summary_of_the_review": "This paper proposes a higher fidelity method for the evaluation of predictive intervals under multimodal predictions. The authors also propose a method to make multimodal predictions. Unfortunately, the author’s evaluation setup assumes that their proposed method is the only one to produce multimodal predictions (which is not the case). This creates a drastically unfair comparison.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P4FoNVqmnLZ", "reviewer": "Reviewer_8k1c", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n- Studies an interesting and well-motivated problem.\n- Contains thorough quantitative and qualitative experiments to both motivate the problem and show improvement over the prior work.\n\nWeaknesses:\n[Presentation]\n- The writing needs a lot of improvements. Many sentences do not follow the scientific writing style. Here I only mention the typos but there are many other cases where the sentences need to be re-written in a more elegant way.\n- Fourth line of Abstract: it -> they\n- Third line of introduction: something missing in the sentence probably you meant: functions \"+of\" the NN\n- Last paragraph of page 1: a single continuous PI provides low-quality PIs -> a single continuous PI \"+method?\" provides low-quality PIs. In many other cases, PI methods and PIs are used interchangeably which should not be the case.\n- Last sentence in section 2: The dot before (so-called MVE_ens) should be omitted.\n- First paragraph of page 4: which reflects HQ principle and \"+is\" differentiable.\n- After eq.6: Then, the coverage probability... -> why \"then\"?\n- After eq.6: L_PICP is defined as the mean square error with the target proportion γ with the max(0, x) operation. -> some words missing?\n- Before eq.7: number of the mixture -> number of the \"components\"\n- Last paragraph of 4.2: λ is a hyperparameter that determines how much trade-off between L_PICP and L_MPIW. -> some words missing?\n- Last paragraph of 4.2: If PICP obtained through training is less than γ, increase λ -> writing should be improved.\n- First sentence of 4.3 needs to be re-written\n- In the Hyperparameters section: All neural networks... -> all neural networks\n- last two paragraphs of 5.2 need to be re-written\n- title of 4.2: typo in architecture\n\n[Questions]\n- In Figure 2, why is the blue line sometimes showing a negative value while it should correspond to the width of the prediction interval?\n- In Figure 2, how do you justify your choice of training and testing samples? You keep the points with x>4 or x<-4 but should it not be the case that you shuffle your points and randomly pick a percentage as test data points? The regressor is never trained on points with x>4 or x<-4 while asked to predict them.\n- Figure2 is missing legend.\n- Figures 3,4 are missing axes labels and ticks.\n- Why are the experiments for the Protein dataset repeated 5 times and the other datasets repeated 20 times?\n\n\n[Other Concerns]\n- You mention you normalize the target distribution based on the entire data but you have to rather normalize based on only the training data.\n- There is not enough details to fully understand your method. I do not find the exact mathematical definition of F_i which is an important part of your approach. \n- A discussion regarding the time complexity of the algorithm in terms of the number of components in the mixture model (K) is missing.", "summary_of_the_paper": "The paper addresses the problem of determining prediction intervals (PI) in regression task. The prediction interval problem can be summarized as predicting a lower and upper bound between which the potential observation falls with a certain probability. The paper proposes a method to report the prediction interval as the union of disjoint intervals, in contrast with the previous methods which report a unified continuous interval. The motivation is that if the conditional density function has multiple modes, a single prediction interval may not be well descriptive of the uncertainty of the predictive model. To achieve this goal, they propose a differentiable objective function together with a Neural Network architecture that produces the union of disjoint prediction intervals. Through experiments, they show that multimodality often exists in real-world datasets and that their method manages to produce prediction intervals of higher quality (in terms of the commonly used metrics to assess the quality of the prediction intervals such as coverage probability and interval width) compared to the previous work.", "summary_of_the_review": "While the paper studies an interesting problem, it suffers from presentation issues which makes it hard to grasp the main technical contribution of the paper. The paper seems to be written in a rushed way and consists of many sentences which have missing words or are grammatically incorrect. This gets bolder as we move on to the later parts of the paper. I tried to mention some of the typos and the incomplete sentences as I went through the paper but they are too many to mention in this review. Many parts of the paper need to be re-written in order for it to get ready for publication. \n\nThe novelty is in re-writing the objective function, which tries to minimize prediction interval length while maintaining a certain coverage probability, in a differentiable manner and then optimizing it.  Some technical details are also missing (such as discussion on the complexity of the approach). Overall I think the paper is not ready for publication and needs improvements both in technical content and presentation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for producing a mixture of (disjoint) predictive distributions for deep learning models rather than a single predictive distribution.  The reviewers in general found that the idea had strong potential, was well motivated and addresses an important and under-appreciated problem in deep learning.  They seemed to find the proposed approach of using mixture density networks to be sensible.  However, the reviewers seemed to find that the paper was unclear in presentation and grammatically, as if hastily written.  One reviewer noted that they would not be able to reproduce the method given the confusing presentation.  The reviewers also found that the experiments didn't adequately evaluate their method empirically.  Unfortunately, the reviewers all agreed that the paper is not quite ready for publication (5, 3, 5).  Careful rewriting of the paper and the technical contributions and strengthening the experiments would go a long way towards improving this paper for a future submission.
Decision: Reject