Year: 2021

Abstract:
This paper redefines prediction intervals (PIs) as the form of a union of disjoint intervals. PIs represent predictive uncertainty in the regression problem. Since previous PI methods assumed a single continuous PI (one lower and upper bound), it suffers from performance degradation in the uncertainty estimation when the conditional density function has multiple modes. This paper demonstrates that multimodality should be considered in regression uncertainty estimation. To address the issue, we propose a novel method that generates a union of disjoint PIs. Throughout UCI benchmark experiments, our method improves over current state-of-the-art uncertainty quantification methods, reducing an average PI width by over 27%. Through qualitative experiments, we visualized that the multi-mode often exists in real-world datasets and why our method produces high-quality PIs compared to the previous PI.

text: This paper redefines prediction intervals (PIs) as the form of a union of disjoint intervals. PIs represent predictive uncertainty in the regression problem. Since previous PI methods assumed a single continuous PI (one lower and upper bound), it suffers from performance degradation in the uncertainty estimation when the conditional density function has multiple modes. This paper demonstrates that multimodality should be considered in regression uncertainty estimation. To address the issue, we propose a novel method that generates a union of disjoint PIs. Throughout UCI benchmark experiments, our method improves over current state-of-the-art uncertainty quantification methods, reducing an average PI width by over 27%. Through qualitative experiments, we visualized that the multi-mode often exists in real-world datasets and why our method produces high-quality PIs compared to the previous PI.

heading: 1 INTRODUCTION
text: Deep neural networks (NNs) show remarkable performance in predicting a target for regression problems. However, the prediction is not enough to make it trustworthy: minimization of objective functions the NN leads to network outputs which approximate the conditional averages of the target data with no information about sampling errors and prediction accuracy. Moreover, if the target is multivalue, NN output can be far from the actual target in the regression problems. Incorporating the predictive uncertainty into the deterministic approximation generated by NNs improves the reliability and credibility of the predictions. This issue is being discussed in various domains such as autonomous driving (Feng et al., 2018), object detection (He et al., 2019), solar energy forecasting (Galván et al., 2017), electricity demands and price estimation (Shrivastava & Panigrahi, 2015), and sensor anomaly detection (Pang et al., 2017).
Prediction interval (PI) represents and quantifies predictive uncertainty in the regression problem. Pearce et al. (2018); Tagasovska & Lopez-Paz (2019); Salem et al. (2020) have recently provided competitive performance by generating a PI to estimate predictive uncertainty. PI describes predictive uncertainty for each sample in the form of two values (lower and upper bound) between which a potential observation falls with a certain probability (e.g., 95% or 99%). PI can provide the amount of uncertainty for each sample by the width of PI. It also provides the possible range of prediction by bounds. It is a self-evident principle that high-quality PI should be as narrow as possible while containing some specified proportion of data points (hereafter referred to as the HQ principle). The quality of a PI is often evaluated by the metric derived from the HQ principle (Khosravi et al., 2010; Galván et al., 2017; Pearce et al., 2018; Tagasovska & Lopez-Paz, 2019; Salem et al., 2020).
Previous methods estimate the regression uncertainty with a single continuous PI, but it may suffer from performance degradation in the regression having multimodality. A toy example in Figure 1 is a one-dimensional regression example that has two modes. We observe that a single continuous PI (gray shade) provides unnecessarily large PIs to fill in the gap between the two modes compared to disjoint PIs (blue shade). This means that a single continuous PI provides low-quality PIs in terms of the HQ principle. Including intervals that are unlikely to contain future observations makes PIs less reliable. Note that this issue becomes severe as the distance between modes increases. We qualitatively confirmed that multimodality often exists in real-world regression datasets through approximating conditional probability density function. We also confirmed that state-of-the-art methods
generate low-quality PIs on real-world samples with multimodality. This is covered in more detail in Section 5.4.
Considering multimodality has been successful at handling the underlying stochastic structure in various fields (Ameijeiras-Alonso et al., 2019; Lerch et al., 2020). Concerning multimodality, various works such as clustering, multi-object detection (Yoo et al., 2019), missing data reconstruction (Smieja et al., 2018), multiple-choice learning (Lee et al., 2017), and multi-output prediction (Guzman-Rivera et al., 2014) have been conducted. However, recent regression uncertainty estimation studies do not consider multimodality in depth.
In this work, we redefine PI as a union of disjoint PIs due to the limitation of a single continuous PI in multimodality (Section 3). Since prior PI methods and loss functions do not apply to the union of disjoint PIs, we propose a new differentiable objective function and NN architecture that produce the union of disjoint PIs (Section 4). Additionally, we use the ensemble method to boost the performance for both in- and out-of-distribution regions (Section 5.2). As a result, our method improves over current state-of-the-art methods, reducing an average PI width by 27% throughout eleven real-world datasets (Section 5.3). In addition, our method can provide the coverage probability of each disjoint PI (e.g., 20% chance of being between 1 and 3, 75% chance of being between 5 and 9). This means that our method gives information about how reliable each interval is (Section 5.5).

heading: 2 RELATED WORK
text: There are two approaches for estimating the predictive uncertainty for regression problems: Bayesian and non-Bayesian. In the Bayesian approach, NN parameters are considered as a distribution, and the uncertainty is calculated by marginalizing the parameters (Graves, 2011; Blundell et al., 2015; Hernández-Lobato & Adams, 2015; Gal et al., 2017; Khan et al., 2018; Wu et al., 2018; Yao et al., 2019; Izmailov et al., 2020). Though theoretically grounded, an approximation is needed since calculating the posterior distribution of NN parameters is computationally intractable. It also requires high computational demand in the inference time. The non-Bayesian approach, on the other hand, defines the output of NN as parameters to describe the predictive uncertainty. It is usually less computational than the Bayesian approach. However, since the NN parameters are fixed, non-Bayesian methods have a limitation in expressing the model uncertainty. Therefore, the deep ensemble with random initialization is additionally used to deal with model uncertainty. Several papers in the non-Bayesian branch have recently provided competitive performance (Lakshminarayanan et al., 2017; Pearce et al., 2018; Tagasovska & Lopez-Paz, 2019; Salem et al., 2020). Our paper focuses on the Non-Bayesian approach, especially for the regression problem. Therefore, we would take a closer look at the non-Bayesian methods by dividing them into PI and non-PI methods.
As PI methods for non-Bayesian methods, Khosravi et al. (2010) propose the Lower Upper Bound Estimation (LUBE) method that produces PI for the first time. Followed by that, Pearce et al. (2018) propose a quality-driven (QD) loss function that is compatible with gradient descent optimization. They also propose an ensemble method for PI with multiple predicted lower and upper bounds to estimate the model uncertainty. Salem et al. (2020) retrofit the QD loss function and propose a new ensemble method by fitting the split normal mixture distribution (Wallis, 2014) to the
PI and averaging the distribution, where they name it as SNM-QD+. It increases the robustness of the training process compared to the QD method. However, SNM-QD+ has difficulties searching hyperparameters because the loss function contains various hyperparameters to achieve the advantages. Tagasovska & Lopez-Paz (2019) propose the simultaneous quantile regression (SQR) and the orthonormal certificates (OC) to estimate data noise and model uncertainty, respectively. However, this strategy generates PI only by the SQR without an ensemble method, and model uncertainty from OC is not included in the PI. Therefore, the PI of SQR does not consider the model uncertainty. Aforementioned loss functions and methods can only generate a single continuous PI but not a union of disjoint PIs.
As non-PI methods, Mean-Variance Estimation (MVE) (Nix & Weigend, 1994) uses a NN with two output nodes that are considered as a mean and a standard deviation of the conditional probability distribution. Since NN parameters are fixed, it cannot deal with the model uncertainty. Lakshminarayanan et al. (2017) demonstrate the deep ensemble of multiple MVE with random initialization improves the performance, especially in out-of-distribution regions. (so-called MVEens). Fort et al. (2019) shows that ensemble with random initialization may sample different modes in function space and therefore perform well in exploring model uncertainty.

heading: 3 UNION OF DISJOINT PREDICTION INTERVALS

heading: 3.1 PROBLEM SETUP
text: Consider a dataset {xi, yi}Ni=1 where xi is an input and yi is a target. For each data point {xi, yi}, the disjoint set of PIs that covers the desired given proportion γ ∈ [0, 1] is defined as follows:
PIi = J(i)⋃ j=1 [Lij , Uij) (1)
where Pr(yi ∈ PIi) ≥ γ and Lij ≤ Uij < Li(j+1) for all j Lij and Uij is a lower and upper bound of jth PI related with ith data point. J (i) is the number of disjoint intervals when PI is expressed with the smallest number of disjoint intervals. That is, J (i) is unique for a given interval. Note J (i) may have a different value for each data point. The previous methods assume a single continuous PI that is J (i) = 1 for all i.

heading: 3.2 PERFORMANCE METRIC: PICP AND MPIW
text: To measure the quality of PI methods based on the HQ principle, let Prediction Interval Coverage Probability (PICP ) and Mean Prediction Interval Width (MPIW ) be defined as,
PICP = c
N where c = N∑ i=1 ci and ci = { 1, if yi ∈ PIi 0, otherwise
(2)
MPIW = 1
N N∑ i=1 J(i)∑ j=1 (Uij − Lij) (3)
PICP measures the ratio of the target that is captured within PIs while MPIW measures total length of PIs over the entire samples. According to the HQ principle, PIs should minimize MPIW subject to PICP ≥ γ (e.g. γ = 0.95 or 0.99). This metric is widely used to compare performance of PI-related methods (Khosravi et al., 2010; Pearce et al., 2018; Tagasovska & Lopez-Paz, 2019; Salem et al., 2020).

heading: 4 DISTRIBUTION-DRIVEN-DISJOINT METHOD
text: We propose a learning-based method that generates distribution-driven disjoint (DDD) PIs, and we call our method as DDD method. The DDD method produces high-quality PIs without the assumption that J (i) = 1. To generate multiple disjoint PIs with a learning-based method, we need to
formulate a differentiable loss function that reflects the HQ principle. However, this is a challenging problem because c from (2) is non-differentiable. Pearce et al. (2018); Salem et al. (2020) proposed a QD and QD+ loss function in the form of constraint optimization by approximating c in a differentiable way. Tagasovska & Lopez-Paz (2019) employed a pinball loss which reflects HQ principle and differentiable. However, these loss functions have a limitation in that they work for a single continuous PI which has J (i) = 1.
To derive a new differentiable loss function (so-called DDD loss) for multiple disjoint intervals, we first approximate the conditional distribution given input, p̂(yi|xi), with a Gaussian mixture model. Then, we derive the DDD loss by using the cumulative density function of the Gaussian mixture (why we call our method distribution-driven disjoint). Our DDD method trains NN by minimizing the DDD loss.
Another major problem is that the optimal number of disjoint prediction intervals J (i)opt may differ for each p̂(yi|xi), making it hard to implement [L̂ij , Ûij) J(i)
j=1 as an output of NN. To deal with the problem, after producing K intervals regardless of conditional density, the union process removes the overlapping part. We propose a novel architecture that can implement this in a differentiable way. Additionally, we employed a simple ensemble method to improve performance for both inand out-of-distribution observations.

heading: 4.1 DERIVATION OF THE DDD LOSS
text: In this section we derive the DDD loss, LDDD, that reflects the HQ principle: PIs should minimize MPIW subject to PICP ≥ γ. LDDD combine LMPIW and LPICP with a Lagrangian, λ, controlling the importance of width vs. coverage. Each term is defined as:
LDDD = LMPIW + λLPICP (4)
LPICP = max(0, γ − 1
N N∑ i=1 ĉi) 2, LMPIW = 1 N N∑ i=1 J(i)∑ j=1 (Ûij − L̂ij) (5)
where ĉi = J(i)∑ j=1 (Fi(Ûij)− Fi(L̂ij)). (6)
The coverage probability ĉi is an approximation of ci in Equation (2). The function Fi is a cumulative density function of the distribution p̂(yi|xi). Then, the coverage probability ĉi is calculated from the cumulative density function Fi. LPICP is defined as the mean square error with the target proportion γ with the max(0, x) operation. The max(0, x) operation is used to minimize the loss only when the average coverage probability, 1N ∑ i ĉi, is lower than the given proportion γ. The loss function LMPIW is naively derived from the definition of MPIW in equation (3).
For LDDD to be differentiable for L̂ij and Ûij , ĉi must be differentiable for L̂ij and Ûij . Therefore, Fi should be differentiable. We approximate the p(yi|xi) with the mixture density network with Expecation-Maxmization algorithm (Greff et al., 2017). This is not a distributional assumption because any distribution can be approximated by the Gaussian mixture model when the number of the mixture is sufficient (Bishop, 1994; Sung, 2004). When parameters of the Gaussian mixture distribution are {µi,σi,πi}, an approximated probability distribution p̂(yi|xi) is defined as
f(xi;µ,σ,π) = K∑ k=1 πk(xi) N (yi |µk(xi), σ2k(xi)), (7)
where K is the number of mixture components. K can be considered large enough if increasing K does not reduce the negative likelihood loss of the mixture density network. Setting K=5 worked well for our experiment in section 5. For brevity, we refer to µk(xi) as µik, and same abbreviation applied to σk and πk.
Algorithm 1 Pseudocode for the DDD Network 1: Definition 2: µi,σi,πi : Input of NN where size(µi) = size(σi) = size(πi) = K.
The means, standard deviations, and weights of the Gaussian mixture p̂(yi|xi). 3: (−)i , (+) i : Output of NN where size( (−) i ) = size( (+) i ) = K, (−) i ≥ 0, (+) i ≥ 0
4: Train 5: L̂ = µ− (−) and Û = µ+ (+). 6: L̂ = L̂[argsort(L̂)], Û = Û [argsort(L̂)] . ascending order in k-related dimension 7: for i← 1 · · ·N do 8: for k ← 1 · · ·K do 9: Ûik = L̂ik + ReLU(Ûik − L̂ik)
10: L̂i(k+1) = Ûik + ReLU(L̂i(k+1) − Ûik) if k 6= K 11: end for 12: end for 13: Minimize LDDD = 1N ∑N i=1 ∑K j=1(Ûik − L̂ik) + λ max(0, γ − 1 N ∑N i=1 ĉi) 2
where ĉi = ∑K j=1(Fi(Ûik)− Fi(L̂ik))

heading: 4.2 DDD NETWORK ARCHTECTURE
text: This section describes a neural network architecture that generates distribution-driven-disjoint PIs (so-called DDD network) as pseudocode in Algorithm 1. The architecture is classified into four blocks according to the major objectives and explained in detail. All blocks achieve each objective in a differentiable form.
Input Generation (line 2) Input of the DDD network is [µi,σi,πi] because µi, σi, and πi are used to compute the loss function. The input xi is converted into [µi,σi,πi] by the mixture density network. For this, it is necessary to train the mixture density network before training the DDD network. The Neural Expectation-Maximization algorithm (Greff et al., 2017) is used for the training.
Multiple Interval Generation (line 3-5) This block generates K multiple intervals. The last nodes of NN are squared to ensure that the outputs of NN ([ (−)i , (+) i ]) are positive. The primary aspect is that this block restricts each interval to include each peak µik of the Gaussian mixture model (line 5). This aspect contributes to producing well-calibrated PIs because it is self-evident that wellcalibrated PIs should contain more than one peak. If the output of NN is defined as lower bounds and upper bounds without the restriction, it is experimentally confirmed that it often converges to low-quality PI.
Union: overlap prevention (line 6-12) This block implements the union process in a differentiable way using sorting and max(0, x) functions (the sorting process does not interfere with backpropagation). Through this process, K intervals can be converted to J (i) blocks. As a result, the DDD method provides the union of disjoint PIs.
Minimize LDDD (line 13). By training the DDD network to minimize LDDD, the union of disjoint PIs becomes high-quality. λ is a hyperparameter that determines how much trade-off between LPICP and LMPIW . If PICP obtained through training is less than γ, increase λ. This increments PICP , along with MPIW .

heading: 4.3 ENSEMBLE METHOD
text: Optimizing the several NN models with random initialization and then aggregating them is known to be effective for the uncertainty estimation in the out-of-distribution region (Lakshminarayanan et al., 2017). Since then, recent studies (Pearce et al., 2018; Salem et al., 2020) have promoted the ensemble of PIs.
We train M different mixture density newtork models for p̂(yi|xi) with random initialization followed by averaging them, fens(xi) = 1M ∑ m f (m)(xi), where f (m)(·) is the probability density
function in equation (7). Then, we can acheive cumulative density function of the ensembled model from F ens(xi) = 1M ∑ m F
(m)(xi). Finally, we train the DDD network proposed in 4.2 to achieve estimating the predictive uncertainty. When M models are ensembled, not K but KM intervals are generated in the multiple interval generation block.

heading: 5 EXPERIMENT

heading: 5.1 IMPLEMENTATION DETAIL
text: Dataset We use UCI regression benchmarking datasets (Dua & Graff, 2017). While previous works use nine datasets (Boston, Energy, Kin8nm, Naval, Power, Protein, Wine, and Yacht) for evaluation, we add two more datasets that are the Parkinson-Telemonitoring and Bike-Sharing. For each dataset, the experiment is repeated 20 times with different random seeds except for the Protein dataset, where it is repeated five times. Each dataset is split into train, valid, and test set with the ratio of 0.81, 0.09, and 0.1, with random shuffling for each experiment. The target y is standardized to zero mean and unit variance based on the entire data in each dataset. More specifics about data pre-processing are provided in the Appendix A.
Methods We compare DDD (ours) with MVE (Lakshminarayanan et al., 2017) and QD+ (Salem et al., 2020) where the ensemble method of each of them are named DDDens, MVEens, and SNMQD+, respectively. SQR (Tagasovska & Lopez-Paz, 2019) participates in the comparison without an ensemble since Tagasovska & Lopez-Paz (2019) did not propose an ensemble method for its PI generation. Considering QD+ (Salem et al., 2020) is a retrofit version of QD (Pearce et al., 2018), QD is not included in the comparison. We build the PI of the MVEens method based on its mean and standard deviation like Pearce et al. (2018) did.
Hyperparameters While a largerK (the number of mixtures) is preferable in theory (Sung, 2004), it needs more data and increased model complexity in practice. In our method, K = 5 was sufficient to apply in the UCI datasets. We experimented with the same hyperparameters for all methods as follows: All neural networks have two hidden layers with 50 units except for the Protein dataset, where 100 units are used, with each layer having the rectified linear activation function. We set γ = 0.95 for the ratio of PIs to be covered. The ensemble size is set to M = 5. Adam optimizer (Kingma & Ba, 2014) is used for the optimization, and we did not use the dropout (Srivastava et al., 2014) and the weight decay.
For a fair comparison, hyperparameters should be set to derive near-optimal performance for each method. The grid search is used for the hyperparameter search on the learning rate for MVEens and the mixture density network of the DDDens method. For the DDDens network, we search only on the lambda contained in the loss function (the same learning rate was used as the first phase). However, in the case of SNM-QD+, it is difficult to optimize because the loss function includes multiple hyper-parameters. Therefore, we take the quantitative results from Salem et al. (2020) for the nine existing UCI datasets. For the additional two datasets, we run 300 hyperparameter searches with the code provided by the paper’s authors.
We set λ so that DDDens has PICP similar to other methods. It is easy to compare the performance between methods from the perspective of the HQ principle if the PICP is similar. This is because if PICP is similar, onlyMPIW needs to be compared. Note that, as λ increases, PICP and MPIW increase, and vice versa.

heading: 5.2 EFFECT OF THE ENSEMBLE METHOD
text: Figure 2 presents qualitative results to confirm the effects of ensemble in in-distribution and outof-distribution. It describes uncertainty estimation of three single DDD models and the ensemble of the three. The blue dotted line means the amount of uncertainty (yticklabels on the right side). Data points are generated through y = 1(0.02x3 + 0.02 2) + (1 − 1)(x + 0.02 2) where 1 ∼ Bernoulli( 12 ) and 2 ∼ N(0, 3
2). Training data points are not included in x < −4 and x > 4 to demonstrate the effect of DDDens at those out-of-distribution regions. The models used in Figure 2 are the result of adjusting the λ value so that PICP for the validation set is 0.95 ± 0.01. Since the data points are generated to have homoscedastic noise, the amount of uncertainty in in-distribution
is almost constant. In out-of-distribution, the amount of uncertainty increase toward both ends. In other words, the blue dotted line should have a U shape if high-quality. We can observe that the ensemble method produces more tight PIs for the in-distribution region (compared to the left side of #1) and quantifies better the uncertainty for the out-of-distribution region (compared to both sides of #1 and the left side of #2 and #3). Put simply, the ensemble model has a clearer U shape compared to other single models.
We quantitatively compared DDDens with a single DDD model on the UCI benchmark datasets. According to the HQ principle, when PICP is greater than or equal to, and MPIW is smaller, better performance is achieved. For all datasets, we confirmed that DDDens achieves higher PICP than DDD while having lower to equal MPIW that implies outperformance of DDDens. The comparison result is presented in appendix B.
Combining all the experimental results, we can conclude that our ensemble method boosts the performance for both in- and out-of-distribution.

heading: 5.3 BENCHMARKING EXPERIMENT
text: Table 1 compares DDDens, MVEens, SNM-QD+ and SQR on the 11 UCI datasets. For each datasets, we compare DDDens with the best performance method among MVEens, SNM-QD+ and SQR and measure how much MPIW is reduced. Improvement means how much DDDens reduces MPIW. DDDens and the best method for each dataset are marked in bold. It is a better method if the method has a higher or equal PICP and a smaller MPIW . However, if both PICP and MPIW are higher or smaller at the same time, comparison is not clear. Therefore, we set the criterion that the best performance method of each dataset is determined to have the smallest MPIW among those with a PICP of 0.94 or higher.
DDDens has higher or equal PICP and a smallerMPIW compared to the best performance methods in 10 out of 11 datasets. This confirms that DDDens provides well-calibrated PIs for predictive uncertainty estimation compared to the state-of-the-art methods. We can observe minor improvements like 1% or 3%, and dramatic improvements like 72% or 93%. It is reasonable to infer that datasets with significant improvement have multimodality, and those with low improvement have unimodality. The reason why DDDens shows better performance and multimodality of the datasets are discussed in Section 5.4 through a qualitative experiment.

heading: 5.4 QUALITATIVE EXPERIMENT
text: We draw p̂(yi|xi) through the mixture density network for the following reasons: 1. to find out whether the conditional probability density of samples is multimodal. 2. to ascertain the reason for the outperformance of DDDens. Each plot in Figure 5 is the sample of the Protein, Parkinson, Kin8, and Energy dataset. The light blue line denotes the approximated the conditional density p̂(yi|xi). Note that, although approximated by a Gaussian mixture model, p̂(yi|xi) can express a unimodal distribution (See Kin8 samples and Power-Plant sample #1).
In the Protein and Parkinson samples, we observe that our mixture model describes a multimodal distribution. DDD provides multiple disjoint prediction intervals covering the target and does not contain less plausible regions than SQR and MVE. In other words, even with a much shorter PI,
PICP
MPIW
PICP
MPIW
Figure 3: Qualitative comparison between DDD, SQR, and MVE. Blue dotted line represents target value. The solid red lines, blue lines, and black lines are the PIs of DDD, SQR, and MVE repectively. DDD, SQR, and MVE were configured to have the same PICP for a fair comparision.
DDD can cover future outcomes. This explains the outperformance of DDDens having 26% and 72% improvement. In the Kin8 samples, p̂(yi|xi) is very close to unimodal. In the Power sample #2, #3, and #4, p̂(yi|xi) are bimodal, but the distance between modes is short so that DDD has a
union of PIs that looks like a single continuous PI. The length of each PI generated by the three methods does not differ significantly in Kin8 and Power samples. This explains the reason why there is a small improvement in Kin8 and Power datasets.
There may be some misspecification because p̂(yi|xi) approximated by the mixture density network is not ground-truth. Even considering the misspecification, the experimental results are sufficient to explain that multimodality exists in the real world data and the reason for the outperformance of our method.

heading: 5.5 PIS WITH COVERAGE PROBABILITY

heading: 6 CONCLUSION
text: This paper proposes a novel method that generates disjoint PIs and demonstrates the need for disjoint PIs to estimate the predictive uncertainty of the regression problems. We confirm that the ensemble method improves performance for estimating uncertainty in both in- and out-of distribution. Through the UCI benchmarking datasets, we have shown quantitatively that the DDD method outperforms state-of-the-art methods. The DDD method improves over current state-of-the-art methods, reducing an average PI width by over 27%. Additionally, by approximating the conditional density function, we showed that multimodality often exists in real-world data, and we qualitatively explained why DDD generates well-calibrated PIs compared to previous methods. Our method also provides coverage probability corresponding to each PI that is simple but informative summarization of the conditional probability of the target. We can conclude that our method can provide well-calibrated and informative PIs for predictive uncertainty estimation in various fields.

heading: A DATA PRE-PROCESSING
text: We use the same dataset and pre-processing implemented in SNM-QD+ Salem et al. (2020) 1. We add two more datasets that are the Parkinson-Telemonitoring and Bike-Sharing from UCI datasets:
Parkinson Telemonitoring2: "subject id" is excluded Seoul Bike Sharing Demand 3: "date", "seasons", "holiday", and "functioning day" are excluded.
B QUANTITATIVE COMPARISON BETWEEN DDD AND DDDens
Table 2 compares DDDens with a single DDD model on the UCI benchmark datasets. DDDens has higher PICP and lower or equal MPIW over DDD throughout all datasets except for the Yacht. In the Yacht dataset, DDDens has similar MPIW and much higher PICP , so it can be considered that DDDens is high-quality. This result implies that the ensemble method improves performance in terms of the HQ principle.

heading: C MORE MULTIMODAL SAMPLES
text: We draw p̂(yi|xi) through the mixture density network in the same way as Section 5.4. The Bike Sharing dataset shows clear multimodality and DDD produce high-quality PIs compared to MVE and SQR. The Wine dataset has a discretized outcome because it is for the regression problem of guessing the grade of wine. DDD generates very short intervals for each grade position compared to others. For datasets with discretized outcomes like the Wine samples, it is much more helpful to obtain the coverage probability for each PI than the total width of PIs (e.g., grade 1 with a probability of 20%, grade 2 with a probability of 45%, and grade 3 with a probability of 30%).

Review summaries:
This paper provides the algorithm for the construction of prediction intervals composed of disjoint intervals. The authors proposed the motivation of disjoint intervals well, and the motivating example is impressive. The algorithms are also well accommodated with the statistical or learning-based prediction intervals, which contribute to the assessment of the uncertainty of prediction in general.  

The paper addresses the problem of determining prediction intervals (PI) in regression task. The prediction interval problem can be summarized as predicting a lower and upper bound between which the potential observation falls with a certain probability. The paper proposes a method to report the prediction interval as the union of disjoint intervals, in contrast with the previous methods which report a unified continuous interval. The motivation is that if the conditional density function has multiple modes, a single prediction interval may not be well descriptive of the uncertainty of the predictive model. To achieve this goal, they propose a differentiable objective function together with a Neural Network architecture that produces the union of disjoint prediction intervals. Through experiments, they show that multimodality often exists in real-world datasets and that their method manages to produce prediction intervals of higher quality (in terms of the commonly used metrics to assess the quality of the prediction intervals such as coverage probability and interval width) compared to the previous work.

This paper notes that existing approaches to predictive interval (PI) generation and evaluation assume unimodal predictive distributions. This results in unnecessarily loose PIs in the presence of multimodality in the predictive distribution or multimodality in the targets. The authors first propose to extend the definition of PI to be a union of disjoint intervals, allowing for more fidelity in evaluation. They then propose a method to explicitly generate multimodal PIs. This method is based on feeding the output of a mixture density network (conditional GMM) through a secondary NN that outputs a set of lower and upper bounds for PIs. The authors compare their method against alternative heteroscedastic noise models in terms of PICP and MPIW on 11 UCI datasets.