Distribution-Driven Disjoint Prediction Intervals for Deep Learning

This paper redefines prediction intervals ( PIs ) as the form of a union of disjoint intervals . PIs represent predictive uncertainty in the regression problem . Since previous PI methods assumed a single continuous PI ( one lower and upper bound ) , it suffers from performance degradation in the uncertainty estimation when the conditional density function has multiple modes . This paper demonstrates that multimodality should be considered in regression uncertainty estimation . To address the issue , we propose a novel method that generates a union of disjoint PIs . Throughout UCI benchmark experiments , our method improves over current state-of-the-art uncertainty quantification methods , reducing an average PI width by over 27 % . Through qualitative experiments , we visualized that the multi-mode often exists in real-world datasets and why our method produces high-quality PIs compared to the previous PI . 1 INTRODUCTION . Deep neural networks ( NNs ) show remarkable performance in predicting a target for regression problems . However , the prediction is not enough to make it trustworthy : minimization of objective functions the NN leads to network outputs which approximate the conditional averages of the target data with no information about sampling errors and prediction accuracy . Moreover , if the target is multivalue , NN output can be far from the actual target in the regression problems . Incorporating the predictive uncertainty into the deterministic approximation generated by NNs improves the reliability and credibility of the predictions . This issue is being discussed in various domains such as autonomous driving ( Feng et al. , 2018 ) , object detection ( He et al. , 2019 ) , solar energy forecasting ( Galván et al. , 2017 ) , electricity demands and price estimation ( Shrivastava & Panigrahi , 2015 ) , and sensor anomaly detection ( Pang et al. , 2017 ) . Prediction interval ( PI ) represents and quantifies predictive uncertainty in the regression problem . Pearce et al . ( 2018 ) ; Tagasovska & Lopez-Paz ( 2019 ) ; Salem et al . ( 2020 ) have recently provided competitive performance by generating a PI to estimate predictive uncertainty . PI describes predictive uncertainty for each sample in the form of two values ( lower and upper bound ) between which a potential observation falls with a certain probability ( e.g. , 95 % or 99 % ) . PI can provide the amount of uncertainty for each sample by the width of PI . It also provides the possible range of prediction by bounds . It is a self-evident principle that high-quality PI should be as narrow as possible while containing some specified proportion of data points ( hereafter referred to as the HQ principle ) . The quality of a PI is often evaluated by the metric derived from the HQ principle ( Khosravi et al. , 2010 ; Galván et al. , 2017 ; Pearce et al. , 2018 ; Tagasovska & Lopez-Paz , 2019 ; Salem et al. , 2020 ) . Previous methods estimate the regression uncertainty with a single continuous PI , but it may suffer from performance degradation in the regression having multimodality . A toy example in Figure 1 is a one-dimensional regression example that has two modes . We observe that a single continuous PI ( gray shade ) provides unnecessarily large PIs to fill in the gap between the two modes compared to disjoint PIs ( blue shade ) . This means that a single continuous PI provides low-quality PIs in terms of the HQ principle . Including intervals that are unlikely to contain future observations makes PIs less reliable . Note that this issue becomes severe as the distance between modes increases . We qualitatively confirmed that multimodality often exists in real-world regression datasets through approximating conditional probability density function . We also confirmed that state-of-the-art methods generate low-quality PIs on real-world samples with multimodality . This is covered in more detail in Section 5.4 . Considering multimodality has been successful at handling the underlying stochastic structure in various fields ( Ameijeiras-Alonso et al. , 2019 ; Lerch et al. , 2020 ) . Concerning multimodality , various works such as clustering , multi-object detection ( Yoo et al. , 2019 ) , missing data reconstruction ( Smieja et al. , 2018 ) , multiple-choice learning ( Lee et al. , 2017 ) , and multi-output prediction ( Guzman-Rivera et al. , 2014 ) have been conducted . However , recent regression uncertainty estimation studies do not consider multimodality in depth . In this work , we redefine PI as a union of disjoint PIs due to the limitation of a single continuous PI in multimodality ( Section 3 ) . Since prior PI methods and loss functions do not apply to the union of disjoint PIs , we propose a new differentiable objective function and NN architecture that produce the union of disjoint PIs ( Section 4 ) . Additionally , we use the ensemble method to boost the performance for both in- and out-of-distribution regions ( Section 5.2 ) . As a result , our method improves over current state-of-the-art methods , reducing an average PI width by 27 % throughout eleven real-world datasets ( Section 5.3 ) . In addition , our method can provide the coverage probability of each disjoint PI ( e.g. , 20 % chance of being between 1 and 3 , 75 % chance of being between 5 and 9 ) . This means that our method gives information about how reliable each interval is ( Section 5.5 ) . 2 RELATED WORK . There are two approaches for estimating the predictive uncertainty for regression problems : Bayesian and non-Bayesian . In the Bayesian approach , NN parameters are considered as a distribution , and the uncertainty is calculated by marginalizing the parameters ( Graves , 2011 ; Blundell et al. , 2015 ; Hernández-Lobato & Adams , 2015 ; Gal et al. , 2017 ; Khan et al. , 2018 ; Wu et al. , 2018 ; Yao et al. , 2019 ; Izmailov et al. , 2020 ) . Though theoretically grounded , an approximation is needed since calculating the posterior distribution of NN parameters is computationally intractable . It also requires high computational demand in the inference time . The non-Bayesian approach , on the other hand , defines the output of NN as parameters to describe the predictive uncertainty . It is usually less computational than the Bayesian approach . However , since the NN parameters are fixed , non-Bayesian methods have a limitation in expressing the model uncertainty . Therefore , the deep ensemble with random initialization is additionally used to deal with model uncertainty . Several papers in the non-Bayesian branch have recently provided competitive performance ( Lakshminarayanan et al. , 2017 ; Pearce et al. , 2018 ; Tagasovska & Lopez-Paz , 2019 ; Salem et al. , 2020 ) . Our paper focuses on the Non-Bayesian approach , especially for the regression problem . Therefore , we would take a closer look at the non-Bayesian methods by dividing them into PI and non-PI methods . As PI methods for non-Bayesian methods , Khosravi et al . ( 2010 ) propose the Lower Upper Bound Estimation ( LUBE ) method that produces PI for the first time . Followed by that , Pearce et al . ( 2018 ) propose a quality-driven ( QD ) loss function that is compatible with gradient descent optimization . They also propose an ensemble method for PI with multiple predicted lower and upper bounds to estimate the model uncertainty . Salem et al . ( 2020 ) retrofit the QD loss function and propose a new ensemble method by fitting the split normal mixture distribution ( Wallis , 2014 ) to the PI and averaging the distribution , where they name it as SNM-QD+ . It increases the robustness of the training process compared to the QD method . However , SNM-QD+ has difficulties searching hyperparameters because the loss function contains various hyperparameters to achieve the advantages . Tagasovska & Lopez-Paz ( 2019 ) propose the simultaneous quantile regression ( SQR ) and the orthonormal certificates ( OC ) to estimate data noise and model uncertainty , respectively . However , this strategy generates PI only by the SQR without an ensemble method , and model uncertainty from OC is not included in the PI . Therefore , the PI of SQR does not consider the model uncertainty . Aforementioned loss functions and methods can only generate a single continuous PI but not a union of disjoint PIs . As non-PI methods , Mean-Variance Estimation ( MVE ) ( Nix & Weigend , 1994 ) uses a NN with two output nodes that are considered as a mean and a standard deviation of the conditional probability distribution . Since NN parameters are fixed , it can not deal with the model uncertainty . Lakshminarayanan et al . ( 2017 ) demonstrate the deep ensemble of multiple MVE with random initialization improves the performance , especially in out-of-distribution regions . ( so-called MVEens ) . Fort et al . ( 2019 ) shows that ensemble with random initialization may sample different modes in function space and therefore perform well in exploring model uncertainty . 3 UNION OF DISJOINT PREDICTION INTERVALS . 3.1 PROBLEM SETUP . Consider a dataset { xi , yi } Ni=1 where xi is an input and yi is a target . For each data point { xi , yi } , the disjoint set of PIs that covers the desired given proportion γ ∈ [ 0 , 1 ] is defined as follows : PIi = J ( i ) ⋃ j=1 [ Lij , Uij ) ( 1 ) where Pr ( yi ∈ PIi ) ≥ γ and Lij ≤ Uij < Li ( j+1 ) for all j Lij and Uij is a lower and upper bound of jth PI related with ith data point . J ( i ) is the number of disjoint intervals when PI is expressed with the smallest number of disjoint intervals . That is , J ( i ) is unique for a given interval . Note J ( i ) may have a different value for each data point . The previous methods assume a single continuous PI that is J ( i ) = 1 for all i . 3.2 PERFORMANCE METRIC : PICP AND MPIW . To measure the quality of PI methods based on the HQ principle , let Prediction Interval Coverage Probability ( PICP ) and Mean Prediction Interval Width ( MPIW ) be defined as , PICP = c N where c = N∑ i=1 ci and ci = { 1 , if yi ∈ PIi 0 , otherwise ( 2 ) MPIW = 1 N N∑ i=1 J ( i ) ∑ j=1 ( Uij − Lij ) ( 3 ) PICP measures the ratio of the target that is captured within PIs while MPIW measures total length of PIs over the entire samples . According to the HQ principle , PIs should minimize MPIW subject to PICP ≥ γ ( e.g . γ = 0.95 or 0.99 ) . This metric is widely used to compare performance of PI-related methods ( Khosravi et al. , 2010 ; Pearce et al. , 2018 ; Tagasovska & Lopez-Paz , 2019 ; Salem et al. , 2020 ) . 4 DISTRIBUTION-DRIVEN-DISJOINT METHOD . We propose a learning-based method that generates distribution-driven disjoint ( DDD ) PIs , and we call our method as DDD method . The DDD method produces high-quality PIs without the assumption that J ( i ) = 1 . To generate multiple disjoint PIs with a learning-based method , we need to formulate a differentiable loss function that reflects the HQ principle . However , this is a challenging problem because c from ( 2 ) is non-differentiable . Pearce et al . ( 2018 ) ; Salem et al . ( 2020 ) proposed a QD and QD+ loss function in the form of constraint optimization by approximating c in a differentiable way . Tagasovska & Lopez-Paz ( 2019 ) employed a pinball loss which reflects HQ principle and differentiable . However , these loss functions have a limitation in that they work for a single continuous PI which has J ( i ) = 1 . To derive a new differentiable loss function ( so-called DDD loss ) for multiple disjoint intervals , we first approximate the conditional distribution given input , p̂ ( yi|xi ) , with a Gaussian mixture model . Then , we derive the DDD loss by using the cumulative density function of the Gaussian mixture ( why we call our method distribution-driven disjoint ) . Our DDD method trains NN by minimizing the DDD loss . Another major problem is that the optimal number of disjoint prediction intervals J ( i ) opt may differ for each p̂ ( yi|xi ) , making it hard to implement [ L̂ij , Ûij ) J ( i ) j=1 as an output of NN . To deal with the problem , after producing K intervals regardless of conditional density , the union process removes the overlapping part . We propose a novel architecture that can implement this in a differentiable way . Additionally , we employed a simple ensemble method to improve performance for both inand out-of-distribution observations .

Summary: This paper notes that existing approaches to predictive interval (PI) generation and evaluation assume unimodal predictive distributions. This results in unnecessarily loose PIs in the presence of multimodality in the predictive distribution or multimodality in the targets. The authors first propose to extend the definition of PI to be a union of disjoint intervals, allowing for more fidelity in evaluation. They then propose a method to explicitly generate multimodal PIs. This method is based on feeding the output of a mixture density network (conditional GMM) through a secondary NN that outputs a set of lower and upper bounds for PIs. The authors compare their method against alternative heteroscedastic noise models in terms of PICP and MPIW on 11 UCI datasets.