PIVEN: A Deep Neural Network for Prediction Intervals with Specific Value Prediction

1 INTRODUCTION . Deep neural networks ( DNNs ) have been achieving state-of-the-art results in a large variety of complex problems . These include automated decision making and recommendation systems in the medical domain ( Razzak et al. , 2018 ) , autonomous control of drones ( Kaufmann et al. , 2018 ) and self driving cars ( Bojarski et al. , 2016 ) . In many of these domains , it is crucial not only that the prediction made by the DNN is accurate , but rather that its uncertainty is quantified . Quantifying uncertainty has many benefits , including risk reduction and more reliable planning ( Khosravi et al. , 2010 ) . In regression , uncertainty is quantified using prediction intervals ( PIs ) , which offer upper and lower bounds on the value of a data point for a given probability ( e.g. , 95 % ) . Existing non-Bayesian PI generation methods can be roughly divided into two groups : a ) performing multiple runs of the regression problem , as in dropout ( Gal & Ghahramani , 2016 ) or ensemble-based methods ( Lakshminarayanan et al. , 2017 ) , then deriving post-hoc the PI from prediction variance , and ; b ) dedicated architectures for the PI generation ( Pearce et al. , 2018 ; Tagasovska & Lopez-Paz , 2019 ) . While effective , both approaches have shortcomings . On the one hand , the former group is not optimized for PIs generation , having to convert a set of sampled values into a distribution . This lack of PI optimization makes using these approaches difficult in domains such as financial risk mitigation or scheduling . For example , providing a PI for the number of days a machine can function without malfunctioning ( e.g. , 30-45 days with 99 % certainty ) is more valuable than a prediction for the specific time of failure . On the other hand , the latter group—PI-dedicated architectures —provides accurate upper and lower bounds for the prediction , but lacks in the accuracy of its value predictions . Consequently , these approaches select the middle of the interval as their value prediction , which is ( as we later demonstrate ) a sub-optimal strategy that makes assumptions regarding the value distribution within the interval . The shortcomings of PI-dedicated architectures with regard to value prediction are supported both by Pearce et al . ( 2018 ) and by our experiments in Section 5 . We propose PIVEN ( prediction intervals with specific value prediction ) , a novel approach for simultaneous PI generation and value prediction using DNNs . Our approach combines the benefits of both above-mentioned groups by producing both a PI and a value prediction , while ensuring that the latter is within the former . We follow the experimental procedure of recent works , and compare our approach to current best-performing methods : Quality-Driven PI ( QD ) by Pearce et al . ( 2018 ) ( a dedicated PI generation method ) , and Deep Ensembles ( DE ) by DeepMind ( Lakshminarayanan et al. , 2017 ) . Our results show that PIVEN outperforms QD by producing narrower PIs , while simultaneously achieving comparable results to DE in terms of value prediction . Additional analysis on large image datasets and synthetic data shows that PIVEN performs well on skewed value distributions and can be effectively combined with pre-trained DNN architectures . 2 RELATED WORK . Modeling uncertainty in deep learning has been an active area of research in recent years ( Pearce et al. , 2018 ; Qiu et al. , 2019 ; Gal & Ghahramani , 2016 ; Lakshminarayanan et al. , 2017 ; Keren et al. , 2018 ; Kendall & Gal , 2017 ; Geifman et al. , 2018 ; Ovadia et al. , 2019 ) . Studies in uncertainty modeling and regression can be generally divided into two groups : PI-based and non-PI-based . Non-PI approaches utilize both Bayesian ( MacKay , 1992 ) and non-Bayesian approaches . The former methods define a prior distribution on the weights and biases of a neural net ( NN ) , while inferring a posterior distribution from the training data . Non-Bayesian methods ( Gal & Ghahramani , 2016 ; Lakshminarayanan et al. , 2017 ; Qiu et al. , 2019 ) do not use initial prior distributions . In ( Gal & Ghahramani , 2016 ) , Monte Carlo sampling was used to estimate the predictive uncertainty of NNs through the use of dropout over multiple runs . A later study ( Lakshminarayanan et al. , 2017 ) employed a combination of ensemble learning and adversarial training to quantify model uncertainty . In an expansion of a previously-proposed approach ( Nix & Weigend , 1994 ) , each NN was optimized to learn the mean and variance of the data , assuming a Gaussian distribution . Recently , Qiu et al . ( 2019 ) proposed a post-hoc procedure using Gaussian processes to measure uncertainty . PI-based approaches are designed to produce a PI for each sample . Keren et al . ( 2018 ) propose a post-processing approach that considers the regression problem as one of classification , and uses the output of the final softmax layer to produce PIs . Tagasovska & Lopez-Paz ( 2019 ) propose the use of a loss function designed to learn all conditional quantiles of a given target variable . LUBE ( Khosravi et al. , 2010 ) consists of a loss function optimized for the creation of PIs , but has the caveat of not being able to use stochastic gradient descent ( SGD ) for its optimization . A recent study ( Pearce et al. , 2018 ) inspired by LUBE , proposed a loss function that is both optimized for the generation of PIs and can be optimized using SGD . Each of the two groups presented above tends to under-perform when applied to tasks for which its loss function was not optimized : Non-PI approaches produce more accurate value predictions , but are not optimized to produce PI and therefore produce bounds that are less tight . PI-based methods produce tight bounds , but tend to underperform when producing value predictions. . Recent studies ( Kivaranovic et al. , 2019 ; Romano et al. , 2019 ) attempted to produce both value predictions and PIs by using conformal prediction with quantile regression . While effective , these methods use a complex splitting strategy , where one part of the data is used to produce value predictions and PIs , while the the other part is to further adjust the PIs . Recently , Salem et al . ( 2020 ) proposed a method for combining the two , together with post-hoc optimization . Contrary to these approaches , PIVEN produces PIs with value predictions in an end-to-end manner by relying on a novel loss function . 3 PROBLEM FORMULATION . In this work we consider a neural network regressor that processes an input x ∈ X with an associated label y ∈ R , where X can be any feature space ( e.g. , tabular data , age prediction from images ) . Let ( xi , yi ) ∈ X × R be a data point along with its target value . Let Ui and Li be the upper and lower bounds of PIs corresponding to the ith sample . Our goal is to construct ( Li , Ui , yi ) so that Pr ( Li ≤ yi ≤ Ui ) ≥ 1− α . We refer to 1− α as the confidence level of the PI . We define two quantitative measures for the evaluation of PIs , as defined in Khosravi et al . ( 2010 ) . First , we define coverage as the ratio of dataset samples that fall within their respective PIs . We measure coverage using the prediction interval coverage probability ( PICP ) metric : PICP : = 1 n n∑ i=1 ki ( 1 ) where n denotes the number of samples and ki = 1 if yi ∈ ( Li , Ui ) , otherwise ki = 0 . Next , we define a quality metric for the generated PIs with the goal of producing as tight a bound as possible while maintaining adequate coverage . We define the mean prediction interval width ( MPIW ) as , MPIW : = 1 n n∑ i=1 Ui − Li ( 2 ) When combined , these metrics enable us to comprehensively evaluate the quality of generated PIs . 4 METHOD . 4.1 SYSTEM ARCHITECTURE . The proposed architecture is presented in Figure 1 . It consists of three components : Backbone block . The main body block , consisting of a varying number of DNN layers or sub-blocks . The goal of this component is to transform the input into a latent representation that is then provided as input to the other components . It is important to note that PIVEN supports any architecture type ( e.g. , dense , convolutions ) that can be applied to a regression problem . Moreover , pre-trained architectures can also be used very easily , with PIVEN being added on top of the architecture for an additional short training . For example , we use pre-trained VGG-16 and DenseNet architectures in our experiments . Upper & lower-bound heads . L ( x ) and U ( x ) produce the lower and upper bounds of the PI respectively , such that Pr ( L ( x ) ≤ y ( x ) ≤ U ( x ) ) ≥ 1− α where y ( x ) is the value prediction and 1− α is the predefined confidence level . Auxiliary head . The auxiliary prediction head , v ( x ) , enables us to produce a value prediction . v ( x ) does not produce the value prediction directly , but rather produces the relative weight that should be given to each of the two bounds . We define the value prediction using , y = v · U + ( 1− v ) · L ( 3 ) where v ∈ ( 0 , 1 ) . By expressing y as a function of U and L , we ( a ) bound them together and improve their performance ( see Section 4.4 ) , and ; ( b ) ensure that the value prediction falls within the PI . The goal of the loss function of our approach , denoted as LPIV EN , is to balance two conflicting goals : generating tight PIs and producing accurate value predictions . LPIV EN achieves this goal by synchronizing two loss functions that are designed to assess each of the two goals described above . 4.2 NETWORK OPTIMIZATION . Our goal is to generate narrow PIs , measured by MPIW , while maintaining the desired level of coverage PICP = 1− α . However , PIs that fail to capture their respective data point should not be encouraged to shrink further . We follow the derivation presented in Pearce et al . ( 2018 ) and define captured MPIW ( MPIWcapt ) as the MPIW of only those points for which Li ≤ yi ≤ Ui , MPIWcapt : = 1 c n∑ i=1 ( Ui − Li ) · ki ( 4 ) where c = ∑n i=1 ki . Hence , we seek to minimize MPIWcapt subject to PICP ≥ 1− α : θ∗ = arg min θ ( MPIWcapt , θ ) s.t PICPθ ≥ 1− α where θ is the parameters of the neural net . To enforce the coverage constraint , we utilize a variant of the well-known Interior Point Method ( Potra & Wright , 2000 ) , resulting in an unconstrained loss : LPI = MPIWcapt , θ + √ n · λΨ ( 1− α− PICPθ ) Ψ ( x ) : = max ( 0 , x ) 2 where Ψ is a quadratic penalty function , n is the batch size which was included because a larger sample size would increase confidence in the value of PICP ( thus increasing the loss ) and λ is a hyperparameter controlling the relative importance of width vs. coverage . We use a default value of λ = 15 in all our experiments , and perform further analyze of this parameter in Section 6.3 In practice , optimizing the loss with a discrete version of k ( see eq . 4 ) fails to converge , because the gradient is always positive for all possible values . We therefore define a continuous version of k , denoted as ksoft = σ ( s · ( y− L ) ) σ ( s · ( U− y ) ) , where σ is the sigmoid function , and s > 0 is a softening factor . The final version of LPI uses the continuous and discrete versions of k in its calculations of the PICP and MPIWcapt metrics , respectively . The discrete k enables us to assign a score of zero to points outside the interval , while ksoft produces continuous values that enable gradient calculations . Neural networks optimized by the abovementioned objective are able to generate well-calibrated PIs . However , they have two significant drawbacks—overfitting , and lack of value prediction capabilities . Overfitting . The MPIWcapt term in LPI focuses only on the fraction c of the training set where the data points are successfully captured by the PI . As a result , the network is likely to overfit to a subset of the data on which it was able to perform well in the first place . Our reasoning is supported by our experiments in sections 5 and 6 . Lack of value prediction capabilities . In its current form , LPI is not able to perform value prediction , i.e. , returning accurate prediction for the regression problem . To overcome this limitation , one can return the middle of the PI , as done in Pearce et al . ( 2018 ) and Tagasovska & Lopez-Paz ( 2019 ) . This approach performs well only if the dataset value distribution is uniform or Gaussian ( i.e. , where choosing the mean minimizes prediction loss ) . For skewed distributions , as is often the case in the real world , this approach leads to sub-optimal results , as we show in Section 5.4 . We propose a novel loss function that combines the generation of both PIs and value predictions . To optimize the output of v ( x ) ( the auxiliary head ) , we minimize the standard regression loss , Lv = 1 n n∑ i=1 ` ( vi · Ui + ( 1− vi ) · Li , yi ) ( 5 ) where ` is a regression objective against the ground-truth . Our final loss function is a convex combination of LPI , and the auxiliary loss Lv . Thus , the overall training objective is : LPIV EN = βLPI + ( 1− β ) Lv ( 6 ) where β is a hyperparameter that balances the two goals of our approach : producing narrow PIs and accurate value predictions . In all our experiments , we chose to assign equal priorities to both goals by setting β = 0.5 . We performed no hyperparameter optimization .

Summary: This paper proposes a new objective function for training regression networks with prediction intervals. The goal is to provide tight confidence bounds to accompany predictions, which is of course important for practical deployments of ML systems where uncertainty quantification is critical. Previous work has largely assumed that uncertainty is symmetrical, or even Gaussian distributed, which is often not the case in practice. The innovation of this work is to simultaneously predict the bounds for a given confidence level and a point prediction within those bounds, which is not necessarily the mean. It is accomplished by predicting 3 values: an upper bound, a lower bound, and a mixing parameter that allows making the point estimate as a weighted sum of the bounds. Experimental results are provided on 3 datasets, with 2 baseline methods compared against.