ChePAN: Constrained Black-Box Uncertainty Modelling with Quantile Regression

1 INTRODUCTION . The present paper proposes a novel method for adding aleatoric uncertainty estimation to any pointwise predictive system currently in use . Considering the system as a black box , i.e . avoiding any hypothesis about the internal structure of the system , the method offers a solution to the technical debt debate . The concept of technical debt was introduced in 1992 to initiate a debate on the long-term costs incurred when moving quickly in software engineering ( Sculley et al . ( 2015 ) ; Cunningham ( 1992 ) ) . Specifically , most of the predictive systems currently in use have previously required much effort in terms of code development , documentation writing , unit test implementation , preparing dependencies or even their compliance with the appropriate regulations ( e.g. , medical ( Ustun & Rudin ( 2016 ) ) or financial models ( Rudin ( 2019 ) ) may have to satisfy interpretability constraints ) . However , once the system is being used with real-world problems , a new requirement can arise regarding the confidence of its predictions when the cost of an erroneous prediction is high . That being said , replacing the currently-in-use system may not be advisable in the short term . To address this issue , k “ 0 the coefficients of the integrated polynomial , β the black box function and P the conditional prediction of the quantile τ . the aim of this work is to report any information that is useful for auditing the system ’ s associated uncertainty without modifying its predictions . In general terms , sources of uncertainty can be understood by analysing the conditional members of this joint distribution : ppy , xq “ ş M ppy | x , MqppM | xqppxq dM where M P M is the family ( assumed non-finite ) of models being considered . Not all methods developed to model uncertainty can be applied in the black-box scenario , since the main hypothesis is that the black box is a fixed single model and unknown internally . Here , we refer specifically to those solutions that model epistemic uncertainty , which requires modelling ppM | xq . By epistemic , we mean that uncertainty which can derive from ignorance about the model , including , for example , ensemble models ( Lakshminarayanan et al . ( 2017 ) ) , Bayesian neural networks ( Rasmussen ( 1996 ) ; Blundell et al . ( 2015 ) ; Hernández-Lobato & Adams ( 2015b ) ; Teye et al . ( 2018 ) ) or MC-Dropout ( Gal & Ghahramani ( 2016 ) ) . However , the black box could be a non-parametric predictive system or even a handcrafted rulebased system , as shown in Figure 1 . Hence the reason for studying aleatoric uncertainty ( Der Kiureghian & Ditlevsen ( 2009 ) ; Kendall & Gal ( 2017 ) ; Brando et al . ( 2019 ) ) , which originates from the variability of possible correct answers given the same input data , ppy | xq . This type of uncertainty can be tackled by modelling the response variable distribution . For instance , imposing a conditional normal distribution where the location parameter is the black-box function and the corresponding scale parameter is learnt . However , the more restricted the assumptions made about this distribution , the more difficult it will be to model heterogeneous distributions . One solution to this limitation is the type of regression analysis used in statistics and econometrics known as Quantile Regression ( QR ) , which will provide a more comprehensive estimation . Unlike classic regression methods , which only estimate a selected statistic such as the mean or the median , QR allows us to approximate any desired quantile . The main advantage of this method is that it allows confidence intervals to be captured without having to make strong assumptions about the distribution function to be approximated . Recently , several works ( Dabney et al . ( 2018a ) ; Tagasovska & Lopez-Paz ( 2018 ) ; Brando et al . ( 2019 ) ) have proposed a single deep learning model that implicitly learns all the quantiles at the same time , i.e . the model can be evaluated for any real value τ P r0 , 1s to give a pointwise estimation of any quantile value of the response variable . Nevertheless , these QR solutions are not directly applicable to the uncertainty modelling of a black box because the predicted quantiles need to be linked to the black-box prediction in some way . In the present paper , we propose a novel method for QR based on estimating the derivative of the final function using a Chebyshev polynomial approximation to model the uncertainty of a blackbox system . Specifically , this method disentangles the estimation of a selected statistic β of the distribution ppy | xq from the estimation of the quantiles of ppy | xq ( shown in Figure 2 ) . Hence , our method is not restricted to scenarios where we can jointly train both estimators , but can also be applied to pre-existing regression systems as a wrapper that produces the necessary information to evaluate aleatoric uncertainty . Additionally , the proposed method scales to several real-world data sets . This paper is organised as follows . Section 2 states the real-world motivation of the current research as well as the contribution it will be presented . Section 3 introduces the problem of QR and reviews the classic approach to use with neural networks , showing how it can not be applied directly to constrained black-box uncertainty modelling . Section 4 explores an approach for modelling the derivative of a function using neural networks . The two previous sections provide the baseline for developing our proposed model and its properties , which is presented in Section 5 . And finally , in Section 6 , we show how our model can be applied in large data sets and defines a new way of modelling the aleatoric uncertainty of a black box . The results are then summarised in the conclusion . 2 RESEARCH GOAL AND CONTRIBUTION . The present article was motivated by a real-world need that appears in a pointwise regression forecasting system of a large company . Due to the risk nature of the internal problem where it is applied , uncertainty modelling is important . However , similarly to the medical or financial cases presented in the introduction , interpretability requirements were essential in defining the model currently used by the company , which does not report confidence any prediction made . The need for this research arises in cases where the replacement of the aforementioned system is not advisable in the short term , despite the ongoing need for the uncertainty estimation of that system . Definition of constrained black-box uncertainty modelling From the probabilistic perspective , solving a regression problem involves determining a conditional density model , qpy | xq . This model fits an observed set of samples D “ pX , Y q “ pxi , yiq | xi P RD , yi P R ( n i “ 1 , which we assume to be sampled from an unknown distribution ppy | xq . i.e . the real data . Given this context , the pointwise forecasting system mentioned above is a function , β : RD Ñ R , which tries to approximate a certain conditional summary statistic ( a percentile or moment ) of ppy | xq . Regarding the notation , we will call the “ constraint ” the known or assumed summary statistic that is approximated by βpxq ( e.g . if β is reducing the mean square error , then it corresponds to the conditional mean . Otherwise , if it minimises the mean absolute error , it corresponds to the median ) . Importantly , in the constrained black-box uncertainty modelling context , the mismatch between the real conditional statistic and the black box , β , becomes a new source of aleatoric uncertainty that is different from the one derived from the data . However , the way to model it continues to be by estimating ppy | xq . Therefore , a poorly estimated β will impact the modelling of ppy | xq , given that we always force the constraint to be satisfied ( as shown in Figure 3 of the Experiment section ) . So far , we have attempted to highlight the fact that we do not have a strong hypothesis about the internals of this β function , we have only assumed that it approximates a certain statistic of ppy | xq . Accordingly , we call this function the “ constrained black box ” . This flexible assumption will enable us to consider several pointwise models as β , as shown in Figure 1 . The overall goal of the present article is , taking a pre-defined black box βpxq that estimates a certain conditional summary statistic of ppy | xq , to model qpy | xq under the constraint that if we calculate the summary statistic of this predicted conditional distribution , it will correspond to βpxq . As mentioned in the Introduction , since we have a fixed black box , we are unable to apply Bayesian techniques such as those that infer the distribution of parameters within the model , ppM | xq . In general , even though they are very common techniques in generic uncertainty modelling , no such epistemic uncertainty techniques can be applied in this context due to the limitation of only having a single fixed model . In addition , it should be noted that not all models that estimate ppy | xq can be used in the constrained black-box uncertainty modelling context . To solve this problem , we require models that predict qpy | xq but also force the chosen conditional summary statistic of qpy | xq to have the same value as βpxq . The main contribution of this work is to present a new approach that allows us not only to outperform other baseline models when tackling this problem , but also to decide which kind of constraint we wish to impose between βpxq and qpy | xq . The qpy | xq will be approximated using Quantile Regression ( explained in Section 3 ) and the constraint will be created considering the integration constant of the qpy | xq derivative ( shown in Section 5.1 ) . 3 CONDITIONAL QUANTILE REGRESSION . In Quantile Regression ( QR ) , we estimate q in a discrete manner by means of quantiles , which does not assume any typical parametric family distribution to the predicted p , i.e . it goes beyond central tendency or unimodality assumptions . For each quantile value τ P r0 , 1s and each input value x P RD , the conditional quantile function will be f : r0 , 1s ˆ RD Ñ R. In our case , we use deep learning as a generic function approximator ( Hornik et al . ( 1989 ) ) to build the model f , as we shall see later . Consequently , f is a parametric function that will be optimised by minimising the following loss function with respect to their weights w , Lpx , y , τq “ ` y ´ fwpτ , xq ˘ ¨ ` τ ´ 1ry ă fwpτ , xqs ˘ ( 1 ) where 1rcs denotes the indicator function that verifies the condition c. Equation 1 is an asymmetric convex loss function that penalises overestimation errors with weight τ and underestimation errors with weight 1´ τ . Recently , different works ( Dabney et al . ( 2018b ; a ) ; Wen et al . ( 2017 ) ) have proposed deep learning models that minimise a QR loss function similar to Equation 1 . For instance , in the field of reinforcement learning , the Implicit Quantile Network ( IQN ) model was proposed ( Dabney et al . ( 2018a ) ) and subsequently applied to solve regression problems as the Simultaneous Quantile Regression ( SQR ) model ( Tagasovska & Lopez-Paz ( 2019 ) ) or the IQN in ( Brando et al . ( 2019 ) ) . These models consist of a neural network ψ : r0 , 1s ˆ RD Ñ R such that it directly learns the function f that minimises Equation 1 , i.e . f “ ψ . In order to optimise ψ for all possible τ values , these models pair up each input x with a sampled τ „ Up0 , 1q from a uniform distribution in each iteration of the stochastic gradient descent method . Thus , the final loss function is an expectation over τ of Equation 1 . However , these QR models can not be applied to the constrained black-box scenario , given that they do not link their predicted quantiles with a pointwise forecasting system in a constrained way ( Section 5.1 ) . Other models , such as quantile forests , have a similar limitation . In the next section , we introduce the other main part required to define our proposed method .

Summary: This paper proposes a novel approach to modeling uncertainty, as an layer added-on to an otherwise black-box system. The ChePAN uses a neural network to estimate per-quantile roots of a chebyshev polynomial, then uses a quantile regression loss to fit these coefficients using backpropagation. Importantly, the Chebyshev polynomial formulation gives the practitioner some flexibility around deciding which statistic of the conditional CDF should be matched by the black box system. Examples are given for matching the 0 and 1 quantiles, for cases of min and max estimation, as well as matching either of median or mean.