ChePAN: Constrained Black-Box Uncertainty Modelling with Quantile Regression
Authors: Axel Brando
Joan Gimeno
Jose Antonio Rodriguez-Serrano
Jordi Vitria
Venue:  (2021)
Keywords: Uncertainty modelling
Deep Learning
Black Box
Aleatoric
Quantile Regression
Chebyshev Polynomial
Neural networks

Abstract: Most predictive systems currently in use do not report any useful information for auditing their associated uncertainty and evaluating the corresponding risk. Taking it for granted that their replacement may not be advisable in the short term, in this paper we propose a novel approach to modelling confidence in such systems while preserving their predictions. The method is based on the Chebyshev Polynomial Approximation Network (the ChePAN), a new way of modelling aleatoric uncertainty in a regression scenario. In the case addressed here, uncertainty is modelled by building conditional quantiles on top of the original pointwise forecasting system considered as a black box, i.e. without making assumptions about its internal structure. Furthermore, the ChePAN allows users to consistently choose how to constrain any predicted quantile with respect to the original forecaster. Experiments show that the proposed method scales to large size data sets and transfers the advantages of quantile regression to estimating black-box uncertainty.

Under review as a conference paper at ICLR 2021
CHEPAN: CONSTRAINED BLACK-BOX UNCERTAINTY MODELLING WITH QUANTILE REGRESSION
Anonymous authors Paper under double-blind review
ABSTRACT
Most predictive systems currently in use do not report any useful information for auditing their associated uncertainty and evaluating the corresponding risk. Taking it for granted that their replacement may not be advisable in the short term, in this paper we propose a novel approach to modelling confidence in such systems while preserving their predictions. The method based the Chebyshev Polynomial Approximation Network (the ChePAN), a new way of modelling aleatoric uncer- tainty a regression scenario. In the case addressed here, uncertainty is modelled by building conditional quantiles on top of the original pointwise forecasting sys- considered black box, i.e. without making assumptions about its internal structure. Furthermore, the ChePAN allows users to consistently choose how to constrain any predicted quantile with respect the original forecaster. Experi- ments show that the proposed method scales to large size data sets and transfers the advantages of quantile regression to estimating black-box uncertainty.
INTRODUCTION
Constrained black-box uncortainty modelling in context
Illustrative problem Predicting the temperature by looking out off the window.
Predictive system
y
21RC
1ply1=)
Le. any quaatile of ply
a rulo-buun bassod system
subject cervan statieii
ply x) teg. the camn. the mastian, mustuum minisuma).
OC ?
pointwise forerasting model
uncectality Wingger
Figure 1: Description of the uncertainty modelling black-box predictive system, 3. This mod- elling is done by means of an uncertainty wrapper (the only part of the ChePAN that requires a neural network), which produces all of the distribution p(y x) as quantiles, Op(y x)) The ChePAN ensures that the original prediction of 3 corresponds desired statistic of p x). i.e. the constraint.
The present paper proposes a novel method for adding aleatoric uncertainty estimation any point- wise predictive system currently in use. Considering the system as a black box, i.e. avoiding any hy- pothesis about the internal structure the system, the method offers a solution the technical debt debate. The concept of technical debt was introduced in 1992 to initiate a debate on the long- -trm costs incurred when moving quickly in software engineering (Sculley (1992)). Specifically, most of the predictive systems currently in use have previously required much effort in terms of code development, documentation writing, unit test implementation, preparing de- pendencies or even their compliance with the appropriate regulations (e.g., medical (Ustun & Rudin (0066) or financial models (Rudin (2019) may have to satisfy interpretability constraints). How- ever, once the system is being used with real-world problems, the confidence its predictions when the cost of an erroneous prediction is high. That being said, replacing the currently-in-uu system may not be advisable in the short term. To address this issue,
al. (2015): Cunningham
requirement
arise regarding

---

Under review as a conference paper at ICLR 2021
DCT
Iotegration
Pta-1
Cd-1
Cd-1
mom fixed root evaluation
Pia-2
Cd-2 2
Cd
P(T, x:;
Pti
C1
C₁
Pip
CO
Co
pointwise prodictive
T
Figure 2: Graphic representation of the ChePAN. For any degree d. {pt.) ) are evaluations of the initial Chebyshev polynomial expansion, [= their coefficients, {C*) )- the coefficients of the integrated polynomial, 33 the black box function and P the conditional prediction of the quantile т.
the aim this work is to report any information that is useful for auditing the system's associated uncertainty without modifying its predictions.
In general terms, sources of uncertainty can be understood by analysing the conditional members of this joint distribution: p(y, æ) Shc ply x, MMM x)) (x) dM where M M is the family (assumed non-finite) of models being considered.
Not all methods developed to model uncertainty can be applied in the black-box scenario, since the main hypothesis is that the black box is a fixed single model and unknown internally. Here, refer specifically to those solutions that model epistemic uncertainty, which requires modelling p(M M æ). By epistemic, we mean that uncertainty which can derive from ignorance about the model, including, for example, ensemble models (Lakshminarayanan et al. (2017), Bayesian neural networks (Rasmussen) (996)); Blundell et al. (2015); Hernández- & Adams (015b): Teye al. (018)) or MC-Dropoud (Gal & Ghahramani (2016)). (2ah)).aan(2016)).
However, the black box could be a non-parametric predictive system or even a handcrafted rule- based system, as shown in Figure [ Hence the reason for studying aleatoric uncertainty (Der Ki- ureghian& & Ditlevsen (2009); Kendall & Gal (2017); [Brando et al. (2019)), which originates from the variability of possible correct answers given the same input data, ply æ). This type of un- certainty can be tackled by modelling the response variable distribution. For instance, imposing conditional normal distribution where the location parameter is the black-box function and the cor- responding scale parameter is learnt. However, the more restricted the assumptions made about this distribution, the more difficult it will be to model heterogencous distributions. One solution to this limitation is the type of regression analysis used in statistics and econometrics known Quantile Regression (QR), which will provide a more comprehensive estimation.
Unlike classic regression methods, which only estimate selected statistic such as the mean the median, QR allows us to approximate any desired quantile. The main advantage of this method is that it allows confidence intervals to be captured without having make strong assumptions about the distribution function to be approximated.
Recently, several works (Dabney al. (2018a); Tagasovska & Lopez-Paz 2018 Brando et al. (2019) have proposed single deep learning model that implicitly learns all the quantiles at the same time, I.e. the model can be evaluated for any real value 7 e [ő, 1J to give pointwise estimation of any quantile value of the response variable. Nevertheless, these QR solutions are not directly applicable to the uncertainty modelling of black box because the predicted quantiles need to be linked to the black-box prediction in some way.
In the present paper, we propose a novel method for QR based on estimating the derivative of the final function using a Chebyshev polynomial approximation to model the uncertainty of black- box system. Specifically, this method disentangles the estimation of a selected statistic 33 of the distribution p(u x) from the estimation of the quantiles of p(y x) (shown in Figure Z). Hence, our method is not restricted to scenarios where we can jointly train both estimators, but can also be applied to pre-existing regression systems as wrapper that produces the necessary information to evaluate aleatoric uncertainty. Additionally, the proposed method scales to several real-world data sets.
2

---

Under review as a conference paper at ICLR 2021
This paper is organised as follows. Section 2 states the real-world motivation of the current search as well as the contribution it will be presented. Section introduces the problem of QR and reviews the classic approach to use with neural networks, showing how it cannot be applied directly to constrained black-box uncertainty modelling. Section Saatiinnexxiae 4xxplores an approach for modelling the derivative of function using neural networks. The two previous sections provide the baseline for developing our proposed model and its properties, which is presented in Section [ And finally, in Section 6 we show how our model can be applied in large data sets and defines new way of mod- elling the aleatoric uncertainty of a black box. The results are then summarised in the conclusion.
2 RESEARCH GOAL AND CONTRIBUTION
The present article was motivated by real-world need that appears in a pointwise regression fore- casting system of a large company. Due to the risk nature of the internal problem where it is applied, uncertainty modelling is important. However, similarly the medical or financial cases presented in the introduction, interpretability requirements were essential in defining the model currently used by the company, which does not report confidence any prediction made. The need for this research arises in cases where the replacement of the aforementioned system is not advisable in the short term, despite the ongoing need for the uncertainty estimation of that system.
Definition of constrained black-box uncertainty modelling
From the probabilistic perspective, solving a regression problem involves determining a condi- tional density model, qly æ). This model fits an observed set of samples D (X. Y) {(2,, 36) xx IR. Mi IR. | which we assume to be sampled from an unknown distribution p(y x). i.e. the real data. Given this context, the pointwise forecasting system mentioned above is function, 3: RD - R. which tries to approximate a certain conditional summary statistic (a percentile or moment) of p(y =z).
Regarding the notation, we will call the "constraint" the known or assumed summary statistic that approximated by ((x) (e.g. if 3 is reducing the mean square error, then it corresponds to the conditional mean. Otherwise, if it minimises the mean absolute error, it corresponds to the median).
Importantly, in the constrained black-box uncertainty modelling context, the mismatch between the real conditional statistic and the black box, B, becomes a new source of aleatoric uncertainty that different from the one derived from the data. However, the way to model it continues to be by estimating p(y x). Therefore, poorly estimated / will impact the modelling of p(y | I), given that we always force the constraint to be satisfied (as shown in Figure 3 the Experiment section).
So far, we have attempted to highlight the fact that do not have strong hypothesis about the internals of this 3 function, we have only assumed that it approximates a certain statistic of p(y x)) Accordingly, we call this function the "constrained black box". This flexible assumption will enable us to consider several pointwise models as B, as shown in Figure 1
The overall goal of the present article is, taking a pre-defined black box (3(x)) that estimates certain conditional summary statistic of p(y) x). to model q(3) x)) under the constraint that if we calculate the summary statistic of this predicted conditional distribution, it will correspond to 3(x).
As mentioned in the Introduction, since we have a fixed black box, we are unable to apply Bayesian techniques such those that infer the distribution of parameters within the model, p(M x). In general, even though they are very common techniques in generic uncertainty modelling, no such epistemic uncertainty techniques can be applied in this context due to the limitation of only having single fixed model.
In addition, it should be noted that not all models that estimate p④y .) can be used in the con- strained black-box uncertainty modelling context. To solve this problem, we require models that predict qyy x) but also force the chosen conditional summary statistic of g(y x)) to have the same value (xx) The main contribution of this work is to present a new approach that allows us not only to outperform other baseline models when tackling this problem, but also to decide which kind of constraint we wish to impose between 3xx) and g(y x). The g(y x) will be approximated using Quantile Regression (explained in Section 3 and the constraint will be created considering the integration constant of the ((y) x)) derivative (shown in Section 5.1
3

---

Under review as a conference paper at ICLR 2021
3 CONDITIONAL QUANTILE REGRESSION
In Quantile Regression (QR), we estimate @ in discrete manner by means of quantiles, which does not assume any typical parametric family distribution to the predicted p. i.e. it goes beyond central lendency or unimodality assumptions.
For each quantile value [0. 1] and each input value RD, the conditional quantile function will be f: [0. x IR_ - R. In our case, we use deep learning as a generic function approximator (Hornik et al. (189)) build the model f. as we shall see later. Consequently, f is a paramet- ric function that will be optimised by minimising the following loss function with respect to their weights w.
[( y, r) = (u - (T, æ)) (r I[V fw (T,
(1)
where [[c] denotes the indicator function that verifies the condition c. Equation | is an asymmetric convex loss function that penalises overestimation errors with weight 7 and underestimation errors with weight T.
Recently, different works (Dabney et al. (2018b Wen et al. (017]) have proposed deep learning models that minimise a QR loss function similar to Equation| For instance, the field of reinforce-
ment learning, the Implicit Quantile Network (IQN) model was proposed (Dabney et al. (201 8a))
and subsequently applied to solve regression problems as the Simultaneous Quantile Regression (SQR) model (Tagasovska & Lopez-Paz (2019) the IQN in (Brando al. (2019)). These models consist of a neural network ıb: [0, 1] R
IR such that it directly learns the function f that
minimises Equation| []] i.c. f v. In order optimise & for all possible values, these models pair up each input a with a sampled v 44(0, 1) from uniform distribution in each iteration of the stochastic gradient descent method. Thus, the final loss function is an expectation over т of Equation]]
However, these QR models cannot be applied the constrained black-box scenario, given that they do not link their predicted quantiles with pointwise forecasting system in constrained way (Section ). Other models, such as quantile forests, have a similar limitation. In the next section, we introduce the other main part required to define our proposed method.
4 MODELLING THE DERIVATIVE WITH A NEURAL NETWORK
Recently, a non-QR approach was proposed to build a monotonic function based on deep learning: the Unconstrained Monotonic Neural Network (UMNN) (Wehenkel & Louppe (2019). The UMNN estimates the derivative of a function by means of neural network, é, which has its output restricted to strictly positive values, i.e. approaching (2) such that
H(2) S ((t) dt H(O).
Therefore, if the neural network 6(2) OH (z) O, this is in fact sufficient condition force (2)) to be monotone.
To compute the integral of 2H the UMNN approximates the integral of Equation 2 using the 05 Clenshaw-Curts quadrature, which has closed expression. The UMNN is designed to obtain general monotonic function with respect to all the model inputs, 2. but our interest is to build a partial monotonic function with respect the quantile value, as will explain hereafter.
The partial monotonic function will be obtained using the Clenshaw- Curtiss Network (CCN) model, which is extension of the UMNN model introduced in Section A.3 of the Appendix and an intermediate step we took to arrive at the main proposal of the current article. Importantly, we have not included it in the main article because it cannot be applied the constrained black-box uncertainty modelling scenario (as described in Section A.3).
5 CHEPAN: THE CHEBYSHEV POLYNOMIAL APPROXIMATION NETWORK
In this section, we will extend the UMNN to a model that is only monotonic with respect to the quantile input Moreover, will exploit the fact that the quantile domain is in [0, 1] provide
4

---

Under review as a conference paper at ICLR 2021
an approach which is uniformly defined over all of the interval. We call this approach the Cheby- shev Polynomial Approximation Network (ChePAN), which allows us to transfer the advantages of quantile regression to the constrained uncertainty modelling of black box.
As Figure 2 shows, the ChePAN contains a neural network Φ: [0, x IPD - R. that only pro- duces positive outputs and models the derivative of the final function with respect to T. The goal is to optimise the neural networks Φ(T, x) by calculating the coefficients a truncated Chebyshev poly- nomial expansion p(T, ax; d) of degree with respect to т. That is, we will use Chebyshev poly- nomial (described in Section A.1 of the Appendix) to give a representation of the neural network, ø. uniformly defined in r E [o, 1]. After that, we will use its properties to model the uncertainty of black box in a constrained way (described in Section 5.1).
Internally, the ChePAN considers a finite mesh of quantile values, called Chebyshev roots, (1x)d=] = [0, 1J and defined by
T(k
T(k +
1 + -2' O
ts
cos
The truncated Chebyshev expansion of a function can be interpreted as a linear transformation using a set of evaluations of & at the roots, i.e. {0(42. x) This linear transformation gives a vector of coefficients, which are known as Chebyshev coefficients and depend on i.e. {cx (2:)))) ) as illustrated in Figure 2
The implementation of linear transformation generally has square complexity. However, the transformation involved in Chebyshev coefficients can be computed efficiently with a Odd complexity. In fact, the algorithm that speeds the computation is based on the Fast Fourier Transform (FFT) and known as the Discrete Cosine Transform of type-II (DCT-II) (discussed in Section A. of the Appendix).
Once the Chebyshev coefficients (x) have been computed, we can write them in a linear combi- nation of Chebyshev polynomials Tt,(t,, i.e.
1
d-1
p(T, x;; d) = 2 20n((r) +
Σ Ci (x) - 1),
(4)
where (1) defined recurrently as (o(() 1, (i()) = and (() 21)x(() T()
for k > 1. These polynomials T7 do not need to be explicitly computed to evaluate p on quantile (Clenshaw (1955)).
Note that, given the construction of the coefficients (x(x), the p(fR, x; d) is equal to 6(fx, æ) at each of the root points fą. These equalities must be understood in terms of machine precision in the numerical representation system, classically ~ 10-16 in double-precisiie precision arithmetic. In Figure we denote this root evaluation step as Pt..
or ~ 10-3 in single-
The final goal is to provide P(7, æx; d) so that it approximates the integral of p. that is So p(t, x; d) dt. Specifically, the integral will also be the integral of the neural network o,
P(T, d) & ((7, 2)) [ o(t, x)) dt K(x).
(5)
Since Φ(T, æ) is defined as positive for all T E [O, 1]. then P(T, æ; d) will be an increasing function with respect to
Additionally, given that p(r, x; d) is a Chebyshev polynomial (defined in Equation 4 its integral w.r.t. is simply the integral of the Chebyshev polynomial Tg, which corresponds shev polynomial. Using the recurrent definition of Tg, we deduce the indefinite integrals
Cheby-
(1(() dt - (1),) S [T:00 dt
Tz(t)
(o())
J (T((1) di Tt-1(t)
Th+1(t)
(6)
4
4
2(k + 1)'
which leads to the conclusion that P can be given in terms of Chebyshev coefficients as well. Thus,
Co(x) + Σ Cx(e))((27-1), 1),
P(T, x; d) = 2
(7)
5

---

Under review as a conference paper at ICLR 2021
where the coefficients (x) have recurrent expression in terms of Toeplitz matrix (see Clenshaw 1955 ). Indeed, by ordering the coefficients of the integral in Equation we deduce that
(ax)))= = Ck- (x) c++ (ar)
0 < k < d 1, Ca-1 = Ca- (
(8)
4k
4(d - 1)"
and Cu(x) depends on the constant of integration ((x) in Equation 5 and the other coefficient
values in Equation 7
This freedom of the predicted T in (2) allows us to impose a new condition,
which becomes uniform condition all of the intervals [o, 1]. Section 5.1 we will discuss how to define the Co( depending on the black box desired.
5.1 ADDING AN UNCERTAINTY ESTIMATION TO A BLACK-BOO Box PREDICTION SYSTEM
In this subsection, we tackle the constrained black -lx uncertainty modelling problem introduced in Section The main assumption is that we have pointwise predictive system, which will refer to as (x)) and approximates a desired statistic such as the mean, median or a certain quantile of pty m)) shown in Figure It is not necessary for this system be deep learning model or even parametric. All that the ChePAN requires to train its neural network, é, are the corresponding 8-evaluation values of the training
i.c. {æ, (x)). Thus, the ChePAN calculates the conditioned response distribution to the input without assuming asymmetry or unimodality with respect to this distribution, as well as associating the desired statistic of this distribution to ((x)).
The formula used to calculate the constant of integration, Co(x). will depend on which statistic we choose"| If we impose the quantile r to 3 (which we shall call ChePAN-B=qo). then
Co(x) = 23(x) - Σ (=(x)(-1)*.
(9)
However, if we force the quantile 1 to be the (which we shall call ChePAN-B=qj), then
Co(x) = 23(x)) 2 Σ Cx(x).
(10)
For instance, the prediction of extreme weather events involves the forecasting system to predict the maximum minimum values of p(y a=). In these cases, this pre-trained system could be used as in Equation Saatiioonooor Equation 10 respectively, to determine the overall quantile distribution of p(v x))
taking as reference point.
If the median (equivalently, 0.5) is the 3 (which we shall call ChePAN-B=Med) then
Co(x) = 2,(xx)
Σ (C())))((()).
(11)
even
Finally, the mean forced to be the 3 (which we shall call ChePAN-B=Mean then
Ca(a)
Ca(x) 23(x) -
(12)
odd
Additionally, (x)) can approximated by means of another neural network, which can be simul- tancously optimised with @(T, æ). We will use this approach to compare the ChePAN and other baseline models in the results section regarding black-box modelling.
EXPERIMENTS
The source code used to reproduce the results of the ChePAN in the following experiments can be found in the Github repository The DCT-II method referred to in Section was used in the aforementioned source code.
All details of how such formulas are reached can be found in the supplementary material.
The camera-ready version of this paper will include all of the source codes to reproduce the experiments.

---

Under review as a conference paper at ICLR 2021
Table 1: Mean and standard deviation of the QR loss value, mean + std, of 10 executions for each Black box -Uncertainty wrupper using of the test distributions in Figure and three data sets (described in Section A.6). The ranges that overlap with the best range are highlighted in bold.
Asymoettrc
Symmetric
Uniform
Multimodal
Year-MSD
BCN-RPF
YVC-RPF
RF
NN
42.37 t: 0.04 23.19 +: 1.00
66.44 0.26
151.51 i: 0.24
57.50 i .05 23.47 + .14
27.27 1: .39
RF
-.P
42.88
22.10 0.08
67.13
153.06 0.22
57.58
+ .17
28.06 .12
RF
ChePAN
41.52 0.35
23.19 + 0.70
65.98
0.20
148.39 = 0.16
48.28 + .18
23.17 .07
28.16 1 ..44
Boost
NN
42.42 : 0.05 23.35
0.99
66.38 0.26
149.35 1 0.40
51.17 .88
24.52
27.79 088
XGBoost
-LP
42.90 0.02 23.02
0.43
67.13
0.17
150.94 0.12
51.24 .02
22.63
27.86
XGB. -ChePAN
41.95 = 0.40 23.69 0.68 65.89
0.17
146.20
0.30
48.54 +
22..0
22.00
N
43.63 + 2.89
23.70
67.45
1.68
148.78 2.88
49.00 244
27.28 1.25
28.62 22.
LP
43.46 +: 0.15 20.72 + 0.47
68.06
0.82
149.99
+ 0.64
48.67 = .88
23.51 + 28
22.32 .06
ChePAN
41.72 0.24 22.94
1.81 68.55
6.61
145.93 + 3.14
46.76 + .25
20.67
40
21.97 .1
In this section, we describe the performance of the proposed models compared to other baselines. The main goal is show that by using QR the ChePAN an improvement on other black-box uncertainty modelling baselines because it avoids centrality or unimodality assumptions, while also allowing users to choose how constrain the predicted quantiles with respect to the black-box prediction.
6.1 MODELS UNDER EVALUATION
Exponential power distributions satisfy the condition that one of the parameters corresponds to the mode. Thus, those models that approximate such parametric distributions where the mode parameter is the black box function and estimate the other parameter related to uncertainty can be used as baselines.
The Heteroscedastic Normal distribution (N) Similarly to (Bishop (1994); Kendall &
Gal (2017); Tagasovska & Lopez-Paz (019); Brando et al. (2019)) two neural networks, μ and Ø, can be used to approximate the conditional normal distribution, NNu(z)), o (x)), such that they maximise the likelihood.
In the black-box scenario proposed here, / is the black-box function and we only need to optimise the neural network. Once optimised, the desired quantile can be obtained with function.
1), J e (0, 1), where erf is the inverse error F(r, .)) = pax) + (xx)/2 erf crf-l(2: ((2
The Heteroscedastic Laplace distribution (LP) As more robust alternative outlier values, conditional Laplace distribution, LP (m(z), Mx)). can be considered. Here, the quantile function is F(r, (ax)) + (blog(2r)) (r) [[r x 2 (blog(2 -r)) 1[r 21.
I (0, 1).
The Chebyshev Polynomial Approximation Network (ChePAN) In order to use the same black boxes the other baselines, Equation 12 considered, given that these black boxes are optimising the mean square error. Other alternative equations are considered in the pseudo code and in Figure 6 of the supplementary material.
6.2 DATA SETS AND EXPERIMENT SETTINGS
All experiments (2019), ) running in a workstation with Titan (Pascal) GPU and GeForce RTX 2080 GPU. All the details of the data sets used and model paramepas for the results section are described in the supplementary material.
implemented in TensorFlow (Abadi al. (2015) and Keras (Chollet et al.
6.3
RESULTS
Table shows a comparison of uncertainty modelled for two given black-box systems (a Random Forest (RF) (Liaw et al. (2002) and XGBoost (Chen & Guestrin (2016)) in four data sets. The

---

Under review as a conference paper at ICLR 2021
Estimated quanciles Food Bata (RF) mean syothetic
Asymmetric
Symmetric
Unifarm
Multimodal
Ediinttall Leantt Beta (NN) meon
systhetic data
Asymmetric
Symmetric
Uniform
Multimodal
the upper part of the figure, the learnt quantiles, Φ, are noisy because their mean is the black box defined as an inaccurate MSE Random Forest (RF), 3, following Equation [2] In the lower part, ø and 3 are learnt and asymmetries and multimodalities can be seen more clearly, while still respecting the constraint in Equation 12
Figure 3: Heterogeneous synthetic distribution proposed by (Brando et al. (2019)).
first four columns correspond to each part of the synthetic distribution proposed by (Brando et al. (2019) and shown in Figure 3 the fifth column is the full Year Prediction MSD UCI dataset (Dua & Grafi (201 predicting the release year of song from 90 audio features and, finally, the last two columns correspond predicting the room price forecasting of Airbnb flats (RPF) in Barcelona and Vancouver, extracted from (Brando et al. (2019)) The mean of the QR loss value (see Equation [ is evaluated for ten thousand randomly selected quantiles for ten executions of each model {m²))=)
Neet Σ
(y: - Sm, (m, ai)) (T₃ [ys < Smy (T,, xx)))
Lmy (Xtest; Ytest)
Σ
(13)
Ntest N₂
where Neest is the number of points in the test set, N, 10, 000 the number of Monte Carlo samplings and fm, any of the models considered in Table [ Considering how the QR loss is defined in Equation [ its value not only informs us about each system's performance but also how generically calibrated its predicted quantiles are.
Furthermore, in Table I we observe that the ChePAN outperforms other methods in most cases due to it transferring the capacity to capture asymmetries and multimodalities of QR in p(y x)) to the black-bor box problem, where our uncertainty modelling needs to be restricted in order to maintain the corresponding statistic associated with the black box.
This restriction of conserving the black box can be seen qualitatively in the upper part of Figure 3 where such restriction must be met in any situation, i.e. if performance worsens because the black box, (zx), not correctly fitted (as described in Section 2 In this case, B(x) an inaccurate Random Forest predicting the mean. Importantly, the ChePAN propagates the ((x) noise to the predicted quantiles (in blue) because the constraint is always forced. On the other hand, the ability of ChePAN model heterogeneous distributions using QR is better displayed the lower part of
Figure 3 In this case, the black box is a neural network that is learnt concurrently with the quantiles. Since the black box better approximated, the quantiles are better.
Finally, since Table I shows that there is similar performance order between the baselines when using the RF or XGBoost, we also want show additional experiments that directly measure the calibration of the predicted quantiles and compare the predicted width of certain desired intervals. Following the UCI data sets used (Hemáánezz-oobtoo & Adams (0115)); Gal & Ghahramani (016);; Lakshminarayanan et al. (2017); Tagasovska T & Lopez-Paz 2019)). we performed two em- pirical studies to assess this point in a black-box scenario where the black box is an MSE- ASS--t.oost..... Following the proposed hidden layers architecture in (Tagasovska & Lopez- Paz (2019) ), the Pre- diction Interval Coverage Probability (PICP) and the Mean Prediction Interval Width (MPIW) are reported in Table 3 of the appendix considering the 0.025 and the 0.975 quantiles. For the sake of
8

---

Under review as a conference paper at ICLR 2021
XGBoost -N
XGBoost LP
XGBoost ChePAN
Peckec
Concrete
14.40 1.7
14.35 1.5
5.99 I 4.4
28-
Power
6.31
6.63 i 1.6
4.15 = 1.4
Wine
4.40 1.2
4.34 1.2
4.58 1.6
Yacht
8.93 + 2.3
6.98 + 2.4
14.20 + 3.9 3.9
Naval
6.61 2.8
7.21 2.1
6.90
Energy
6.12 2.1
5.05 + 2.0
6.26 2.2
22-
Boston
15.55 3.1
15.30 2.1
4.99
Kin8nm
3.55 + 0.6
4.79 - 0.5
5.25 0.8
Protein 7.88 + 1.9
7.30 0.3
5.39 0.8
Gelected spAntiio
Figure 4: Plot with performance terms of calibration. The table contains the mean and standard deviation of all the folds using the mean absolute error hetween the empirical predicted calibration and the perfect ideal calibration of 980 equidistant quantiles using Equation [4
completeness, in Figure 4 and its associated table we have also computed an additional metric not only to verify the calibration of the 0.025 and 0.975 quantiles, but also to obtain a measure of gen- eral calibration considering the entire quantile distribution. Given N, Nz-equidistant set of quantiles to evaluate, т [10-
10-2], the % actual test data that falls into each predicted quantile can be compared to each real quantile value as follows,
N.
Neast
CallfiXteat Yrest, T)
1
Σ
Il < f(T,((()))))) æx]]
(14)
Nr
Neeat
In addition, extra figures showing the disentangled visualisation of this calibration metric from each quantile can be found in Figure of the Appendix. As all of the figures and tables show, in terms of calibration, the ChePAN generally displays a better performance in the black-box scenario than the other models.
7 CONCLUSION
The uncertainty modelling of black-box predictive system requires the designing of wrapper so- lutions that avoid assumptions about the internal structure the system. Specifically, this could be non-deep learning model (such as the presented in Table [ and Figure 3) even parametric predictive system, as proposed in Figure T Therefore, not all models or types of uncer- tainties can be considered using this framework.
The present paper introduces the Chebyshev Polynomial Approximation Network (ChePAN) model, which is based on Chebyshev polynomials and deep learning models and has dual purpose: firstly, it predicts the aleatoric uncertainty of any pointwise predictive system; and secondly, it respects the statistic predicted by the pointwise system.
To conclude, then, the ChePAN transfers the advantages of Quantile Regression (QR) to the prob- lem modelling aleatoric uncertainty estimation in another existing and fixed pointwise predictive system (denoted as 3 and referred to as a black box). Experiments using different large-scale real data sets and a synthetic one that contains several heterogeneous distributions confirm these novel features.
REFERENCES
M. Abadi, P. Barham, Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving. M. Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th [USENIX) Symposium Operating Systems Design and Implementation ((OSDI) 16), pp. 265-283, 2016.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew

---

Under review as a conference paper at ICLR 2021
Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015. URL httt::://ww.tens//// orgg/ Software available from tensorflow.org.
Christopher M Bishop. Mixture density networks. 1994.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXtv:1550.00542. 2015.
Axel Brando, Jose A Rodriguez-Serrano, Jordi Vitria, and Alberto Rubio Muñoz. Modelling het- erogeneous distributions with an uncountable mixture of asymmetric laplacians. In Advances in Neural Information Processing Systems, pp. 8836-88468 8846, 2019.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pp. 785-794, 2016.
Chollet et al. Keras (2015), 2019.
C. W. Clenshaw. A note on the summation of Chebyshev series. Math. Tables Aids Comput., 9: 118-120, 1955. ISSN 0891-6837.
Murray Cox. Inside airbnb: adding data to the debate. Inside Airbub [Interntt.,ciited May 2019]. Available: http://insideairbbb.cc 2019.
Ward Cunningham. The wycash portfolio management system. ACM SIGPLAN OOPS Messenger, 4(2):29-30, 1992.
W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit quantile networks for distributional re- inforcement learning. In International Conference on Machine Learning, pp. 1104-1113, 2018a.
W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018b.
Germund Dahlquist and À ke Björck. Numerical methods in scientific computing. Vol. I. So- ciety for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2008. ISBN 978- 0-898716-44-3. doi: 10. 10.111371.97880888777777 URL http://ooo..o011.11111111 9780898717788
Armen Der Kiureghian and Ove Ditlevsen. Alcatory epistemic? does it matter? Structural safety, 3152):105-112, 2009.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017a. URLhttp://arcchie ics. uci. www.wu...mmmmmmmmmmmm
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017b. URL/attp://archiee ics. uci. www..eu/mm...
D. Elliott. Error analysis of an algorithm for summing certain finite series. J. Austral. Math. Soc., 8:
213-221, 1968. ISSN 15.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. arXiv:1506.00 02 02242,1422220 2015.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050~1059, 2016.
M. Hernández-Lood and R. Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In ICML., pp. 1861-1869, 2015a.
10

---

Under review as a conference paper at ICLR 2021
José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learn- ing of bayesian neural networks. In International Conference on Machine Learning, pp. 1861- 1869, 2015b.
Kurt Horiik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural networks, 2(5):359-366, 1989.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, pp. 5574 -5784, 2017.
Balaji Lakshminarayanan Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Andy Liaw, Matthew Wiener, et al. Classification and regression by randomforest. R news, 2(3): 18-22, 2002.
H. Majidian. the decay rate of Chebyshev coefficients. Appl. Numer Math., 113:44-53, 2017. ISSN 0168-9274. doi: 10. 10166:..pmum2216.111111 11.0004 URL https:://doii org/ 10. 1016/
5. Jappumm 2016. 11. .004
A. C. R. Newbery. Error analysis for polynomial evaluation. Math. Comp., 28:789 793, 1974, ISSN 0025-5718. doi: 10.2307/2005700. URL
Carl Edward Rasmussen. A practical monte carlo implementation bayesian learning. In Advances in Neural Information Processing Systems, pp. 598-604, 1996.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.
David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. Hidden technical debt in machine learning systems. In Advances in newral information processing systems, pp. 2503-2511, 2015.
N. Tagasovska and D. Lopez-Paz. Frequentist uncertainty estimates for deep learning. Bayesian Deep Learning workshop NeurIPS, 2018.
Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning. In Advances in Neural Information Processing Systems, pp. 6417-6428, 2019.
Mattias Teye, Hossein Azizpour, and Kevin Smith. Bayesian uncertainty estimation for batch nor- malized deep networks. arXiv preprint
2018.
L. N. Trefethen. Is Gauss quadrature better than Clenshaw-Curtis' SIAM Rev, 50(1):67-87, 2008. ISSN 0036-1445 doi: 10.1137/060659831. URL https:://cc. org 11377 060659831
Berk Ustun and Cynthia Rudin. Supersparse linear integer models for optimized medical scoring systems. Machine Learning, 102(3)344-391, 2016.
A. Wehenkel and G. Louppe. Unconstrained monotonic neural networks. In Advances in Neural Information Processing Systems, 2019.
R. Wen, K. Torkkola, B. Narayanaswamy, and D. Madeka. A multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11055, 2017.
H. Zheng, Z. Yang, W. Liu, Liang, and Y. Li. Improving deep neural networks using softplus units. In 2015 International Joint Conference Neural Networks (IJCNN, pp. 1-4. IEBE, 2015.
11


Reviews:
[{"review_id": "PZoBM8T0AXl", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-------------------\n# Update\n\nAfter reading the updated version of the paper, I think my main issues with presentation have been addressed. I now see this work as introducing a relatively niche problem and solving it elegantly. The text is clear enough for an average ICLR attendee to understand and contains additional background information in the appendix. I have increased my score to a 7.\n\n-------------------\n\n### Summary / Contributions \n\nThis paper adapts quantile regression with NNs to settings in which a summary statistic is fixed (some percentile or moment). This is done by having a NN predict the coefficients of a Chebyshev polynomial which is then integrated to evaluate the predicted cumulative density function over targets. Freedom in the choice of the constant of integration is used to enforce a summary statistic provided by a black box model (BBM).  \n\nThe method is used to provide heteroschedastic noise models for black box regression systems and validated on standard regression benchmarks (UCI).  \n\n### Originality and Significance\n\nTo the best of my knowledge, the novel aspects of this work are:\n\n* The development of a derivative based approach to quartile regression which allows for the output distribution to be conditioned on an input x. This could be used very generally as a heteroscedastic noise model.\n* The extension of the above approach to allow enforcing some summary statistic like distribution mean or percentiles.\n* The introduction of post-hoc noise modelling for black-box decision making systems as a task. \n\n#### Pros:\n* Sound solution to a (to the best of my knowledge) new task: constrained heteroscedastic noise modelling with free-form / implicit distributions.\n* Good empirical performance on benchmark datasets. \n\n#### Cons\n* The paper is not very well written and hard to follow. I had to read it multiple times to understand it well. I felt important details where glossed over or missing in the provided explanations. \n* I would have liked to see the method being evaluated further by using it with a wider variety of black box models and different amounts of training data. \n* Motivation of adding noise models to black box predictors is a bit weak considering no evaluation is performed on real-world scenarios with BB models in deployment.  \n\n### Other Comments:\n\n#### Method: \n\nThis method is completely agnostic to the BBM and, as such, its uncertainty might be in contradiction with the BBM given statistic:\nWhen a noise model is learnt in conjunction with a predictive model (e.g. a NN that predicts mean and st deviation values for each input) , aleatoric uncertainty usually reflects what the model can’t predict. In the limited data regime, this is often a crude approximation to the inherent uncertainty in the generative process of the data. A not very flexible (e.g. linear) model will see non-linearity in the function being learnt as noise in the data. When using a very flexible model (e.g. NN) might be able to fit all datapoint and not see anything as noise.  As such, I would have liked to see more exhaustive evaluation of the proposed methods in the following settings:\n\n*  It would be interesting to see the effectiveness of this technique on varying sized datasets when the NN used for quantile prediction is very flexible.\n*  What happens when the BBM is not noisy (as in Fig 3.) but instead biased because it is not flexible enough for problem at hand. This his often the case in real world deployments when the model is chosen for interpretability (e.g. decision tree or linear regressor) — I imagine the inability of this approach to consider model specification uncertainty, could cause noise distributions to misbehave when the BBM is very badly specified. \n\n\n#### Experiments\n\nThe provided baseline methods seem relatively weak: just unimodal heteroscedastic noise models. However, I do not know of any existing approaches that impose summary statistic constraints on arbitrarily complex predictive distributions. \n\n#### Presentation\n* The language used is a bit ungrammatical at times. I would recommend going over the paper with a grammar checker. There are some paragraphs that are only 1 sentence and feel a bit disconnected. \n* A brief overview of key preliminaries would be convenient: Relevant properties of Chebyshev polynomials, how the DCT can be used to obtain coefficients, etc\n* Some important concepts are glossed over or referenced to succeeding sections, despite them being necessary to follow the paper well. It is better to reference previous sections than future ones. \n\t* It was not clear to me from the explanation at the end of section 2 why existing quantile regression methods would not be applicable to the constrained black box scenario.\n\t* It was not clear to me from section 3 why the solution to the above problem required using a UMNN style model. \n\t* In Section 4 it is not clear what “Due to the aforementioned truncation in ChePAN” refers to, as above you state both CCN and ChePAN calculate coefficients of truncated expansions. In general, it is not very clear to me how mentioning CCN in section 4 is helpful to the reader. I would recommend just explaining ChePAN in more detail and leaving CCN for the appendix.\n\t* The reference to section 4.1 in the paragraph after equation 3 leaves me wanting an explanation of why we want Chebyshev roots to fit in [0,1] instead of [0, \\tau].\n* What does the bolding in Figure / Table 4 refer to? It seems inconsistent. In table 1 it indicates best methods.\n\n### Summary\nI think this paper contains good contributions but is held back by poor presentation. If this is corrected by the authors, I would be happy to recommend acceptance. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "JYgUaEmpVbF", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to model aleatoric uncertainty for the black-box model, through the combination of Chebyshev polynomial approximation and quantile loss. The method seems novel to the prior work and achieves good results in capturing uncertainty in predictive modeling. My detailed comments are as follows:\n\nStrength:\n1. By constraint the neural network model for the derivative function $\\phi(\\tau, x)$ to be strictly positive, this method solves the quantile crossing issue which is a common problem in simultaneous quantile regression.\n2. This method does not assume any prior knowledge of the model structure, which means one doesn't need to modify the parameters of the original model, and therefore separates the constraint from the model training procedure.\n\nWeakness:\nFrom section 4 and Appendix it looks like the way to compute each coefficient is in a sequential manner, which may not be efficient enough.\n\nAdditional Questions:\n1. If ChePAN is a separate module of the black-box model, is this similar to a post-processing method for pre-trained models to generate user-specified uncertainty intervals?\n2. Following the previous question, if that is the case, does that mean we need to train the black-box model to produce $\\phi(\\tau, x)$ for later use? Or add an additional neural network called uncertainty wrapper for that?\n3. It looks like CCN is not discussed in the experiment section.\n\nUPDATE after author response:\nEstimating arbitrary quantiles to approximate the predictive distribution of a black-box model is a novel problem to look into. In response, the authors have attempted to address my major concern and they are mostly clear. I would suggest the authors add more discussion on the motivation of the problem settings, e.g., whether it arises from an actual business problem, why satisfying constrained black-box uncertainty problem is a must-have, etc.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AaylN7Hs6Hs", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of uncertainty estimation and it proposes to compute quantiles of a black-box predictor using Chebyshev-interpolated quantile regression, where the interpolation parameters are computed with the aid of a neural network. \n\nThe paper is generally well-written, though I am leaning toward rejecting the paper because the ideas are poorly motivated and the text does a poor job of convincing readers there is an open problem being addressed here that prior work has not already done. I've provided a few comments and questions below to support my position. \n\nThe paper’s clarity could be improved by explicitly describing the research question it considers and motivating its solution choices by connecting them to the research question and related work. Can the authors make these details clear: what is the research question you consider and what contributions does this paper offer?\n\nCurrently, it is not clear where this paper sits in relation to prior work. It was mentioned that two starkly different methods -- Dabney et al 2018a and Tagasovska & Lopez-Pax 2018 -- are related to the considered problem but also not applicable, because “the predicted quantiles need to be linked to the black-box prediction in some way.” It seems like such a connection can be established by creating a dataset where the predictions are the targets. Can the authors elaborate on their stance here?\n\nCan the authors please clarify their use of the term “aleatoric uncertainty”? Typically, aleatoric uncertainty describes the inherent variability of a stochastic signal. In contrast, the epistemic uncertainty corresponds to the variability in a stochastic signal’s estimate. This latter type of uncertainty is something that shrinks as more data is collected. The proposed method seems to estimate the uncertainty of a predictor, which would mean it is estimating the epistemic uncertainty of the estimator. However, the paper refers to this as aleatoric. What is the distinction used here?  \n\nThe proposed approach has questionable utility, because estimating the uncertainty of a prediction is arguably a job for Bayesian methods. The current paper makes very little mention of this and takes for granted that a point estimator is provided, but that one wishes to have uncertainty estimates of the predictions. How does the proposed approach compare to simply computing / estimating a posterior over the prediction? The true posterior will of course be difficult to compute in many domains, but the related work and experimental design should acknowledge the existence of alternatives here: e.g. Bayesian neural nets, variatoinal inference. Can the authors describe when it would be more appropriate to use the proposed method over a Bayesian approach?\n\nCan the authors please justify their use of the UMNN architecture and describe why a simpler alternative, such as standard quantile regression or interpolated quantile regression suffice? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-TgZ_Hjy2jB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary / Weaknesses\n\nI have to confess that I found the paper to be extremely confusing. It is clear that the submission is still several edits away from a version that can be published.\n\nAs far as I can see, the paper lacks a clear, formal description of the problem, and as a result, it is difficult to see precisely where the existing methods fall short (or even which ones are relevant), and how the proposed procedure addresses these shortcomings. Again, as far as I can see, there is little to no theoretical justification for the proposed method in the paper; the proposed method appears to be a heuristic based on approximations that may be reasonable, but requires some justification.\n\n---\n## Recommendation\nI recommend a reject.\n\n---\n## Questions\nPlease give a formal description of the problem the proposed method is trying to address. Also, please give brief but clear descriptions of the existing approaches and their shortcomings.\n\nI am particularly baffled by the sentence \"[the QR models] cannot be be applied to the constrained black-box scenario given that they does (sic) not link their predicted quantiles with a pointwise forecasting system in a constrained way.\" The way I interpret this sentence is that the existing methods do not model the quantiles (or some other functional) of the _output_ of some pointwise forecasting system, but rather they model functionals of Y | X. If this is the correct interpretation, then I do not see why it would be of interest to model the predictions of a black-box forecasting system rather than Y | X.\n\nI am also confused by the experimental setup. Why do N and LP make sense as comparison methods?\n\nAlso, there are several instances of errors of grammar / syntax.\n\n---\n## Disclaimer\nI did not have the time to look at the supplement.\n\n---\n## Update\nI thank the authors for responding to my questions and revising the paper.\n\nAs indicated in the first sentence of my review, the submission's biggest flaw is in poor presentation. I have read the revised version, but I am afraid to say that I remain disappointed. Although I can see that the authors have tried to address some of the common concerns, the changes do not go far enough. Given that the current version is a version in which \"all [...] comments and suggestions have been taken into account,\" I do not see a reason to give the authors a benefit of the doubt, and have decided to keep my score as is. However, because my evaluation appears to be quite different from those of the other reviewers, I think it may be possible that they are seeing something in this paper that I am incapable of. Since I cannot profess to have a complete understanding of the content, I have decided to downgrade my confidence score.\n\nI will give two examples of ambiguous / confusing language. They are both from Section 2 after the paragraph heading **Definition of constrained black-box uncertainty modeling**.\n\n1. (The last sentence of the 1st paragraph) \"Given this context, the pointwise forecasting system mentioned above is a function $\\beta:\\mathbb{R}^D \\to \\mathbb{R}$, which tries to approximate a certain conditional summary statistic (a percentile or moment) or $p(y | x)$.\n\n- My understanding of this sentence is that $\\beta$ is a _statistic_ (i.e., a function of $\\mathcal{D} = (\\mathbf{x}_i,y_i)_{i=1}^{n}$) such that $\\beta \\approx \\beta^*$, where $\\beta^*$ is some functional of interest of the conditional distribution $p(y | \\mathbf{x})$. For example, $\\beta^*$ could be the conditional mean $\\beta^*(\\mathbf{x}) = \\mathbb{E}[Y | X = \\mathbf{x}]$, in which case $\\beta$ is usually obtained by minimizing the MSE, or $\\beta^*$ may be the conditional $\\alpha$-quantile, in which case $\\beta$ is some data approximation of the conditional $\\alpha$-quantile.\n- Now, compare the above sentence to the following sentence from the author response: \"For instance, $\\beta(\\mathbf{x})$ can be the conditional mean of $p(y | \\mathbf{x})$.\"\n\nMy point here is that throughout the paper, $\\beta$ is used to refer to both a functional of the conditional distribution or a data-based approximation for the said functional. I find the lack of distinction puzzling, if not downright confusing.\n\n2. (The 5th paragraph. I think this is supposed to address my objection about the lack of a precise problem statement.) \"The overall goal of the present article is, taking a pre-defined black box $\\beta(x)$ that estimates a certain conditional summary statistic of $p(y | \\mathbf{x})$, to model $q(y | \\mathbf{x})$ under the constraint that if we calculate the summary statistic of this predicted conditional distribution, it will correspond to $\\beta(\\mathbf{x})$.\"\n\n- What is $q(y | \\mathbf{x})$, and what is its precise relationship to $p(y | \\mathbf{x})$? The first occurrence of $q(y | \\mathbf{x})$ is followed by the description that it is \"a conditional density model,\" but this can mean many things. Judging from the remainder of the article, $q(y | \\mathbf{x})$ is either a model defined through a location-scale family or a model defined via a specification of quantiles. The proposed method is exclusively concerned with the latter kind of models. If this is the case, it would help the reader to have this said explicitly when the symbol $q$ is first introduced.\n- The usage of the term \"statistic\" does not appear to be standard. A _statistic_ is a function of the data $\\mathcal{D}$, and is therefore a random quantity. A percentile or a moment of $p(y | \\mathbf{x})$, on the other hand, is a functional of the conditional distribution $p(y | \\mathbf{x})$, and would be non-random.\n- Even ignoring the previous points, the sentence made little sense to me. I will say more on this after the following paragraph.\n\nMore on the subject of clarity, notation-wise, I am bothered by the proliferation of $p$'s. Aside from the occurrence in $p(M | \\mathbf{x})$, which is rare and hence not a cause for concern, $p$ is used to represent the conditional density (distribution) $p(y | \\mathbf{x})$, as well as a generic Chebyshev polynomial approximation $p(\\tau,\\mathbf{x};d)$. It is possible that I am failing to see some deeper connection here, but if none exists, I would prefer a more clear system of notations.\n\nReturning to the problem statement, after re-reading the paper several times and with the help of the author response, I have arrived at the following interpretation:\n\nFind an estimate / model $q(y | \\mathbf{x})$ for the conditional density (distribution) $p(y | \\mathbf{x})$ such that $q$ and $p$ have the same $\\beta$. Here, $q$ is given by specifying the quantiles, and is approximated by a Chebyshev polynomial of degree $d$. The usual QR methods are inadequate, because the model they output will not have the same conditional percentile / moment / etc. as $\\beta$.\n\nHowever, in situations in which $\\beta$ is also being approximated, what is the point in constraining the functional at $\\beta$? As pointed out by the authors, if $\\beta$ is inaccurate, then this error is propagated to the quantiles, and the performance is worse. If this is the case, why is it of practical interest to enforce the constraint? Had the authors given a concrete, illustrative example, this question may not have arisen. However, absent a practical motivation, it is hard to understand why the constraint needs to be enforced at all.\n\nIn any case, suffice it to say that I am still experiencing significant difficulty pinning down the exact goal of the paper.\n\nAt the time of writing my initial review, I had hoped that the issues with clarity would be fixed in the revision, leaving me free to evaluate the paper based on the merits of the proposal. However, this has not been the case. Now that I am more familiar with the paper, I realize that the biggest issue about the clarity has much to do with the organization, ambiguous language, etc. Some of it has been addressed in the revision, but the effort did not go far enough. \n\nEven leaving aside any reservation I have about clarity from the point of view of methodology and/or theory, I have to say that I am deeply dissatisfied about the lack of practical motivation. I believe the authors intended the paragraph after Section 2 to be a response to concerns about poor motivation (which have been raised by other reviewers, not I): \"The present article was motivated by a real-world need that appears in a pointwise regression forecasting system of a large company. Due to the risk nature of the internal problem where it is applied, uncertainty modeling is important.\" It would have been nice if the authors had chosen to respond by giving a _concrete_ example, complete with a real data set and an actual task to which their method is an adequate solution. However, the example given in the added paragraph is far too generic and vague to be useful as an illustrative example.\n\nFinally, when I asked for theoretical justification, I was looking for some actual proofs about e.g., consistency guarantees, approximation quality, etc.\n\n---\n## Minor\nI find the size of the text in the plots to be much too small and hard to read. Also, please label all axes in Figure 3.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-1DcLB7cFYd", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel approach to modeling uncertainty, as an layer added-on to an otherwise black-box system. The ChePAN uses a neural network to estimate per-quantile roots of a chebyshev polynomial, then uses a quantile regression loss to fit these coefficients using backpropagation. Importantly, the Chebyshev polynomial formulation gives the practitioner some flexibility around deciding which statistic of the conditional CDF should be matched by the black box system. Examples are given for matching the 0 and 1 quantiles, for cases of min and max estimation, as well as matching either of median or mean.\n\nThe authors make a case for why previous/related approaches to black box uncertainty modeling have deficiencies e.g. the roots considered by the Clenshaw-Curtis quadrature extension of UMNN (CCN) are bounded by the quantile being considered, which is somehow limiting, though I was a little hazy on why. \nI would be curious whether the authors have considered splines as another alternative, at least for modeling black boxes known to be in a bounded, e.g. 0-1, range. https://arxiv.org/abs/1906.04032 is perhaps relevant there.\n\nThe idea is quite a neat one. One uses a neural network to emit positive-valued roots of a Chebyshev polynomial, which are optimal function approximators on the 0-1 interval, perfect for a flexible CDF inversion, thus an extremely flexible distribution estimator. This is what enables setting a statistic of choice equal to the output of the black-box system. The actual implementation does not seem overly complex, which should make it widely accessible to practitioners. The idea of combining quantile regression with deep networks for fitting conditional distributions is not new, as the authors acknowledge, but this work provides a mechanism for matching a statistic to that distribution, while also being able to fit that distribution using a QR loss.\n\nThe experiments use datasets comparable to prior/related work. However, tables and plots don't seem to include for comparison reference points like Brando, 2019's UMAL which appears to provide a nicely calibrated estimator for the AirBnB datasets. For example, I would prefer to see the calibration plot on the AirBnB datasets in Fig 4 left side, for comparison with this earlier work. Since the baselines are all internally sourced it's hard to tell if this paper is achieving state of the art calibration, or only providing a novel approach to doing so. On the topic of baselines not particularly comparable to prior work: the \"N\" and \"LP\" baselines, which in prior work often use neural networks for both the mean and the scales, in this work use RF/XGBOOST for the mean. It would be good to understand whether using a NN for the mean estimator yields changed calibration metrics (thus changing the baseline competitiveness).\n\nOn the whole, I think this is an exciting new contribution and would recommend accepting, but I think the paper would benefit from clearer evidence that the approach provides a value-add relative to existing techniques. If I'm meant to understand that \"N\" and \"LP\" baselines are that evidence, I must have misunderstood, because to me those look like custom baselines built for this work.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to model uncertainty by combining quantile
regression and Chebyshev polynomial approximation. The paper addresses
the important problem of uncertainty quantification for black box
models. However, some major concerns remain after the discussion among
the reviewers. In particular, there has been some concerns around the
clarity of the presentation. The proposal lacks a clear use case, e.g.
where satisfying constrained black-box uncertainty problem is a
must-have.

Decision: Reject